Information processing unit, control method for information processing unit, program, information processing system, and external display unit

Abstract

To provide a mechanism capable of improving second-user's visibility to an XR image shared with a first user while suppressing a processing load of rendering.SOLUTION: A CPU 301 of an information processing unit 101 acquires first viewpoint information 600 representing a viewpoint position of a user 401 in an MR image 402 being displayed on a display 203. When the information processing unit 101 acquires second viewpoint information 610 representing a viewpoint position of a supporter 403 in the MR image 402 being displayed on an external display unit 103, the CPU sets a first viewpoint region 601 and a second viewpoint region 611 based upon the first viewpoint information 600 and second viewpoint information 610, and then generates a virtual image by rendering the first viewpoint region 601 and second viewpoint region 611 with higher resolution than that of regions other than the first viewpoint region 601 and second viewpoint region 611, uses the virtual image to generate an MR image 402, and transmits the MR image 402 to the external display unit 103.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, a program, an information processing system, and an external display device. [Background technology]

[0002] In recent years, VR (Virtual Reality) technology and MR (Mixed Reality) technology have become well known. VR technology is a technology that renders a virtual world seamlessly in real time. MR technology is a technology that seamlessly fuses the real world and the virtual world in real time. Among these technologies, those that use HMDs (Head Mounted Displays) are known. Furthermore, one type of MR technology is known to use a video see-through HMD.

[0003] In VR technology, a virtual image is generated by rendering (drawing) CG (Computer Graphics) based on the position and orientation of the HMD. The generated virtual image (hereinafter referred to as a "VR image") is then displayed on a display device such as a liquid crystal display (LCD) or organic light-emitting diode (OLED) device of the HMD, allowing the wearer (user) of the HMD to view it. On the other hand, in MR technology, a captured image of the real space being observed by the user is acquired by a camera. A virtual image is generated by rendering CG based on the position and orientation of the camera. A composite image (hereinafter also referred to as an "MR image") is generated by combining the captured image and the virtual image, and is then displayed on a display device such as a liquid crystal display (LCD) or organic light-emitting diode (OLED) device of the HMD, allowing the wearer (user) of the HMD to view it.

[0004] Therefore, rendering is performed in VR and MR technologies. Rendering requires a high-performance GPU (Graphics Processing Unit) due to the recent increase in image resolution, but seamless real-time rendering requires reducing the processing load on the GPU. One example of a technology that achieves this reduction is foveated rendering. Foveated rendering reduces the processing load on the GPU and the amount of data transmission by increasing the resolution of the viewpoint area and decreasing the resolution of other areas based on the user's line of sight detected by a gaze detection mechanism.

[0005] Furthermore, Patent Document 1 proposes a technology that can reduce the processing load in CG rendering by performing rendering using user focus information. Furthermore, Patent Document 2 proposes a technology that, when an image to which foveated rendering is applied is shared by another person on an external monitor, displays an image that does not feel strange to the other person who is observing the external monitor. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2018-73104 [Patent Document 2] Patent Publication No. 2021-86287 Summary of the Invention [Problem to be solved by the invention]

[0007] There is a technology that allows a user to transmit an image viewed by a user externally via communication technology such as the Internet, thereby allowing others to view the image. Furthermore, this technology allows a user to share the user's viewing environment with a remote supporter, thereby providing the user with work assistance or services from a remote location. In this case, if the user and the supporter share an image in which the viewpoint area is high-resolution and the other areas are low-resolution based on the user's line of sight while wearing an HMD, the rendering processing load can be reduced. However, the supporter may want to focus on a different area from the user. In this case, the supporter has to focus on the low-resolution image of the area they want to focus on, which can cause a problem of not being able to see detailed information.

[0008] This applies not only to the VR and MR images described above, but also to synthetic images (hereinafter referred to as "AR images") generated by AR (Augmented Reality) technology, which seamlessly superimposes the real world and the virtual world in real time. In the following description, when VR images, MR images, and AR images are not distinguished from each other and are referred to collectively as "XR images."

[0009] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a mechanism that can improve a second user's visibility of an XR image shared with a first user while suppressing the processing load of rendering. [Means for solving the problem]

[0010] In order to achieve the above object, the information processing device of the present invention is an information processing device that shares an XR image with an external display device, and is characterized by comprising: a first information acquisition means that acquires first information indicating a first user's focus position in the XR image being displayed on a display; a first determination means that determines whether the information processing device has acquired second information indicating a second user's focus position in the XR image being displayed on the external display device from the external display device; a setting means that, when the information processing device has acquired the second information from the external display device, sets a first area and a second area based on the first information and the second information; a rendering means that generates a virtual image by rendering the first area and the second area at a higher resolution than an area different from the first area and the second area; a generation means that generates the XR image using the virtual image; and a transmission means that transmits the XR image to the external display device. [Effects of the Invention]

[0011] According to the present invention, it is possible to improve the visibility of an XR image shared with a first user by a second user while suppressing the processing load of rendering. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 2 is a schematic block diagram showing the functional configuration of the information processing system. [Figure 2] FIG. 2 is a schematic block diagram showing the hardware configuration of an HMD. [Figure 3] FIG. 1 is a schematic block diagram illustrating a hardware configuration of an information processing device. [Figure 4] FIG. 2 is a diagram showing an example of use of an HMD and an external display device in the first embodiment. [Figure 5] 10 is a flowchart illustrating an operation of the information processing device. [Figure 6] FIG. 1 is a diagram showing an example of foveated rendering in an MR image. [Figure 7] FIG. 2 is a schematic block diagram showing the functional configuration of an external display device. [Figure 8] FIG. 2 is a schematic block diagram showing the hardware configuration of an external display device. [Figure 9] 5 is a flowchart showing the operation of the external display device in the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of use of an HMD and an external display device in the second embodiment. [Figure 11] 10 is a flowchart showing the operation of the external display device in the second and third embodiments. [Figure 12] FIG. 11 is a diagram showing an example of use of an HMD and an external display device in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can perform the same function. Also, any component may be added. Furthermore, any two or more configurations (features) of each embodiment can be combined. Note that identical components are denoted by the same reference numerals in each drawing, and description thereof will be omitted.

[0014] First Embodiment The first embodiment will be described below with reference to FIGS.

[0015] [Configuration of information processing system] FIG. 1 is a schematic block diagram showing the functional configuration of an information processing system according to a first embodiment. As shown in FIG. 1, the information processing system 100 includes an information processing device 101, a video see-through HMD 102, and an external display device 103. The information processing device 101 includes a captured image acquisition unit 111, a gaze detection unit 112, a data storage unit 113, a processing determination unit 114, a virtual space generation unit 115, a rendering execution unit 116, an image generation unit 117, an image output unit 118, and a communication unit 119. The information processing device 101 is a small box computer that can be worn over a user's shoulder. The information processing device 101 is not limited to this and may be, for example, a laptop personal computer (PC), a mobile terminal such as a smartphone or a tablet, or a desktop personal computer. The HMD 102 is a head-mounted display device worn on a user's head to display images.

[0016] The HMD 102 is connected to the information processing device 101 so as to be able to communicate data with it. The connection between the HMD 102 and the information processing device 101 may be a wired connection or a wireless connection. The external display device 103 is connected to a communication unit 119 of the information processing device 101 via a network. The external display device 103 may be located in a remote location on the Internet or on a local area network (LAN). The connection between the external display device 103 and the communication unit 119 of the information processing device 101 may be a wired connection or a wireless connection.

[0017] FIG. 2 is a schematic block diagram showing the hardware configuration of the HMD 102. As shown in FIG. 2, the HMD 102 has an external camera 201, an eye-gaze camera 202, and a display 203. The external camera 201 is a camera for capturing images of the real space in front of the HMD 102, i.e., in front of each of the user's left and right eyes. The eye-gaze camera 202 is a camera for detecting the line of sight of each of the user's left and right eyes. The display 203 is configured using an organic EL display, a liquid crystal display, or the like, and displays images in front of each of the user's left and right eyes. Note that the information processing device 101 and the HMD 102 are configured as separate entities, but may be configured as an integrated entity by the HMD 102 having the functions of each of the units of the information processing device 101. In this case, the HMD 102 configured as an integrated entity with the information processing device 101 is the information processing device of the present invention.

[0018] 3 is a schematic block diagram showing the hardware configuration of the information processing device 101. As shown in FIG. 3, the information processing device 101 has a hardware configuration including a CPU 301, a GPU 302, a ROM 303, a RAM 304, and an HDD 305. The CPU 301, the GPU 302, the ROM 303, the RAM 304, and the HDD 305 are communicably connected via a system bus 306. The CPU 301 controls each unit connected to the system bus 306. The CPU 301 executes programs stored in the ROM 303, the HDD 305, and the like, thereby realizing, for example, each functional configuration and processing of a flowchart described below.

[0019] The GPU 302 is controlled by the CPU 301 and programs, and executes virtual image generation processing (rendering processing). The ROM 303 has a storage area for storing programs and the like executed by the CPU 301. The RAM 304 has a storage area that functions as a work area when the CPU 301 expands and executes programs. The HDD 305 has a storage area for storing various programs required when the CPU 301 executes processing, and various data including data required for processing, which will be described later.

[0020] Returning to the description of FIG. 1, the functional configuration and the like of the information processing device 101 will be described in detail below. The captured image acquisition unit 111 acquires a real space image for the right eye and a real space image for the left eye of the user wearing the HMD 102 using the external camera 201 of the HMD 102. The captured image acquisition unit 111 stores each of the acquired real space images in the data storage unit 113. The gaze detection unit 112 acquires an image for each of the left and right eyes of the user wearing the HMD 102 using the gaze camera 202 of the HMD 102. The gaze detection unit 112 detects the gaze of the user from each of the acquired images.

[0021] The gaze detection unit 112 calculates the coordinates of the user's point of gaze from the detected user's gaze, and acquires the calculated coordinates as first viewpoint information indicating the user's gaze position. The gaze detection unit 112 stores the acquired first viewpoint information in the data storage unit 113. Note that the detection of the user's gaze may be performed based on an image acquired by the gaze camera 202 as described above, or may be performed by a sensor different from the gaze camera 202. Furthermore, the gaze camera 202 may have a configuration different from that shown in FIG. 2; for example, there may be one for each of the user's left and right eyes, or three or more cameras.

[0022] The processing determination unit 114 determines how to perform rendering processing on the virtual space information in a predetermined area including the user's gaze point calculated by the gaze detection unit 112. Specifically, the processing determination unit 114 determines how to perform foveated rendering, which will be described later. Note that second viewpoint information, which will be described later, is also stored in the data storage unit 113 in the same manner as the first viewpoint information, and how to perform rendering processing on the virtual space information is determined by the processing determination unit 114. The second viewpoint information is information that indicates the focus position of the supporter, which will be described later.

[0023] The virtual space generation unit 115 generates a virtual space in accordance with the determination made by the processing determination unit 114. The rendering execution unit 116 executes virtual image generation processing (rendering processing) for each area in accordance with the determination made by the processing determination unit 114, and generates an image of the virtual space seen from the viewpoint of the virtual space generated by the virtual space generation unit 115 (hereinafter referred to as a "virtual image"). The image generation unit 117 generates a composite image by combining the virtual image generated by the rendering execution unit 116 with a real space image acquired by the captured image acquisition unit 111 and stored in the data storage unit 113. The image generation unit 117 sends the generated composite image to the image output unit 118. Furthermore, the image generation unit 117 converts the generated composite image into a format that can be distributed over a network, and sends the converted image to the communication unit 119.

[0024] The image output unit 118 displays the composite image generated by the image generation unit 117 on the display 203 of the HMD 102. The communication unit 119 transmits the composite image generated by the image generation unit 117 to the external display device 103 via the network. The communication unit 119 receives data such as second viewpoint information (information indicating the supporter's focus position), which will be described later, from the external display device 103 via the network and stores the data in the data storage unit 113. The external display device 103 is a device that receives the composite image from the communication unit 119 of the information processing device 101 by connecting to the network and displays the received composite image. Note that in the first embodiment, a desktop personal computer will be described as the external display device 103. However, the external display device 103 is not limited to a desktop personal computer and may be, for example, a laptop personal computer or a mobile terminal such as a smartphone or tablet.

[0025] [Use cases of information processing systems] A use case of the information processing system 100 will be described below with reference to FIG. 4. FIG. 4 is a diagram showing an example of use of the HMD 102 and the external display device 103. As shown in FIG. 4, a user 401 (first user) wears the HMD 102 and performs work training or the like while viewing a displayed MR image 402 (XR image). Meanwhile, a supporter 403 (second user) at a remote location provides support to the user 401 in the work while viewing the MR image 402 viewed by the user 401 on the external display device 103. At this time, the user 401 performs the work while gazing at a partial area 405 of a virtual object 404 in the MR image 402 displayed on the HMD 102. Meanwhile, the supporter 403 provides support to the user 401 in the work while gazing at a partial area 407 of a virtual object 406 in the MR image 402 displayed on the external display device 103. The MR image 402 is a composite image generated by the image generating unit 117 of the information processing device 101 as described above.

[0026] [Operation flow of information processing device] An operation flow of the information processing device 101 will be described below with reference to FIGS. 5 and 6. FIG. 5 is a flowchart showing the operation of the information processing device 101. FIG. 6 is a diagram showing an example of foveated rendering in an MR image 402. Each process (control method of the information processing device) shown in the flowchart of FIG. 5 is realized by the CPU 301 (computer) loading a program read from the ROM 303, the HDD 305, etc. into the RAM 304 and executing the program. When the flowchart of FIG. 5 starts, first, in step S500, the CPU 301 acquires a real space image using the captured image acquisition unit 111 and stores the acquired real space image in the data storage unit 113. In step S501, the CPU 301 acquires virtual space information of each of the virtual objects 404 and 406 from the data storage unit 113.

[0027] In step S502, the CPU 301 (first detection means) detects the gaze of the user 401 using the gaze detection unit 112. Furthermore, the gaze detection unit 112 calculates the position on the MR image 402 at which the user 401 is gazing, based on the detected gaze of the user 401. In this way, the CPU 301 (first information acquisition means) acquires the coordinates of the gaze point of the user 401 as first viewpoint information (600 in FIG. 6) (first information) using the gaze detection unit 112 (first information acquisition step). Therefore, if the user 401 gazes at a position of interest on the MR image 402, the information processing device 101 can acquire the coordinates of that position as first viewpoint information 600. Note that the first viewpoint information 600 is information related to the gaze points of both eyes of the user 401. However, to avoid redundant explanation and simplify the explanation, the following explanation of the process will be given for one eye. Therefore, in the following description, it is assumed that the same processing is performed on information about either the left or right eye of user 401. Furthermore, CPU 301 may obtain first viewpoint information 600 using gaze detection unit 112 based on data prepared in advance, such as a correspondence table, instead of calculating and obtaining it from the gaze as described above.

[0028] In step S503, CPU 301 (second determination means) determines whether virtual objects 404, 406 are included in the first viewpoint region (601 in FIG. 6). CPU 301 makes this determination by comparing the virtual space information acquired in step S501 with first viewpoint information 600 acquired in step S502. As shown in FIG. 6, first viewpoint region 601 is a circular region (first circular region) centered on first viewpoint information 600. Information that determines first viewpoint region 601, such as the radius or diameter value (first distance) of the circle that forms first viewpoint region 601, may be a value set by user 401 or the like, or may be a value set in advance.

[0029] Furthermore, the outer peripheral region 602 of the first viewpoint region 601 is a region obtained by excluding the first viewpoint region 601 from a larger circular region than the first viewpoint region 601, and is a circular region centered on the first viewpoint information 600, that is, a circular region concentric with the first viewpoint region 601. Information that determines the outer peripheral region 602 (first circular region), for example, the radius or diameter value (third distance) of the circle that forms the outer edge of the outer peripheral region 602, may be a value set by the user 401 or the like, or may be a value set in advance. Note that there may be multiple outer peripheral regions 602, or there may not be any outer peripheral regions 602 at all.

[0030] Returning to the description of FIG. 5, if the CPU 301 determines in step S503 that either the virtual object 404 or 406 is included in the first viewpoint region 601, the process proceeds to step S504. On the other hand, if the CPU 301 determines that neither the virtual object 404 nor 406 is included in the first viewpoint region 601, the process proceeds to step S509, which will be described later. In step S504, the CPU 301 (first determination means) determines whether or not second viewpoint information (610 in FIG. 6) (second information) has been acquired from the external display device 103 via the communication unit 119 (determination step). Note that the CPU 301 makes this determination by determining whether or not the second viewpoint information 610 is stored in the data storage unit 113. If the second viewpoint information 610 is stored in the data storage unit 113, the CPU 301 determines that the second viewpoint information 610 has been acquired, and the process proceeds to step S505. On the other hand, if the second viewpoint information 610 is not stored in the data storage unit 113, the CPU 301 determines that the second viewpoint information 610 has not been acquired, and the process proceeds to step S507, which will be described later.

[0031] In step S505, the CPU 301 (setting means) sets a first viewpoint region 601 and an outer peripheral region 602 of the first viewpoint region 601 using the first viewpoint information 600 (setting step). The first viewpoint region 601 and the outer peripheral region 602 are the first region in the present invention. Furthermore, the CPU 301 (setting means) sets a second viewpoint region (611 in FIG. 6) and an outer peripheral region (612 in FIG. 6) of the second viewpoint region 611 using the second viewpoint information 610 (setting step). The second viewpoint region 611 and the outer peripheral region 612 are the second region in the present invention. As shown in FIG. 6, the second viewpoint region 611 is a circular region (second circular region) centered on the second viewpoint information 610 (the focused position of the supporter 403 in the MR image 402).

[0032] Information for determining the second viewpoint region 611, for example, the radius or diameter value (second distance) of the circle forming the second viewpoint region 611, may be a value set by the supporter 403 or the like, or may be a value set in advance. Furthermore, the outer peripheral region 612 of the second viewpoint region 611 is a region obtained by excluding the second viewpoint region 611 from a region of a circle that is even larger than the second viewpoint region 611, and is an annular region centered on the second viewpoint information 610, that is, an annular region concentric with the second viewpoint region 611. Information for determining the outer peripheral region 612 (second annular region), for example, the radius or diameter value (fourth distance) of the circle forming the outer edge of the outer peripheral region 612, may be a value set by the supporter 403 or the like, or may be a value set in advance. Note that there may or may not be a plurality of outer peripheral regions 612.

[0033] Areas 601, 602, 611, and 612 are areas for determining the target area for foveated rendering. Foveated rendering is a technique in which the image is rendered at a higher resolution toward the center of a person's visual field and at a lower resolution toward the periphery of the visual field, and is a technology that has the effect of significantly reducing the load on hardware. In the first embodiment, a typical method for determining the resolution in foveated rendering will be described below, but any method for determining the size of the area, the level of resolution, etc. may be used.

[0034] In foveated rendering, for example, the resolution of the first viewpoint region 601 is set to a high resolution by the CPU 301. The resolution of the outer peripheral region 602 is set to a lower resolution than that of the first viewpoint region 601 by the CPU 301. The resolution of the second viewpoint region 611 and the resolution of the outer peripheral region 612 are also set by the CPU 301 in a similar manner. The resolution of the first viewpoint region 601 and the resolution of the outer peripheral region 602 may be preset values, or the user 401 or the like may be able to set arbitrary values. The resolution of the second viewpoint region 611 and the resolution of the outer peripheral region 612 may also be preset values, or the supporter 403 or the like may be able to set arbitrary values. Furthermore, the resolution of an out-of-field region 620, which is a region other than the regions 601, 602, 611, and 612, is set by the CPU 301 to an even lower resolution than that of the outer peripheral regions 602 and 612. The resolution of the out-of-field area 620 may also be set in advance, or may be set to any value by the user 401 or the like.

[0035] Returning to the description of Fig. 5, in step S506, CPU 301 (rendering means) applies foveated rendering based on the resolution of the set areas 601, 602, 611, 612, and 620 to perform image rendering processing (rendering step). When the rendering processing is completed, the process proceeds to step S512, which will be described later.

[0036] In step S507, the CPU 301 sets a first viewpoint region 601 and a peripheral region 602 of the first viewpoint region 601, similar to step S505. Furthermore, the CPU 301 sets the resolution of the first viewpoint region 601, the resolution of the peripheral region 602, and the resolution of the out-of-field region 620, similar to step S505. However, in step S507, because the second viewpoint information 610 does not exist, the CPU 301 does not set a second viewpoint region 611 and a peripheral region 612 of the second viewpoint region 611, unlike step S505. Therefore, the CPU 301 does not set the resolution of the second viewpoint region 611 or the peripheral region 612. In step S508, the CPU 301 applies foveated rendering based on the resolutions of the set regions 601, 602, and 620 to perform image rendering processing. When the rendering processing is completed, the process proceeds to step S512, which will be described later.

[0037] In step S509, similarly to step S504, CPU 301 determines whether or not second viewpoint information 610 has been acquired. If CPU 301 determines that second viewpoint information 610 has been acquired, the process proceeds to step S510. On the other hand, if CPU 301 determines that second viewpoint information 610 has not been acquired, the process proceeds to step S512, which will be described later.

[0038] In step S510, CPU 301 sets a second viewpoint region 611 and a peripheral region 612 of the second viewpoint region 611, similar to step S505. Furthermore, CPU 301 sets the resolution of second viewpoint region 611, the resolution of peripheral region 612, and the resolution of out-of-field region 620, similar to step S505. However, unlike step S505, in step S510 CPU 301 does not set a first viewpoint region 601 and a peripheral region 602 of the first viewpoint region 601. Therefore, CPU 301 does not set the resolution of first viewpoint region 601 or the peripheral region 602 either.

[0039] As a result, when the user 401 is not gazing at either the virtual objects 404 or 406, the information processing device 101 does not provide the first viewpoint area 601 and the outer peripheral area 602 that are targets of foveated rendering, thereby reducing the rendering processing load. In step S511, the CPU 301 applies foveated rendering based on the resolution of the set areas 611, 612, and 620, and performs image rendering processing. When the rendering processing is completed, the process proceeds to step S512. Note that the CPU 301 performs each of the processes in steps S503 to S511 described above using the processing determination unit 114, the virtual space generation unit 115, or the rendering execution unit 116.

[0040] In step S512, the CPU 301 (generation means) generates a composite image (i.e., the MR image 402) using the image generation unit 117 (generation process). At this time, the image generation unit 117 generates the composite image by combining the real space image acquired in step S500 with the virtual image rendered in step S506, step S508, or step S511. Furthermore, the image generation unit 117 performs compression and conversion processing of the composite image so that the communication unit 119 can transmit an image that can be displayed on the external display device 103. Meanwhile, the CPU 301 outputs the composite image to the HMD 102 using the image output unit 118. In step S513, the CPU 301 (transmission means) transmits the composite image compressed and converted in step S512 to the external display device 103 using the communication unit 119 (transmission process). Thereafter, the flowchart of FIG. 5 ends.

[0041] [External display device configuration and operation flow] Fig. 7 is a schematic block diagram showing the functional configuration of the external display device 103. As shown in Fig. 7, the external display device 103 has, as its hardware configuration, a main body 701, a display device 702, and an input device 703. As its functional configuration, the main body 701 has a communication unit 711, a storage unit 712, a determination unit 713, a display unit 714, and an input reception unit 715. The display device 702 is a monitor such as an organic EL display or a liquid crystal display. The input device 703 is composed of a keyboard, a mouse, etc.

[0042] The communication unit 711 receives a composite image, i.e., an MR image 402, transmitted from the information processing device 101 via the network. The communication unit 711 transmits the second viewpoint information 610 determined by the determination unit 713 to the information processing device 101 via the network. The storage unit 712 stores the MR image 402 received by the communication unit 711. The determination unit 713 performs a determination process using the MR image 402 stored in the storage unit 712 and input information (e.g., the second viewpoint information 610) notified from the input receiving unit 715. Details of the determination process will be described later. The display unit 714 displays the MR image 402 on the display device 702. The input receiving unit 715 receives input information from the input device 703 and notifies the determination unit 713 of the received input information.

[0043] FIG. 8 is a schematic block diagram showing the hardware configuration of the external display device 103. As shown in FIG. 8, the external display device 103 has a hardware configuration including a CPU 801, a ROM 802, a RAM 803, and a HDD 804. The CPU 801, the ROM 802, the RAM 803, and the HDD 804 are communicatively connected via a system bus 805. The CPU 801 controls the components connected to the system bus 805. The CPU 801 executes programs stored in the ROM 802, the HDD 804, and the like to realize, for example, the above-described functional configurations and the processing of the flowcharts described below. The ROM 802 has a storage area for storing programs executed by the CPU 801, etc. The RAM 803 has a storage area that functions as a work area when the CPU 801 loads and executes programs. The HDD 804 has a storage area for storing various programs required for the CPU 801 to execute processing and various data including data required for processing described below.

[0044] The operation flow of the external display device 103 will be described below with reference to FIG. 9. FIG. 9 is a flowchart showing the operation when the external display device 103 receives the composite image (i.e., the MR image 402) transmitted in step S513. Each process shown in the flowchart in FIG. 9 is realized by the CPU 801 loading a program read from the ROM 802, HDD 804, etc. into the RAM 803 and executing the program. When the flowchart in FIG. 9 starts, first, in step S900, the CPU 801 (receiving means) receives the composite image (hereinafter referred to as the "MR image 402") from the information processing device 101 via the communication unit 711. Furthermore, the CPU 801 (display means) stores the MR image 402 in the storage unit 712 and displays the MR image 402 on the display device 702 using the display unit 714.

[0045] As a result, as shown in FIG. 4 , the supporter 403 can view the same MR image 402 that the user 401 views on the external display device 103. Here, in the MR image 402, the user 401 is gazing at a partial region 405 of the virtual object 404, so the partial region 405 has high resolution. Note that the partial region 405 corresponds to the first viewpoint region 601 and the outer peripheral region 602. There is no problem if the supporter 403 also gazes at the partial region 405 of the virtual object 404, just like the user 401. However, there are cases where the supporter 403 wants to view in detail a region different from the region the user 401 is gazing at. Therefore, hereinafter, a case will be described in which the supporter 403 wants to view in detail the partial region 407 of the virtual object 406 in the MR image 402, as described above.

[0046] Returning to the description of FIG. 9, in step S901, the CPU 801 determines, via the determination unit 713, whether or not the second viewpoint information 610 has been input via the input device 703. This determination is made to enable the supporter 403 to visually recognize the partial region 407 of the virtual object 406 in detail. For this purpose, the CPU 801 (second identification means) (second information acquisition means) first identifies the position selected by the supporter 403 using the input device 703, and receives input information about the position via the input receiving unit 715 and notifies the determination unit 713. Furthermore, the CPU 801 determines, via the determination unit 713, the input information notified from the input receiving unit 715 and the MR image 402 recorded in the storage unit 712, and determines whether or not the position selection made by the supporter 403 was made with respect to the MR image 402. That is, the determining unit 713 determines whether the position selected by the supporter 403 is selected on the MR image 402, thereby determining whether the second viewpoint information 610 has been input by the input device 703.

[0047] If the CPU 801 determines via the determination unit 713 that the supporter 403 selected a position on the MR image 402, i.e., that the second viewpoint information 610 was input via the input device 703, the process proceeds to step S902. On the other hand, if the CPU 801 determines via the determination unit 713 that the supporter 403 did not select a position on the MR image 402, i.e., that the second viewpoint information 610 was not input via the input device 703, the process of the flowchart in FIG. 9 ends. In step S902, the CPU 801 (information transmitting means) transmits, via the communication unit 711, the input information notified from the input receiving unit 715 to the information processing device 101 as second viewpoint information 610 indicating the position of interest of the supporter 403. Thereafter, the process of the flowchart in FIG. 9 ends. Specifically, the second viewpoint information 610 is information about the coordinates of the position selected by the supporter 403 in the MR image 402. Thereafter, the CPU 801 repeatedly executes the processes in the flowchart of FIG.

[0048] [summary] As described above, the supporter 403 can use the input device 703 to specify the area that the supporter 403 wants to view in detail in the MR image 402 in which the area gazed by the user 401 has been increased in resolution, and can view the MR image 402 in which the area has been increased in resolution. In this way, the information processing device 101 can improve the visibility of the supporter 403 for the MR image 402 shared with the user 401 while suppressing the processing load of rendering.

[0049] Moreover, unlike the conventional technology, the support person 403 does not need to prompt the user 401 to view the area that the support person wants to view in detail, and therefore can efficiently support the user 401 in his / her work. Furthermore, the communication load of images between the information processing device 101 and the external display device 103 can be reduced. Note that these points are also true in the second and third embodiments described later.

[0050] Second Embodiment The second embodiment will be described below with reference to FIGS. 10 and 11. Note that in the second embodiment, only the differences from the first embodiment will be described. FIG. 10 is a diagram showing an example of use of an HMD 102 and an external display device 103. As shown in FIG. 10, the external display device 103 has an external camera 1000 in addition to the configuration of the first embodiment. The external camera 1000 can capture an image of the supporter 403. In the external display device 103, a CPU 801 (second detection means) can detect the line of sight of the supporter 403 by image processing the image captured by the external camera 1000.

[0051] The operation flow of the external display device 103 will be described below with reference to Fig. 11. Fig. 11 is a flowchart showing the operation of the external display device 103 when it receives the composite image (i.e., the MR image 402) transmitted in step S513. Each process shown in the flowchart in Fig. 11 is realized by the CPU 801 loading a program read from the ROM 802, HDD 804, etc. into the RAM 803 and executing the program. The flowchart in Fig. 11 will also be used in the description of a third embodiment, which will be described later.

[0052] Step S1100 is similar to step S900 described above, and therefore a detailed description thereof will be omitted. In step S1101, the CPU 801 determines, via the determination unit 713, whether or not the second viewpoint information 610 has been detected. This determination is made by determining whether or not the supporter 403 is gazing at the MR image 402, based on the line of sight of the supporter 403 detected from the image captured by the external camera 1000. If the CPU 801 determines, via the determination unit 713, that the supporter 403 is gazing at the MR image 402, that is, that the second viewpoint information 610 has been detected, the process proceeds to step S1102. On the other hand, if the CPU 801 determines, via the determination unit 713, that the supporter 403 is not gazing at the MR image 402, that is, that the second viewpoint information 610 has not been detected, the flowchart of FIG. 11 ends.

[0053] In step S1102, the CPU 801 (information transmitting means) transmits the second viewpoint information 610 to the information processing device 101 via the communication unit 711. Thereafter, the flowchart in FIG. 11 ends. Specifically, the second viewpoint information 610 is information about the coordinates of the position where the supporter 403 is gazing in the MR image 402. Therefore, the CPU 801 (second information acquiring means) acquires the coordinates of the position where the supporter 403 is gazing from the detected line of sight of the supporter 403 as the second viewpoint information 610 indicating the position of focus of the supporter 403. Thereafter, the CPU 801 repeatedly executes each process in the flowchart in FIG. 11.

[0054] As described above, the supporter 403 can view the MR image 402 in which the area gazed by the user 401 has been increased in resolution by detecting the area that the supporter 403 wants to view in detail using the external camera 1000. In this way, the information processing device 101 can improve the visibility of the supporter 403 in the MR image 402 shared with the user 401 while suppressing the processing load of rendering.

[0055] Third Embodiment The third embodiment will be described below with reference to FIGS. 11 and 12. Note that in the third embodiment, only the differences from the first embodiment will be described. FIG. 12 is a diagram showing an example of using the HMD 102 and the external display device 103. As shown in FIG. 12, in the third embodiment, an HMD worn by the supporter 403 is used as the external display device 103. The external display device 103 (i.e., the HMD worn by the supporter 403) has the above-mentioned eye camera 202 and gaze detection unit 112. This allows the CPU 801 (second detection means) of the external display device 103 to detect the gaze of the supporter 403. Note that the HMD worn by the supporter 403 may be of a video see-through type or a fully immersive type. The operation flow of the external display device 103 will be described below with reference to FIG. 11, which was also used in the description of the second embodiment. Step S1100 is similar to step S900 described above, and therefore a detailed description thereof will be omitted.

[0056] In step S1101, the CPU 801 determines, via the determination unit 713, whether or not the second viewpoint information 610 has been detected. This determination is made by determining whether or not the supporter 403 is gazing at the MR image 402, based on the line of sight of the supporter 403 detected by the external display device 103 (i.e., the HMD worn by the supporter 403). If the CPU 801 determines, via the determination unit 713, that the supporter 403 is gazing at the MR image 402, that is, that the second viewpoint information 610 has been detected, the process proceeds to step S1102. On the other hand, if the CPU 801 determines, via the determination unit 713, that the supporter 403 is not gazing at the MR image 402, that is, that the second viewpoint information 610 has not been detected, the flowchart in FIG. 11 ends.

[0057] In step S1102, the CPU 801 (information transmitting means) transmits the second viewpoint information 610 to the information processing device 101 via the communication unit 711. Thereafter, the flowchart in FIG. 11 ends. Specifically, the second viewpoint information 610 is information about the coordinates of the position where the supporter 403 is gazing in the MR image 402. Therefore, the CPU 801 (second information acquiring means) acquires the coordinates of the position where the supporter 403 is gazing from the detected line of sight of the supporter 403 as the second viewpoint information 610 indicating the position of focus of the supporter 403. Thereafter, the CPU 801 repeatedly executes each process in the flowchart in FIG. 11.

[0058] As a result, the supporter 403 can detect, on the external display device 103 (i.e., the HMD worn by the supporter 403), the area that the supporter 403 wants to view in detail in the MR image 402 in which the area that the user 401 is gazing at has been increased in resolution. Furthermore, the supporter 403 can view the MR image 402 in which the area that the supporter wants to view in detail has been increased in resolution. In this way, the information processing device 101 can improve the visibility of the supporter 403 for the MR image 402 shared with the user 401 while suppressing the processing load of rendering.

[0059] <Other> Although preferred embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and various modifications and alterations are possible within the scope of the present invention. For example, in the information processing device 101 of each embodiment, the CPU 301 acquires the first viewpoint information 600 based on the line of sight of the user 401 with respect to the MR image 402 (step S502). However, the present invention is not limited to this. For example, the CPU 301 (first identification means) may identify a position selected by the user 401 in the MR image 402, and the CPU 301 (first information acquisition means) may acquire the first viewpoint information 600 based on the position selected by the user 401 in the MR image 402. In this way, if the user 401 selects a position of interest in the MR image 402, the information processing device 101 can acquire the coordinates of that position as the first viewpoint information 600. Note that, although not shown, configurations for the user 401 to select a position in the MR image 402 include, for example, an input device of the information processing device 101, a controller of the HMD 102, hand tracking, etc.

[0060] Furthermore, in the information processing device 101 of each embodiment, the CPU 301 (first change means) may change information determining the first viewpoint region 601 (for example, the radius or diameter of a circle forming the first viewpoint region 601) in response to an instruction from the user 401 or the like. This allows the user 401 or the like to change the size of the first viewpoint region 601 whose resolution has been increased in the MR image 402. Furthermore, the CPU 301 (first change means) may change information determining the second viewpoint region 611 (for example, the radius or diameter of a circle forming the second viewpoint region 611) in response to an instruction from the supporter 403 or the like. This allows the supporter 403 or the like to change the size of the second viewpoint region 611 whose resolution has been increased in the MR image 402.

[0061] Although not shown, configurations for the user 401 to make a designation include, for example, an input device of the information processing device 101, a controller of the HMD 102, hand tracking, etc. Furthermore, configurations for the supporter 403 to make a designation include, for example, an input device 703 of the external display device 103 in the first and second embodiments. Furthermore, although not shown in the drawings, configurations for the supporter 403 to make a designation include, for example, a controller of the HMD that is the external display device 103, hand tracking, etc. This point is similar to the configuration when the supporter 403 wearing the HMD of the third embodiment selects a position in the MR image 402.

[0062] Furthermore, in the information processing device 101 of each embodiment, the CPU 301 (second change means) may change information determining the outer peripheral region 602 of the first viewpoint region 601 (for example, the radius or diameter of a circle forming the outer edge of the outer peripheral region 602) in response to a command from the user 401 or the like. This allows the user 401 or the like to change the size of the outer peripheral region 602, the resolution of which has been increased subsequent to that of the first viewpoint region 601 in the MR image 402. Furthermore, the CPU 301 (second change means) may change information determining the outer peripheral region 612 of the second viewpoint region 611 (for example, the radius or diameter of a circle forming the outer edge of the outer peripheral region 612) in response to a command from the supporter 403 or the like. This allows the supporter 403 or the like to change the size of the outer peripheral region 612, the resolution of which has been increased subsequent to that of the second viewpoint region 611 in the MR image 402.

[0063] Furthermore, in step S502 of each embodiment, the CPU 301 (third determination means) of the information processing device 101 may determine whether the first viewpoint information 600 has been acquired. If the CPU 301 determines that the first viewpoint information 600 has been acquired, the process proceeds to step S503. On the other hand, if the CPU 301 determines that the first viewpoint information 600 has not been acquired, the process proceeds to step S509. As a result, when the user 401 is not gazing at the MR image 402, the information processing device 101 can reduce the processing load of rendering by not providing the first viewpoint region 601 and the outer peripheral region 602 that are targets of foveated rendering. This also applies to the case where the user 401 has not selected a position in the MR image 402.

[0064] In each embodiment, the first viewpoint region 601 and the second viewpoint region 611 are circular, but are not limited to this and may be polygonal, for example. Also, the outer peripheral region 602 of the first viewpoint region 601 and the outer peripheral region 612 of the second viewpoint region 611 are annular, but are not limited to this and may be polygonal, for example.

[0065] In each embodiment, the viewpoint information for increasing the resolution of the MR image 402 is two, the first viewpoint information 600 of the user 401 and the second viewpoint information 610 of the supporter 403, but it may be three or more. In this case, foveated rendering is a rendering process that increases the resolution of an area according to the number of viewpoint information.

[0066] In addition, in each embodiment, the images displayed on the HMD 102 and the external display device 103 are MR images in which real space images and virtual images are combined, but this is not limited thereto and may be, for example, AR images. Furthermore, the images displayed on the HMD 102 and the external display device 103 may be VR images consisting of virtual images only. In this case, real space images are not necessary, so the captured image acquisition unit 111 and the external camera 201 can be omitted. Therefore, the HMD 102 may be a fully immersive type.

[0067] The present invention can also be realized by a process in which a program that realizes one or more functions of each of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a dedicated processor (e.g., a circuit such as an ASIC or FPGA) that realizes one or more functions. Furthermore, the present invention can also be realized by a combination of a general-purpose processor and a dedicated processor. Note that the term "processor" refers to a processor in a broad sense and includes both general-purpose processors and dedicated processors. Furthermore, the operations of a processor may not only be performed by a single processor, but may also be performed in cooperation with multiple processors located at physically separate locations.

[0068] The disclosure of each embodiment includes the following configurations, methods, programs, and systems. (Configuration 1) An information processing device that shares XR images with an external display device, a first information acquisition means for acquiring first information indicating a position of interest of a first user in the XR image being displayed on a display; a first determination means for determining whether or not the information processing device has acquired second information indicating a second user's focus position in the XR image being displayed on the external display device from the external display device; a setting means for setting a first area and a second area based on the first information and the second information when the information processing device acquires the second information from the external display device; a rendering means for rendering the first region and the second region at a higher resolution than a region different from the first region and the second region to generate a virtual image; a generating means for generating the XR image using the virtual image; and a transmission means for transmitting the XR image to the external display device. (Configuration 2) A first detection means for detecting a line of sight of the first user with respect to the XR image is provided, 2. The information processing device according to configuration 1, wherein the first information acquisition means acquires the first information based on the line of sight of the first user. (Configuration 3) A first specifying means for specifying a position selected by the first user in the XR image is provided, 3. The information processing device according to configuration 1 or 2, wherein the first information acquisition means acquires the first information based on a position selected by the first user. (Configuration 4) The setting means is setting the first area using the first information and the first distance; 4. The information processing device according to any one of configurations 1 to 3, wherein the second area is set using the second information and the second distance. (Configuration 5) The information processing device according to configuration 4, further comprising first change means for enabling the first distance or the second distance to be changed in response to a designation from the first user or the second user. (Configuration 6) The setting means is A first circular area extending from the first user's focused position to the first distance is set as the first area; 6. The information processing device according to configuration 4 or 5, wherein a second circular area extending from the second user's focused position to the second distance is set as the second area. (Configuration 7) The setting means is In addition to the first circular area, a first annular area between a circle at a third distance from the first user's focused position and the first circular area is set as the first area; The information processing device described in configuration 6 is characterized in that in addition to the second circular area, a second annular area between a circle at a fourth distance from the second user's focus position and the second circular area is set as the second area. (Configuration 8) The information processing device according to configuration 7, further comprising second change means for enabling the third distance or the fourth distance to be changed in response to a designation from the first user or the second user. (Configuration 9) The information processing device described in Configuration 7 or 8, characterized in that the generation means generates the virtual image by rendering the first circular region and the second circular region at a higher resolution than the first annular region and the second annular region. (Configuration 10) A second determination means is provided for determining whether or not a virtual object is included in the first area, 10. The information processing device according to any one of configurations 1 to 9, wherein the setting means does not set the first area when the first area does not include a virtual object. (Configuration 11) A third determination means is provided for determining whether the first information acquisition means has acquired the first information, 10. The information processing device according to any one of configurations 1 to 9, wherein the setting means does not set the first area if the first information acquisition means fails to acquire the first information. (Configuration 12) The information processing device according to any one of configurations 1 to 11, wherein the information processing device is an HMD (Head Mounted Display) equipped with the display. (Method 1) A control method for an information processing device that shares an XR image with an external display device, comprising: a first information acquisition step of acquiring first information indicating a position of interest of a first user in the XR image being displayed on a display; a determination step of determining whether the information processing device has acquired second information indicating a focus position of a second user in the XR image being displayed on the external display device from the external display device; a setting step of setting a first area and a second area based on the first information and the second information when the information processing device acquires the second information from the external display device; a rendering step of rendering the first region and the second region at a higher resolution than a region different from the first region and the second region to generate a virtual image; generating the XR image using the virtual image; a transmission step of transmitting the XR image to the external display device. (Program 1) A program for causing a computer to execute each means of the information processing device according to any one of configurations 1 to 12. (System 1) An information processing device according to any one of configurations 1 to 12; an information processing system comprising the external display device; (Configuration 13) The XR image is shared with the information processing device according to any one of configurations 1 to 12, a receiving means for receiving the XR image from the information processing device; a display means for displaying the XR image; second information acquisition means for acquiring the second information; and an information transmitting unit that transmits the second information to the information processing device. (Configuration 14) A second detection means for detecting a line of sight of the second user with respect to the XR image is provided, 14. The external display device according to configuration 13, wherein the second information acquisition means acquires the second information based on the line of sight of the second user. (Configuration 15) A second specifying means for specifying a position selected by the second user in the XR image is provided, 15. The external display device according to configuration 13 or 14, wherein the second information acquisition means acquires the second information based on a position selected by the second user. [Explanation of symbols]

[0069] 101 Information processing equipment 103 External display device 203 Display 301 CPU (first information acquisition means) (first determination means) (setting means) (rendering means) (generation means) (transmission means) 401 User (First User) 402 MR images (XR images) 403 Supporter (secondary user) 600 First Viewpoint Information (First Information) 601 First Viewpoint Area (First Area) 602 Outer area (first area) 610 Second Viewpoint Information (Second Information) 611 Second Viewpoint Area (Second Area) 612 Outer area (second area)

Claims

1. An information processing device that shares XR images with an external display device, a first information acquisition means for acquiring first information indicating a position of interest of a first user in the XR image being displayed on a display; a first determination means for determining whether the information processing device has acquired second information indicating a focus position of a second user in the XR image being displayed on the external display device from the external display device; a setting unit that sets a first area and a second area based on the first information and the second information when the information processing device acquires the second information from the external display device; a rendering means for rendering the first region and the second region at a higher resolution than a region different from the first region and the second region to generate a virtual image; a generating means for generating the XR image using the virtual image; and a transmitting means for transmitting the XR image to the external display device.

2. a first detection means for detecting a line of sight of the first user with respect to the XR image; The information processing apparatus according to claim 1 , wherein the first information acquisition means acquires the first information based on a line of sight of the first user.

3. a first specifying means for specifying a position selected by the first user in the X-ray image; The information processing apparatus according to claim 1 , wherein the first information acquisition means acquires the first information based on a position selected by the first user.

4. The setting means setting the first area using the first information and the first distance; The information processing apparatus according to claim 1 , wherein the second area is set using the second information and the second distance.

5. 5. The information processing apparatus according to claim 4, further comprising a first change unit that enables the first distance or the second distance to be changed in response to a designation from the first user or the second user.

6. The setting means A first circular area extending from the first user's focused position to the first distance is set as the first area; The information processing apparatus according to claim 4 , wherein a second circular area extending from the second user's focused position to the second distance is set as the second area.

7. The setting means In addition to the first circular area, a first annular area between a circle at a third distance from the first user's focused position and the first circular area is set as the first area; 7. The information processing device according to claim 6, wherein in addition to the second circular area, a second annular area between a circle at a fourth distance from the second user's focus position and the second circular area is set as the second area.

8. 8. The information processing apparatus according to claim 7, further comprising a second change unit that enables the third distance or the fourth distance to be changed in response to a designation from the first user or the second user.

9. The information processing apparatus according to claim 7 , wherein the generating means generates the virtual image by rendering the first circular region and the second circular region at a higher resolution than the first annular region and the second annular region.

10. a second determination means for determining whether a virtual object is included in the first area; The information processing apparatus according to claim 1 , wherein the setting means does not set the first area when the first area does not include a virtual object.

11. a third determination means for determining whether the first information acquisition means has acquired the first information; 2. The information processing apparatus according to claim 1, wherein the setting means does not set the first area when the first information acquisition means fails to acquire the first information.

12. 2. The information processing apparatus according to claim 1, wherein the information processing apparatus is a head mounted display (HMD) having the display.

13. A control method for an information processing device that shares an XR image with an external display device, comprising: a first information acquiring step of acquiring first information indicating a position of interest of a first user in the XR image being displayed on a display; a determination step of determining whether the information processing device has acquired second information indicating a focus position of a second user in the XR image being displayed on the external display device from the external display device; a setting step of setting a first area and a second area based on the first information and the second information when the information processing device acquires the second information from the external display device; a rendering step of rendering the first region and the second region at a higher resolution than a region different from the first region and the second region to generate a virtual image; generating the X-ray image using the virtual image; a transmitting step of transmitting the XR image to the external display device.

14. 2. A program for causing a computer to execute each means of the information processing apparatus according to claim 1.

15. The information processing device according to claim 1 ; an information processing system comprising the external display device;

16. The X-ray image is shared with the information processing device according to claim 1, a receiving means for receiving the X-ray image from the information processing device; a display means for displaying the X-ray image; second information acquisition means for acquiring the second information; and an information transmitting unit that transmits the second information to the information processing device.

17. a second detection means for detecting a line of sight of the second user with respect to the XR image; The external display device according to claim 16 , wherein the second information acquisition means acquires the second information based on a line of sight of the second user.

18. a second specifying means for specifying a position selected by the second user in the X-ray image; 17. The external display device according to claim 16, wherein the second information acquisition means acquires the second information based on a position selected by the second user.

Images