Image processing device, head-mounted display device, image processing method, computer program product, and computer-readable storage medium

Abstract

The invention discloses an image processing apparatus, a head-mounted display apparatus, an image processing method, a computer program product, and a computer readable storage medium. The image processing apparatus includes: a first setting unit configured to set a first region which is a region rendered at a higher resolution than other regions in a VR image; a second setting unit configured to set a second region based on user information related to a first user visually recognizing the VR image and region information related to a region set according to a condition irrelevant to the first user, wherein the second region is pre-processed so as to be rendered at the resolution of the first region; and a generation unit configured to generate the VR image based on the first region and the second region.

Description

Technical Field

[0001] The present invention relates to an image processing device, a head-mounted display device, an image processing method, a computer program product, and a computer-readable storage medium. Background Art

[0002] A head-mounted display (HMD) is a device that can display 360-degree panoramic images on a display. The HMD can be worn on the user's head or by the user like goggles, allowing the user to view images in a virtual space from any viewpoint.

[0003] Services that allow users to experience virtual reality (VR) by displaying virtual space images on a networked head-mounted display (HMD) and providing games and various types of events (such as live performances and seminars) are becoming increasingly popular. VR images representing virtual spaces include both actual images captured by VR cameras and virtual images generated in VR formats.

[0004] Japanese Patent Laid-Open No. 2019-121394 discloses a technology for reducing waiting time from moving a viewpoint until display by pre-acquiring a portion of data for rendering a display image as a reference image viewed from a viewpoint set based on the height of the user's eyes, etc.

[0005] Even if a reference image is acquired in advance from the user's own viewpoint and a high-resolution image is generated without delay using the reference image, if the user moves their gaze to an area not visually recognized by the user, image generation may be delayed, resulting in the absence of the reference image. Furthermore, if the generated high-resolution image also includes an area not visually recognized by the user, the processing load on the HMD increases. Summary of the Invention

[0006] The present invention provides an image processing device that can perform low-load generation of a high-resolution image without delay when a user viewing a VR image moves his / her line of sight.

[0007] The image processing device of the present invention includes: a first setting unit, which is configured to set a first area, which is an area in a VR image rendered at a higher resolution than other areas; a second setting unit, which is configured to set a second area based on user information related to a first user who visually identifies the VR image and area information related to an area set according to conditions unrelated to the first user, wherein the second area is applied with preliminary processing so as to be rendered at the resolution of the first area; and a generation unit, which is configured to generate the VR image based on the first area and the second area.

[0008] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A is a diagram showing the configuration of a VR system;

[0010] Figure 1B is a diagram showing a functional configuration of an HMD;

[0011] Figure 2A and Figure 2B A diagram describing foveated rendering.

[0012] Figure 3A and Figure 3B A diagram used to describe the area where the user's gaze moves to visual recognition;

[0013] Figure 4 A diagram used to describe the movement of the user's gaze to an area not visually recognized;

[0014] Figure 5A is a diagram showing a configuration of a VR system according to a first embodiment;

[0015] Figure 5B is a diagram showing a functional configuration of an HMD;

[0016] Figure 6 is a flowchart illustrating the setting process of the second area;

[0017] Figure 7 is a flowchart illustrating an acceptance determination process related to a sharing request;

[0018] Figure 8 is a diagram for describing the effects of the first embodiment;

[0019] Figure 9 is a diagram for describing the arrangement of the second area in the third embodiment; and

[0020] Figure 10 : is a diagram for describing the setting of the second area in the fourth embodiment. DETAILED DESCRIPTION

[0021] First embodiment

[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Figure 1A and Figure 1B Describes the configuration of a virtual reality (VR) system and the functional configuration of a head-mounted display (HMD) that provides virtual reality (VR) services. Figure 1A This is a diagram illustrating the configuration of a VR system. Figure 1AThe VR system shown includes an HMD 101 , an access point 102 , and a host server 103 .

[0023] The HMD 101 worn by the user has multiple detection units and can detect the user's line of sight, etc. The access point 102 is a wireless network access point such as a Wi-Fi router and is used for network connection of the HMD 101 to external terminals. The host server 103 is connected to the HMD 101 via the access point 102.

[0024] The host server 103 transmits image generation information used to generate VR images to the HMD 101. When providing a VR service in which multiple users participate, the host server 103 centrally manages user information (user information) received from the HMDs 101 worn by each user. The host server 103 assigns position information in the virtual space to each user and transmits image generation information used to generate VR images in which multiple users exist in the same space to each user's HMD 101. Upon receiving the image generation information from the host server 103, the HMD 101 uses the received image generation information to generate VR images for the users to view.

[0025] Will refer to Figure 1B Describes the process of generating a VR image and displaying the generated image to a user. Figure 1B 10 is a block diagram illustrating a functional configuration of the HMD 101. The HMD 101 has an image generating unit 104, an information acquiring unit 105, and a display unit 108. The image generating unit 104 includes a processing unit 106 and an image rendering unit 107.

[0026] The information acquisition unit 105 acquires image generation information transmitted from the host server 103 and user information of the user wearing the HMD 101. The processing unit 106 of the image generation unit 104 performs processing to generate an image corresponding to the user's status based on the image generation information and user information acquired by the information acquisition unit 105. For example, the processing unit 106 identifies the area the user is viewing based on the distribution of the user's line of sight. The image rendering unit 107 renders and generates an image based on the processing performed by the processing unit 106. For example, the image rendering unit 107 renders the area identified as the area the user is viewing with high precision. The display unit 108 displays the image generated by the image generation unit 104 to the user.

[0027] exist Figure 1AIn the VR system shown, an HMD 101 worn by a user receives image generation information held by a host server 103 and generates images to be displayed to the user using an image generation unit 104. In order to provide the user with natural VR images, the HMD 101 is preferably capable of displaying high-resolution images without delay, regardless of the direction the user is looking.

[0028] To address this need, an image generation technology known as "foveated rendering" is known. Foveated rendering generates a display image at high resolution in the center of the image, while reducing the resolution of the surrounding area. The HMD 101 uses foveated rendering, which enables the user to display the portion they are viewing at high resolution while reducing the resolution of other portions, thereby providing the user with a high-resolution image. Furthermore, by reducing the resolution of portions not being viewed by the user, the HMD 101 can suppress unnecessary computational costs for image generation and efficiently utilize hardware-based computing resources.

[0029] Figure 2A and Figure 2B is a graph used to describe foveated rendering. Figure 2A 1 is a schematic diagram of a VR image visually recognized by a user wearing the HMD 101. The VR image includes an omnidirectional image (spherical image) captured by an omnidirectional camera (spherical camera), a panoramic image having a wider picture range (effective picture range) than the display range that can be displayed on the display unit 108 at one time, and the like. Figure 2A The black square mark in is the detection result of the user's gaze position.

[0030] Figure 2A In the figure, a region 201 surrounded by a solid line is a region where the user's gaze position is detected in a concentrated manner. A region 202 surrounded by a dotted line (exclusion region 201) is a region where fewer user gaze positions are detected than in region 201. A region 203 surrounded by a dashed line (exclusion regions 201 and 202) is a region that is within the user's field of view but is difficult for the user to recognize.

[0031] In foveated rendering, the central field of view and the peripheral field of view are distinguished by analyzing the distribution information of the sight position. Figure 2A In the illustrated distribution of gaze positions, zone 201 is where the user's gaze positions are detected in a concentrated manner and is classified as the user's central field of view. Zone 202 does not have as many gaze positions detected as zone 201, but gaze positions are detected at closer intervals than zone 203 and is therefore classified as the first peripheral field of view. In zone 203, gaze positions are detected more sparsely than in zone 202 and is therefore classified as the second peripheral field of view.

[0032] The processing unit 106 of the HMD 101 can distinguish the central field of view, the first peripheral field of view, and the second peripheral field of view based on the distribution information of the user's line of sight position, and transmit the information of the distinguished fields of view to the image rendering unit 107.

[0033] Figure 2B is used to describe Figure 2A The central field of view 204 is based on the state of the panoramic image after the foveation rendering. Figure 2A The first peripheral field of view 205 is based on the area 201 in the Figure 2A The second peripheral field of view 206 is based on the area 202 in the Figure 2A The area is distinguished by area 203.

[0034] The image rendering unit 107 generates an image based on the result of the differentiation performed by the processing unit 106. Specifically, the image rendering unit 107 generates a high-resolution image for the central field of view 204, generates an image with a reduced resolution compared to the central field of view 204 for the first peripheral field of view 205, and generates an image with a reduced resolution compared to the first peripheral field of view 205 for the second peripheral field of view 206.

[0035] By generating an image so that the central field of view range has high resolution and distinguishing the user's central field of view by using the user's line of sight position information, the HMD 101 can provide a more immersive VR image.

[0036] Furthermore, the area visually recognizable by the user is pre-analyzed by processing unit 106, so image rendering unit 107 can apply preliminary processing to the first and second peripheral fields of view, thereby rendering the image at high resolution. Consequently, even if the user moves their viewpoint from the central field of view to the first or second peripheral fields of view, image generation unit 104 can generate the image without delay. In the following description, the central field of view, where the image is rendered at high resolution, will be referred to as the "first area." Furthermore, the first and second peripheral fields of view, to which preliminary processing has been applied in order to render the image at high resolution, will be referred to as the "second area."

[0037] Preparatory processing for rendering images at high resolution involves, for example, performing calculations to estimate changes in color tone and lighting patterns based on angle changes, so that high-resolution rendering can be performed. Furthermore, preparatory processing for rendering images at high resolution can include mapping the importance of the user's gaze position within the image based on the results of detecting the user's gaze position, assigning pixel values based on the importance, and so on.

[0038] Figure 2BThis example shows a first area for rendering an image at high resolution and a second area for applying preparatory processing for rendering the image at high resolution, separated based on the user's line of sight. However, the VR system also provides services that are assumed to be exchanged between multiple users. Examples of services provided to multiple users include multiple users playing games together, multiple users traveling together through a virtual space (virtual roaming), and the like.

[0039] In a service format targeting multiple users, users may share viewpoints or the host server 103 may generate an event (such as a volcanic eruption or a whale leap) at unexpected timing. When a viewpoint is shared or an event occurs unexpectedly, there is a possibility that the user's line of sight will move to an area that the user has not visually recognized. In other words, it is preferable to make preparations in advance so that the HMD 101 can perform high-resolution rendering in areas other than the area where the line of sight is detected.

[0040] However, in the case of generating an image based on the user's line of sight position information, it is difficult to handle a situation where the user moves his / her line of sight to an area that he / she does not visually recognize. Figure 3A 、 Figure 3B and Figure 4 Describes the movement of the user's gaze. Figure 3A and Figure 3B A diagram used to describe the area where the line of sight moves to the user's visual recognition. Figure 4 This is a diagram used to describe that the line of sight moves to an area that is not visually recognized by the user.

[0041] Figure 3A It shows a situation where multiple users are playing VR games in the same virtual space. User 301 and user 302 are both playing VR games in the same virtual space. Figure 1A In the VR system described, users wear HMD 101 and can coexist in the same virtual space in a simulated manner based on information from host server 103. Visual recognition area 303 surrounded by a solid line is an area visually recognized by user 301 and user 302.

[0042] Figure 3B Viewed from the direction of arrow 300 Figure 3A The visual recognition area 303 includes a first area 303a and a second area 303b distinguished by gaze point rendering. In the first area 303a, the sight lines of user 301 and user 302 are concentrated, and in the second area 303b, the sight lines of user 301 and user 302 are not concentrated. Figure 3AIn the example shown in FIG, user 301 and user 302 are playing a game with a common purpose set in the VR game, so zone 303 is consistent. In VR services where multiple users are expected to view the same visual recognition zone, users generally do not move their gaze to a zone outside of visual recognition (a zone not visually recognized by the user) at unexpected timing.

[0043] In contrast, Figure 4 A case where the user's line of sight moves from the visual recognition area 303 to an area outside of visual recognition is described. Figure 4 So with Figure 3B In the same way, look down at the floor plan of the virtual space. Figure 4 In the , user 301 is being Figure 3B Visual recognition zone 303 is visually recognized in the same manner. Visual recognition zone 303 includes first area 303a and second area 303b. In addition, user 402 is visually recognizing visual recognition zone 404. Visual recognition zone 404 includes first area 404a where the user's 402 sight line is focused and second area 404b where the sight line is not focused.

[0044] It will be assumed Figure 4 For example, the VR service currently used by users is a virtual tour in which users can freely enjoy traveling in a virtual space. Figure 3A and Figure 3B The VR game service exemplified in Figure 4 The purpose of the event is not limited to the virtual tour, so each user visually recognizes a different zone. That is, the visual recognition zone 404 that the user 402 is visually recognizing is an area outside the visual recognition of the user 301 and is not detected from the HMD 101 that the user 301 is wearing.

[0045] exist Figure 4 In the state shown, when user 402 prompts user 301 to look at an object that user 402 is looking at (a tourist attraction, a virtual building, etc. in a virtual space), user 301 will move his / her gaze to user 402's visual recognition area 404, which is an area outside his / her own visual recognition. There is a concern that if preparatory processing for high-resolution rendering of an image is not applied to visual recognition area 404, which is an area outside of visual recognition, the generation of an image of user 301's visual recognition area 404 will be delayed.

[0046] Delays in image generation are suppressed by generating a high-resolution image or applying preliminary processing to render an image at high resolution for the entire area outside the visual recognition of the user 301. However, rendering a high-resolution image or applying preliminary processing to render an image at high resolution for the entire area outside the visual recognition of the user 301 increases the load on the hardware of the HMD 101. In the first embodiment, by appropriately setting the second area in the area outside the visual recognition of the user 301 to which preliminary processing for rendering an image at high resolution is applied by the HMD 101, low-load generation of a high-resolution image without delay can be achieved even to a position where the line of sight has moved.

[0047] As an example of a display device including the image processing device according to the present invention, the following description relates to a case where a VR image is displayed on an HMD (head-mounted display). Note that the present invention is applicable as long as the image processing device is usable with an HMD. The image processing device may be provided in the HMD or in an electronic device separate from the HMD. The HMD can also be understood as a display device including an image processing device and a display unit.

[0048] The user can wear the HMD on their head or like goggles, allowing them to view VR images from any viewpoint. In addition to 360-degree panoramic images, VR images displayed on the HMD include real images captured with a VR camera and virtual images generated based on VR formats. The following description assumes a scenario where images of a virtual space are displayed on an HMD connected to a network, and VR services such as games and various types of events (live performances, seminars, etc.) are provided to the user.

[0049] Figure 5A The configuration of the VR system according to the first embodiment is illustrated. Figure 5B 501A and 501B. Figure 5A and Figure 5B Zhongyu Figure 1A and Figure 1B Common description.

[0050] Figure 5AThe following illustrates the configuration of a VR system in which user A and user B are participating in the same virtual tour. The following describes image generation processing for a case in which user A shifts their line of sight using HMD 501A, which user A is wearing. HMD 501A is connected to host server 503 via access point 502A. HMD 501B, which user B is wearing, is connected to host server 503 via access point 502B. Note that HMD 501A and HMD 501B will also be collectively referred to as "HMD 501." Furthermore, access point 502A and access point 502B will also be collectively referred to as "access point 502."

[0051] The host server 503 receives information about users wearing the HMD 501 connected to the host server 503 and information about the location of the HMD 501, and centrally manages this information. The host server 503 transmits information about users wearing other HMDs 501 and information about the location of the other HMDs 501 to the HMD 501. Based on the information received from the host server 503, the HMD 501 can generate images showing multiple users simulating the presence of multiple users in the same virtual space.

[0052] The HMD 501 includes a detection unit and can detect (acquire) the user's gaze position, speech uttered by the user, the user's head movement (including the speed at which the user's head is rotated), the user's pulse, and the like as user information. The detection unit of the HMD 501 is, for example, an inertial sensor for detecting motion, a sound collecting device such as a microphone for detecting sound, a noise level meter, a pulse meter for measuring pulse, a biometric sensor for detecting various types of biometric information, and an eye gaze detection system for detecting the user's gaze.

[0053] Furthermore, the HMD 501 can be operated by a controller connected through a wired or wireless connection to the HMD 501. The HMD 501 also detects a message (instruction) issued by a button operation or the like performed on the controller as user information.

[0054] When multiple users participate in a virtual tour, there may be cases where a user requests to share his / her visual recognition area with other users (to have other users see his / her visual recognition area). For example, if user B, who is visually recognizing a VR image provided by the host server 503, issues a request to share his / her visual recognition area with user A (hereinafter referred to as a "sharing request"), user B's sharing request is transmitted to the host server 503.

[0055] Note that the visual recognition area is an area designated by user B as a shared object (hereinafter also referred to as a "designated area"), and is not limited to the area that user B is visually recognizing (the area designated by user B's line of sight), and may be an area located in the direction indicated by user B. User B may designate the designated area through voice (speech) operation or by operating a predetermined operating member. The predetermined operating member may be an operating member equipped with the HMD 501B, or may be an operating member of a controller for operating the HMD 501B.

[0056] User B can, for example, issue a sharing request for a designated area by uttering words such as "Look here," "Look at this," or "Wow" to user A. HMD 501B can determine that a sharing request has been issued by recognizing predetermined words (such as those described above) uttered by user B. Furthermore, if user B specifies an area using a controller, HMD 501B can determine that a sharing request has been issued. Furthermore, HMD 501B can use a detection unit to measure user B's pulse and determine that a sharing request has been issued when the pulse accelerates. The designated area in this case can be the area that user B is visually recognizing.

[0057] The host server 503 acquires (receives) a sharing request made by user B from the HMD 501B currently worn by user B. The host server 503 analyzes where user B's designated area is allocated in the virtual space. The host server 503 transmits information about the analyzed designated area (hereinafter referred to as "designated area information") to the HMD 501A currently worn by user A. The designated area information includes information such as the location and size of the designated area.

[0058] The HMD 501A of the user A sets the second area for applying the preliminary processing to render the image at a high resolution based on the designated area information and the user information received from the host server 503 .

[0059] Will refer to Figure 5B A process for displaying to a user an image generated based on a first region that user A is visually recognizing and a second region to which preliminary processing is applied so as to render the image at a high resolution is described. Figure 5B 501A, HMD 501B) connected to the host server 503 each has Figure 5B The information acquisition unit 505, processing unit 506, image rendering unit 507 and display unit 508 of the HMD 501 are respectively Figure 1BThe information acquisition unit 105 , the processing unit 106 , the image rendering unit 107 , and the display unit 108 of the illustrated HMD 101 are the same, and thus description thereof will be omitted.

[0060] In addition to the processing unit 506 and the image rendering unit 507, the image generation unit 504 of the HMD 501 also includes a judgment unit 509. The judgment unit 509 determines the second area based on the designated area information received from the host server 503 and the user information detected by the detection unit of the HMD 501. The judgment unit 509 transmits the information of the determined second area to the processing unit 506. The processing unit 506 sets the area (central field of view) that the user is visually recognizing as the first area, and also sets the second area based on the information received from the judgment unit 509. The image rendering unit 507 generates a VR image to be displayed on the display unit 508 based on the first area and the second area set by the processing unit 506.

[0061] The process of setting the second area will refer to Figure 6 Provide a description. Figure 6 An example of setting processing of the second area in a case where the HMD 501A worn by the user A receives a sharing request from the user B is shown. Figure 6 The illustrated processing is started by the VR system starting service and the HMD 501A connecting to the host server 503. Note that in the following description, part of the processing performed by the host server 503 may be performed by the HMD 501A, and part of the processing performed by the HMD 501A may be performed by the host server 503.

[0062] In step S601, the processing unit 506 of the HMD 501A sets a first region based on the user information of user A. This first region is to be rendered at a higher resolution than other regions in the VR image. The processing unit 506 can set the first region using the same technique as foveated rendering. For example, based on the information about the gaze position of user A included in the user information, the processing unit 506 sets the region that user A is visually recognizing as the first region.

[0063] In step S602, the host server 503 acquires user information of user B. In step S603, the host server 503 analyzes the user information acquired in step S602 and generates image generation information for creating an image in which user A and user B exist in the same virtual space. The host server 503 transmits the generated image generation information to the information acquisition unit 505 of the HMD 501A.

[0064] In step S604, the host server 503 determines whether there is a sharing request from user B for user A regarding the designated area designated by user B. If there is a sharing request from user B, the host server 503 acquires the designated area information from user B's HMD 501B and transmits it to the information acquisition unit 505 of the HMD 501A, and the flow proceeds to step S605. If there is no sharing request from user B, the host server 503 returns to step S602 and continues analyzing user B's user information.

[0065] In step S605, the determination unit 509 of the HMD 501A determines whether user A has accepted the sharing request from user B. For example, if user A has turned to face the designated area, the determination unit 509 may determine that user A has accepted the sharing request from user B. Alternatively, if user A has not turned to face the designated area, the determination unit 509 may determine that user A has not accepted the sharing request from user B. The determination unit 509 can determine whether user A has turned to face the designated area by acquiring the position and orientation of the HMD 501A through the detection unit. If user A has accepted the sharing request from user B, the process proceeds to step S606. If user A has not accepted the sharing request from user B, the process returns to step S602 and continues analyzing user B's user information.

[0066] In step S606, when determining the second area, the judgment unit 509 determines whether to prioritize the designated area based on the sharing request from user B. The judgment unit 509 determines whether to prioritize the designated area based on user A's user information. For example, the judgment unit 509 determines whether user A has moved their line of sight based on the speed of their head rotation, and prioritizes the designated area if user A has moved their line of sight. If the designated area is prioritized, the process proceeds to step S607. If the designated area is not prioritized, the process proceeds to step S608.

[0067] In step S607, the judgment unit 509 determines a second area in the VR image visually recognized by user A based on the designated area. Note that if the speed of user A's head rotation is faster than a preset speed threshold, there is a possibility that user A immediately moves their gaze to the designated area, so the judgment unit 509 determines the designated area as the second area. Conversely, if the speed of user A's head rotation is slower than the preset speed threshold, there is a possibility that user A stops moving their gaze before reaching the designated area or fixates on an area halfway toward the designated area. In this case, the judgment unit 509 may determine user B's visual recognition area and the interval until reaching user B's visual recognition area as the second area.

[0068] In step S608, the judgment unit 509 determines a second region in the VR image visually recognized by user A based on the user information of user A. In step S608, the user information may be, for example, information about the user's line of sight, and the judgment unit 509 may determine the peripheral field of view distinguished by foveated rendering as the second region. The processing unit 506 sets the second region determined by the judgment unit 509 in step S607 or S608 as the region to which the preliminary processing is applied to render the image at a high resolution.

[0069] exist Figure 6 In the processing of step S601, HMD 501A sets the second area based on the line of sight position information of user A who visually recognizes the VR image and the area information related to the area set according to the conditions unrelated to user A (for example, the designated area specified by user B). HMD 501A generates the VR image based on the first area set in step S601 and the second area set in step S607 or step S608. That is, HMD 501A can generate the VR image by rendering the first area at a high resolution and rendering the second area to which the preliminary processing is applied so as to render the image at a high resolution.

[0070] Will refer to Figure 7 Described in Figure 6 In step S605 , it is determined whether user A accepts the sharing request for the designated area from user B. Figure 7 This is a flowchart illustrating an acceptance determination process related to a sharing request.

[0071] In step S701 , the information acquisition unit 505 of the HMD 501A receives a sharing request for a designated area from user B. The information acquisition unit 505 can recognize that the sharing request for the designated area has been received from user B by receiving designated area information via the host server 503 , for example.

[0072] In step S702 , the information acquisition unit 505 determines whether user A reacts to the request for sharing the designated area from user B by approving the sharing request. It is sufficient for the information acquisition unit 505 to detect a certain reaction to the sharing request as a reaction of user A approving the sharing request.

[0073] The user's response to approving the sharing request includes changes in gaze position, utterances, head movements (rotations), controller manipulation, and pulse changes. The information acquisition unit 505 acquires information about user A's response to approving the sharing request as user information. For example, the information acquisition unit 505 may analyze user A's utterances to determine whether user A has responded by approving the sharing request from user B. Furthermore, the information acquisition unit 505 may receive user A's response from the controller used to operate the HMD 501A and determine whether there has been a response approving the sharing request. Furthermore, if at least one of a change in user A's gaze position, head movement, or pulse change is detected, the information acquisition unit 505 may determine that there has been a response approving the sharing request. If there has been a response approving the sharing request from user A, processing proceeds to step S703. If there has not been a response approving the sharing request from user A, processing returns to step S701.

[0074] In step S703, the information acquisition unit 505 determines whether the detection unit has detected the rotation of user A's head. If the rotation of user A's head has been detected, the information acquisition unit 505 determines that user A has accepted the sharing request for the designated area from user B and proceeds to step S705. If the rotation of user A's head has not been detected, the process proceeds to step S704.

[0075] In step S704, the information acquisition unit 505 determines whether user A's gaze position has moved to the designated area based on the sharing request from user B. If user A's gaze position has not moved to the designated area, the process returns to step S701. If user A's gaze position has moved to the designated area, the information acquisition unit 505 determines that user A intends to accept the sharing request from user B and visually recognize the designated area, and proceeds to step S705.

[0076] In step S705, the information acquisition unit 505 determines that user A has accepted the sharing request for the designated area from user B, and proceeds to step S706. Figure 6 The processing unit 506 is notified that user A has accepted the sharing request for the designated area from user B. The host server 503 and the HMD 501B that user B is wearing are also notified that user A has accepted the sharing request for the designated area from user B.

[0077] Figure 8 is a diagram describing the effects of the first embodiment. User 801 corresponds to Figure 6 and Figure 7 User A in the flowchart of Figure 6 and Figure 7 A case where the line of sight of the user 801 moves from the visual recognition area 803 to the visual recognition area 807 of the user 802 (an area outside the visual recognition of the user 801) will be described. Figure 8 So with Figure 3B In the same way, look down at the floor plan of the virtual space. Figure 8 , user 801 is visually recognizing visual recognition area 803. Visual recognition area 803 includes a first area 803a and a second area 803b. In addition, user 802 is visually recognizing visual recognition area 807. Visual recognition area 807 is the visual recognition area of user 802 and is an area outside the visual recognition of user 801.

[0078] User 802 issues a sharing request to user 801 regarding their visual recognition area 807. If user 801 accepts the sharing request, the processing unit 506 of the HMD 501A sets user 802's visual recognition area 807 to the second area. The image rendering unit 507 applies preparatory processing for rendering images at high resolution to visual recognition area 807, which has already been set as the second area. Therefore, when user 801 moves their gaze to visual recognition area 807 in response to the sharing request from user 802, image generation is not delayed because preparations for rendering the image at high resolution have already been made. Furthermore, HMD 501A does not need to prepare to render images at high resolution for the entire area of the virtual space where user A is present, thus reducing the computing resources required by HMD 501A.

[0079] In the first embodiment described above, the HMD 501 sets the second area based on the gaze position information of the user visually viewing the VR image and area information related to an area set based on conditions unrelated to the user (e.g., a designated area designated by another user). By applying preparatory processing for rendering at a high resolution to the second area, the HMD 501 can achieve image generation with minimal load and without delay, even if the user's gaze suddenly shifts.

[0080] Second embodiment

[0081] The first embodiment shows an example of a VR system assuming that user A and user B are participating in the same virtual tour. The second embodiment shows an example of a VR system assuming that a tour guide exists in addition to the multiple users participating in the virtual tour. The configuration of the VR system and the functional configuration of the HMD 501 according to the second embodiment are similar to those described in the first embodiment. Figure 5A and Figure 5B The illustrated configurations are the same, so description thereof will be omitted. Next, image generation processing performed by the HMD 501A worn by the user A in a case where the user A moves his / her line of sight will be described.

[0082] Figure 9 FIG. 4 shows a situation where multiple users are participating in a virtual tour in the same virtual space. Figure 5A In the VR system described in , user A (user 901) and user B (user 902) are users wearing HMD 501A and HMD 501B, respectively. In addition, a tour guide 911 exists in the virtual space.

[0083] In the second embodiment, a second area to which preliminary processing for rendering an image at a high resolution is applied is set based on area information set based on the line of sight information and the indicated direction of the tour guide 911. A difference from the first embodiment is that the HMD 501A prioritizes the visual recognition area of the tour guide 911 or the area indicated by the tour guide 911 as the second area, compared to the visual recognition area of the other user 902. This is because, in a virtual tour with the tour guide 911 present, due to its nature, users including the user 901 generally view VR images from their own free viewpoints, but each time the tour guide 911 introduces something, they focus on the direction indicated by the tour guide 911.

[0084] Therefore, as the user who issued the sharing request regarding the designated area, roaming guide 911 takes precedence over user 902. That is, in the second embodiment, the user who issued the sharing request regarding the designated area is selected from among multiple users visually recognizing the VR image based on their respective attributes. It is sufficient that the user attributes distinguish between an ordinary user and a user who attracts attention, such as roaming guide 911.

[0085] The host server 503 sets the priority order in consideration of the user participating in the virtual tour and the attribute information of the tour guide 911 (here, whether it is the attribute of the tour guide), etc. The host server 503 transmits information of a designated area of the tour guide 911 having a higher priority order than the user participating in the tour as a sharing request to the user's HMD 501.

[0086] Note that the host server 503 may transmit the information of the designated area of the roaming guide 911 and the information of the designated area of the user, along with the information of the assigned priority order, to the user's HMD 501. For example, regardless of the priority order set by the host server 503, the HMD 501A of user A may select, based on the priority order set by user A, which user's (including the roaming guide 911) designated area is to be set as the second area.

[0087] In the second embodiment described above, the priority order of designated areas for each user is set based on the attribute information of each user wearing the HMD 501 connected to the host server 503. The HMD 501 sets the second area based on the information of the area set according to conditions unrelated to the user himself (such as the priority order of designated areas of other users). By setting the priority order based on the user's attributes among users existing in the same virtual space, the HMD 501 can set the second area to support various forms of VR services. By applying preparatory processing to the second area to render the image at a high resolution, the HMD 501 can suppress the delay of the image generation process of the VR image and reduce the processing load.

[0088] Note that the HMD 501 may set multiple areas among the areas of information assigned with the priority order as the second area. For example, assuming that the user frequently moves their line of sight, the HMD 501 may effectively suppress delays in image generation processing by selecting multiple areas from areas with a higher priority order and setting these areas as the second area.

[0089] Third embodiment

[0090] The first embodiment shows an example of a VR system assuming that user A and user B are participating in the same virtual tour. The second embodiment shows an example of a VR system assuming that a tour guide 911 exists in addition to the multiple users participating in the virtual tour. The third embodiment shows an example of a VR system assuming that multiple users are participating in the same virtual tour but that the tour guide 911 does not exist.

[0091] The configuration of the VR system and the functional configuration of the HMD 501 according to the third embodiment are the same as those described in the first embodiment. Figure 5A and Figure 5B The illustrated configurations are the same, so description thereof will be omitted. Hereinafter, image generation processing performed by the HMD 501A worn by the user A in a case where the user A moves his / her line of sight will be described.

[0092] Figure 10This figure shows a situation where multiple users, other than user A (user 1001), exist in the same virtual space. Areas 1010 to 1015 are areas visually recognized by the users in the virtual space. Area 1010 is the area that user 1001 is visually recognizing. Areas 1011 to 1015 are areas that user 1001 is not visually recognizing (areas not visually recognized).

[0093] Regions 1011 and 1012 are located closer to user 1001 than regions 1013 to 1015. The number of users who see region 1011 is greater than the number of users who see region 1012. That is, a greater number of sight lines are detected in region 1011 than in region 1012.

[0094] Furthermore, compared to areas 1011 and 1012, areas 1013 to 1015 are located farther from user 1001. The number of users who see area 1013 is greater than the number of users who see areas 1014 and 1015. That is, a greater number of gaze positions are detected in area 1013 than in areas 1014 and 1015.

[0095] exist Figure 10 In the illustrated scenario, HMD 501A can set a second area to which preliminary processing is applied to render an image at a high resolution based on information about the distance from user 1001 to each area and information about the concentration of gaze positions of other users in each area. For example, HMD 501A sets the second area based on information about areas visually recognized by a large number of users among the areas visually recognized by each of multiple users other than user 1001. Furthermore, HMD 501A sets the second area based on information about areas visually recognized by each of multiple users other than user 1001 that are closer to the area currently being visually recognized by user 1001.

[0096] The difference from the second embodiment is that, even when user attribute information is not acquired, HMD 501A can set the second area based on the positional relationship with the areas visually recognized by other users and the degree of concentration of gaze positions in these areas. There are cases where the host server 503 does not receive user attribute information from each user's HMD 501 immediately after the VR service begins. Even in this case, HMD 501A can set the second area based on the area closer to user A within the areas visually recognized by other users. Furthermore, HMD 501A can set the second area based on the area where the gaze positions of other users are more concentrated within the areas visually recognized by other users.

[0097] exist Figure 10In the example shown in FIG1 , regarding the HMD 501A of the user 1001, a first priority area having a first priority is set to an area 1011 located at a position close to the user 1001 and where the sight lines of other users are focused. Furthermore, a second priority area having a second priority is set to an area 1012 located at a position close to the user 1001 and where the sight lines of other users are not focused. Furthermore, a third priority area having a third priority is set to an area 1013 located at a position far from the user 1001 and where the sight lines of other users are focused.

[0098] The priority order of each area is set (for each user) at each HMD 501. The priority order of each area may be set by the host server 503 or may be set at each HMD 501.

[0099] In the case where the host server 503 sets the priority order, it is sufficient for the host server 503 to transmit the information of the first priority area to the HMD 501. In addition, the host server 503 may transmit the information of the first to third priority areas together with the information of the priority order to the HMD 501. The HMD 501 may appropriately set the second area using the received information of the areas and the information of the priority order.

[0100] Furthermore, by receiving information for setting the priority order from the host server 503 , the HMD 501 can set the priority order of each area and can appropriately set the second area.

[0101] In the third embodiment described above, the processing unit 506 of the HMD 501 sets the second area based on information about areas set based on user-independent conditions, such as positional relationships with areas currently being visually recognized by other users and the degree of concentration of gaze positions in these areas. The image rendering unit 507 of the HMD 501 generates a VR image based on the first area, which is the user's visual recognition area, and the set second area. By applying preparatory processing to the second area to render the image at a high resolution, the HMD 501 can suppress delays in the image generation process for the VR image and reduce the processing load.

[0102] Fourth embodiment

[0103] In the first to third embodiments, the second region to which preparatory processing is applied, thereby rendering an image at high resolution, is set based on information about a region set according to conditions caused by other users existing in the same virtual space. In the fourth embodiment, an example VR system is shown in which the host server 503 randomly generates events in the virtual space represented by the VR image, rather than events caused by other users.

[0104] The user's line of sight is not limited to moving in response to other users' sharing requests; it can also move when the host server 503 generates an event in the virtual space that is unrelated to the user's intention. If the location of the event is outside the user's visual recognition area, there is a concern that image generation at the location of the event may be delayed. Examples of events generated by the host server 503 include volcanic eruptions, whales leaping in the sea, fireworks, meteors, animal appearances, and the ringing of bells on a clock tower.

[0105] The configuration of the VR system and the functional configuration of the HMD 501 according to the fourth embodiment are the same as those described in the first embodiment. Figure 5A and Figure 5B The configurations shown are the same, so their description will be omitted.

[0106] The host server 503 of the VR system transmits event information related to the event in advance to the user's HMD 501. The event information includes information such as the area and time where the event will occur. Based on the event information, the HMD 501 sets the second area to apply preparatory processing to render the image at a high resolution.

[0107] In the fourth embodiment described above, the HMD 501 sets the second area based on information about the area where the event occurred, which is set based on conditions unrelated to the user. The HMD 501 generates a VR image based on the first area, which is the user's visual recognition area, and the set second area. By applying preparatory processing to the second area to render the image at a high resolution, the HMD 501 can suppress delays in the VR image generation process and reduce the processing load.

[0108] Note that the functional units ( Figure 1B 、 Figure 5B ) may be separate hardware or may be in other forms. The functions of two or more functional units may be implemented by common hardware. Multiple functions of a functional unit may be implemented by separate hardware respectively. Two or more functions of a functional unit may be implemented by common hardware. In addition, each functional unit may be implemented by hardware or may be implemented in other ways. For example, an image processing device may have a processor and a memory storing a control program. The functions of at least some of the functional units of the image processing device may be implemented by the processor reading the control program from the memory and executing the control program.

[0109] The above embodiments are merely examples, and configurations obtained by appropriately modifying or changing the configurations of the above embodiments without departing from the spirit and scope of the present invention are also encompassed by the present invention. In addition, configurations obtained by appropriately combining the configurations of the above embodiments are also encompassed by the present invention.

[0110] According to the present invention, in a case where a user viewing a VR image moves his / her line of sight, low-load generation of a high-resolution image can be performed without delay.

[0111] Note that the various types of control described above may be processes executed by a single piece of hardware (e.g., a processor or circuit) or in other ways. Processing may be shared among multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) to control the entire device.

[0112] In addition, the above-mentioned processor is a processor in a broad sense, including general-purpose processors and special-purpose processors. Examples of general-purpose processors include central processing units (CPUs), microprocessing units (MPUs), digital signal processors (DSPs), etc. Examples of special-purpose processors include graphics processing units (GPUs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. Examples of PLDs include field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc.

[0113] Other embodiments

[0114] The embodiments of the present invention can also be implemented by the following method, that is, software (including computer program products of computer programs / instructions) that perform the functions of the above-mentioned embodiments is provided to a system or device through a network or various storage media, and a computer (central processing unit (CPU), microprocessing unit (MPU)) of the system or device reads and executes the computer program / program method.

[0115] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: a first setting unit configured to set a first area, the first area being an area in the VR image rendered at a higher resolution than other areas; a second setting unit configured to set a second area based on user information related to a first user who visually recognizes the VR image and area information related to an area set according to a condition unrelated to the first user, wherein the second area is subjected to preparatory processing so as to be rendered at a resolution of the first area; as well as A generating unit is configured to generate the VR image based on the first area and the second area.

2. The image processing apparatus according to claim 1, wherein: The region information is information of a region designated by a second user who visually recognizes the VR image.

3. The image processing apparatus according to claim 2, wherein: The area designated by the second user includes an area being visually recognized by the second user.

4. The image processing device according to claim 2 or 3, wherein: The area designated by the second user is designated by the second user through a voice operation or by the second user operating a predetermined operation member.

5. The image processing apparatus according to claim 2 or 3, wherein: The second user is selected from a plurality of users based on respective attributes of the plurality of users who visually recognize the VR image. The image processing apparatus according to claim 1 , wherein: The region information is information of a region being visually recognized by more users among regions being visually recognized by a plurality of users different from the first user.

7. The image processing apparatus according to claim 1 or 6, wherein: The region information is information of a region closer to the region visually recognized by the first user, among regions visually recognized by a plurality of users different from the first user.

8. The image processing apparatus according to claim 1, wherein: The area information is information about an area where an event will occur in the virtual space represented by the VR image.

9. The image processing apparatus according to any one of claims 1 to 3, wherein: The first area is set based on an area that the first user is visually recognizing.

10. The image processing apparatus according to any one of claims 1 to 3, wherein: The VR image is a panoramic image.

11. The image processing apparatus according to any one of claims 1 to 3, further comprising: An acquiring unit is configured to acquire the user information.

12. The image processing apparatus according to any one of claims 1 to 3, wherein: The user information includes information on at least one of a user's gaze position, speech, head movement, and pulse.

13. A head-mounted display device, comprising: The image processing device according to any one of claims 1 to 12; as well as A display configured to display the VR image.

14. An image processing method comprising the following steps: Setting a first region, the first region being a region of the VR image rendered at a higher resolution than other regions; setting a second area based on user information related to a first user who visually recognizes the VR image and area information related to an area set according to a condition unrelated to the first user, wherein the second area is subjected to preparatory processing so as to be rendered at a resolution of the first area; as well as The VR image is generated based on the first area and the second area. 15 . A computer program product comprising a program for causing a computer to execute the steps of the image processing method according to claim 14 . 16 . A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 14 .

Images