Image processing apparatus, head-mounted display device, and image processing method

Abstract

To create a high-resolution image with low load and without delay when a user viewing a VR image moves their line of sight.SOLUTION: An image processing apparatus has: first setting means that sets a first area that is an area where an image is drawn with a higher resolution than another area in a VR image; second setting means that sets a second area to which preparation processing is applied for drawing an image with the resolution of the first area, on the basis of user information on a first user who views the VR image and area information on an area that is set according to a condition not related to the first user; and creation means that creates the VR image on the basis of the first area and the second area.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a head-mounted display device, and an image processing method. [Background technology]

[0002] A head-mounted display (HMD) is a display device that can display 360-degree panoramic images on a screen. By wearing the HMD on the head or like goggles, users can view images of a virtual space from any viewpoint. By displaying images of a virtual space on a network-connected HMD and providing games and various events (live performances, seminars, etc.), users can experience virtual reality. Services that allow users to experience virtual reality (VR) have become widespread. VR images that represent virtual spaces include real images captured by VR cameras and virtual images generated to suit the VR environment.

[0003] Patent document 1 discloses a technology that reduces the waiting time from moving the viewpoint to displaying the image by acquiring in advance part of the data used to draw the displayed image as a reference image seen from a viewpoint set based on the user's eye height, etc. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-121394 Summary of the Invention [Problem to be solved by the invention]

[0005] Even if a reference image seen from the user's own viewpoint is acquired in advance and a high-resolution image can be generated without delay using the reference image, if the user moves their gaze to an area not viewed by the user, image generation may be delayed if the reference image is not available. Furthermore, generating a high-resolution image including an area not viewed by the user increases the processing load on the HMD.

[0006] The present invention aims to provide an image processing device that can generate high-resolution images without delay and with low load when a user viewing a VR image moves their line of sight. [Means for solving the problem]

[0007] The image processing device of the present invention is characterized by having a first setting means for setting a first area, which is an area in a VR image to be drawn at a higher resolution than other areas; a second setting means for setting a second area to which preparatory processing for drawing at the resolution of the first area is applied based on user information about a first user viewing the VR image and area information about an area set according to conditions unrelated to the first user; and a generation means for generating the VR image based on the first area and the second area. [Effects of the Invention]

[0008] According to the present invention, when a user viewing a VR image moves their line of sight, a high-resolution image can be generated without delay and with low load. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating the configuration of a VR system and the functional configuration of an HMD. [Figure 2] FIG. 1 is a diagram illustrating foveated rendering. [Figure 3] 10A and 10B are diagrams illustrating movement of the user's line of sight to an area being viewed. [Figure 4] 10A and 10B are diagrams illustrating movement of the line of sight to an area not visually recognized by the user. [Figure 5] 1 is a diagram illustrating the configuration of a VR system and the functional configuration of an HMD according to a first embodiment. [Figure 6] 10 is a flowchart illustrating a setting process of a second region. [Figure 7] 10 is a flowchart illustrating a process for determining whether to accept a sharing request. [Figure 8] FIG. 1 is a diagram illustrating the effects of the first embodiment. [Figure 9] FIG. 11 is a diagram illustrating setting of a second region in the third embodiment. [Figure 10] FIG. 10 is a diagram illustrating setting of a second region in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment 1> Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The configuration of a VR system that provides VR services and the functional configuration of an HMD will be described with reference to Figures 1(A) and 1(B). Figure 1(A) is a diagram illustrating an example of the configuration of a VR system. The VR system shown in Figure 1(A) includes an HMD 101, an access point 102, and a host server 103.

[0011] The HMD 101 worn by the user is equipped with multiple detection units and can detect the user's line of sight, etc. The access point 102 is a wireless network access point such as a Wi-Fi router, and is used to connect the HMD 101 to an external terminal via a network. The host server 103 is connected to the HMD 101 via the access point 102.

[0012] The host server 103 transmits image generation information used to generate VR images to the HMD 101. When providing a VR service in which multiple users participate, the host server 103 manages user information (user information) received from the HMD 101 worn by each user. The host server 103 assigns position information in the virtual space to each user, and transmits image generation information to each user's HMD 101 for generating a VR image in which multiple users exist in the same space. The HMD 101, which receives the image generation information from the host server 103, uses the received image generation information to generate a VR image for the user to view.

[0013] Referring to Fig. 1(B), a process for generating a VR image and displaying the generated image to a user will be described. Fig. 1(B) is a block diagram illustrating an example of the functional configuration of an HMD 101. The HMD 101 has an image generation unit 104, an information acquisition unit 105, and a display unit 108. The image generation unit 104 includes a processing unit 106 and an image drawing unit 107.

[0014] The information acquisition unit 105 acquires image generation information transmitted from the host server 103 and user information of the user wearing the HMD 101. The processing unit 106 of the image generation unit 104 performs processing to generate an image according to the user's state based on the image generation information and user information acquired by the information acquisition unit 105. For example, the processing unit 106 determines the area the user is looking at based on the distribution of the user's gaze position. The image drawing unit 107 draws and generates an image based on the processing of the processing unit 106. For example, the image drawing unit 107 draws with high precision the area determined to be the user's gaze. The display unit 108 displays the image generated by the image generation unit 104 to the user.

[0015] In the VR system shown in FIG. 1(A), the HMD 101 worn by the user receives image generation information held by the host server 103, and generates an image to be displayed to the user by the image generation unit 104. In order to provide the user with a natural VR image, the HMD 101 must be able to display high-resolution images without delay no matter which direction the user looks. It is preferable that:

[0016] To meet this demand, an image generation technology called foveated rendering is known. Foveated rendering is an image generation technology that generates a high-resolution image in the center of a displayed image and reduces the resolution of the periphery. By utilizing foveated rendering, the HMD 101 can provide the user with a high-resolution image by displaying the part the user is looking at in high resolution and reducing the resolution of other parts. In addition, by reducing the resolution of parts the user is not looking at, the HMD 101 can reduce the calculation cost of unnecessary image generation and make effective use of hardware calculation resources.

[0017] 2(A) and 2(B) are diagrams illustrating foveated rendering. FIG. 2(A) is a schematic diagram of a VR image viewed by a user wearing an HMD 101. VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera), and panoramic images with a wider image range (effective image range) than the display range that can be displayed at one time on the display unit 108. The black square mark shown in FIG. 2(A) is the detection result of the user's gaze position.

[0018] 2(A) is an area surrounded by a solid line 201 where user gaze positions are detected in a concentrated manner. Area 202 surrounded by a dashed line (excluding area 201) is an area where fewer user gaze positions are detected than area 201. Area 203 surrounded by a dashed line (excluding areas 201 and 202) is an area within the user's field of view but is hardly recognized by the user.

[0019] In foveated rendering, central vision and peripheral vision are distinguished by analyzing the distribution information of gaze positions. In the gaze position distribution shown in Figure 2(A), area 201 is the area where the user's gaze positions are detected concentratedly, and is determined to be the user's central vision. Area 202 does not have as many gaze positions detected as area 201, but gaze positions are detected at shorter distance intervals than area 203, so it is determined to be the first peripheral vision. Area 203 has gaze positions detected more dispersedly than area 202, so it is determined to be the user's second peripheral vision.

[0020] The processing unit 106 of the HMD 101 can distinguish between the central visual field, the first peripheral visual field, and the second peripheral visual field from the distribution information of the user's gaze position. The processing unit 106 transmits the information on the distinguished visual fields to the image drawing unit 107.

[0021] Fig. 2(B) is a diagram illustrating the state in which foveated rendering is applied to the panoramic image shown in Fig. 2(A). The central visual field 204 is an area determined based on area 201 in Fig. 2(A). The first peripheral visual field 205 is an area determined based on area 202 in Fig. 2(A). The second peripheral visual field 206 is an area determined based on area 203 in Fig. 2(A).

[0022] The image rendering unit 107 generates an image based on the determination result of the processing unit 106. Specifically, the image rendering unit 107 generates an image with a high resolution in the central visual field 204, generates an image with a lower resolution than the central visual field 204 in the first peripheral visual field 205, and generates an image with a lower resolution than the first peripheral visual field 205 in the second peripheral visual field 206.

[0023] By using information on the user's gaze position to determine the user's central visual field and generating images with high resolution within the central visual field, the HMD 101 can provide more immersive VR images.

[0024] Furthermore, because the processing unit 106 has previously analyzed the area visible to the user, the image rendering unit 107 can apply preparatory processing to the first peripheral field of view and the second peripheral field of view to render an image at high resolution. Therefore, even if the user moves their viewpoint from the central field of view to the first peripheral field of view or the second peripheral field of view, the image generation unit 104 can generate an image without delay. In the following description, the central field of view in which an image is rendered at high resolution is referred to as the first field of view. Furthermore, the first peripheral field of view and the second peripheral field of view to which preparatory processing to render an image at high resolution is applied are referred to as the second field of view.

[0025] The preparatory process for drawing an image at high resolution may include, for example, executing in advance a calculation process for estimating changes in color tone and changes in lighting caused by changes in angle in order to draw at high resolution. The preparatory process for drawing an image at high resolution may also include mapping the importance of the gaze position within the image based on the detection result of the user's gaze position and assigning pixel values ​​according to the importance.

[0026] Figure 2(B) shows an example in which a first area in which a high-resolution image is drawn based on the user's gaze position is separated from a second area in which preparatory processing for drawing the image at high resolution is applied. However, VR systems also provide services that assume interaction between multiple users. Services provided to multiple users include, for example, multiple users playing a game together or traveling in a virtual space together with multiple users (virtual tours).

[0027] In a service format targeted at multiple users, users may share viewpoints at unexpected times, or the host server 103 may generate an event (such as a volcanic eruption or a whale flying). When an unexpected viewpoint is shared or an event occurs, the user's gaze may move to an area that the user is not viewing. In other words, it is desirable to be prepared so that high-resolution rendering can be performed even in areas other than the area where the HMD 101 detects the gaze position.

[0028] However, when generating an image based on information about the user's gaze position, it is difficult to handle situations where the user moves their gaze to an area that is not being viewed. The movement of the user's gaze will be described with reference to Figures 3(A), 3(B), and 4. Figures 3(A) and 3(B) are diagrams illustrating the movement of the user's gaze to an area that is being viewed. Figure 4 is a diagram illustrating the movement of the user's gaze to an area that is not being viewed.

[0029] Figure 3(A) shows a situation in which multiple users are playing a VR game in the same virtual space. User 301 and user 302 are users wearing HMD 101 in the VR system described in Figure 1(A), and can virtually coexist in the same virtual space based on information from host server 103. Viewable area 303, surrounded by a solid line, is the area visible to user 301 and user 302.

[0030] FIG. 3(B) is a plan view of the virtual space shown in FIG. 3(A) as seen from arrow 300. Visual area 303 includes a first region 303a, where the gaze positions of user 301 and user 302 are concentrated, and a second region 303b, where the gaze positions are not concentrated, as determined by foveated rendering. In the example of FIG. 3(A), user 301 and user 302 are playing the VR game with a common goal in mind, so their visual areas 303 are the same. In a VR service in which multiple users are expected to view the same visual area, users typically do not move their gaze to an out-of-visual area (an area not viewed by the user) at unexpected times.

[0031] On the other hand, FIG. 4 shows a case where the user's gaze moves from the visible area 303 to the out-of-visual area. 4 is a plan view looking down on the virtual space, similar to FIG. 3(B). In FIG. 4, user 301 is viewing viewing area 303, similar to FIG. 3(B). Viewing area 303 includes first area 303a and second area 303b. User 402 is viewing viewing area 404. Viewing area 404 includes first area 404a where user 402's gaze is concentrated, and second area 404b where the gaze is not concentrated.

[0032] The VR service used by the users in Fig. 4 is, for example, a virtual tour that allows users to freely enjoy traveling in a virtual space. Unlike the VR game services illustrated in Figs. 3(A) and 3(B), the virtual tour in Fig. 4 does not have a specific event objective, so each user views a different area. In other words, the viewable area 404 viewed by user 402 is an area outside the viewable area of ​​user 301, and is not detected by the HMD 101 worn by user 301.

[0033] 4, if user 402 prompts user 301 to look at an object that user 402 is looking at (such as a tourist attraction in the virtual space or a fictitious building), user 301 will move his or her gaze to user 402's visible area 404, which is an area outside of user 301's field of view. If preparatory processing for rendering an image at high resolution is not applied to visible area 404, which is an area outside of user 301's field of view, there is a risk that image generation for user 301's visible area 404 will be delayed.

[0034] Delays in image generation can be suppressed by generating high-resolution images or applying preparatory processing for drawing images at high resolution for the entire area outside the view of the user 301. However, drawing high-resolution images or applying preparatory processing for drawing images at high resolution for the entire area outside the view of the user 301 increases the load on the hardware of the HMD 101. In the first embodiment, the HMD 101 appropriately sets a second region, out of the area outside the view of the user 301, to which preparatory processing for drawing images at high resolution is applied, so that high-resolution images can be generated without delay and with low load even where the gaze moves.

[0035] The following describes a case where a VR image is displayed on an HMD (head-mounted display device) as an example of a display device having an image processing device according to the present invention. Note that the present invention is applicable to any image processing device that can be used together with an HMD, and the image processing device may be provided in the HMD or in an electronic device separate from the HMD. The HMD can also be considered as a display device having an image processing device and a display unit.

[0036] By wearing an HMD on their head or like goggles, users can view VR images from any viewpoint. VR images displayed on an HMD include 360-degree panoramic images, real images captured by a VR camera, and virtual images generated to suit the VR environment. The following explanation assumes a situation in which a network-connected HMD displays images of a virtual space, providing users with VR services such as games and various events (live performances, seminars, etc.).

[0037] Fig. 5(A) illustrates the configuration of a VR system according to embodiment 1. Fig. 5(B) is a block diagram illustrating the functional configuration of HMDs 501A and 501B. In Figs. 5(A) and 5(B), explanations common to Figs. 1(A) and 1(B) will be omitted.

[0038] FIG. 5A shows the configuration of a VR system when user A and user B are participating in the same virtual tour. Below, we will explain the image generation process when user A moves his / her line of sight using the HMD 501A worn by user A. The HMD 501A is connected to the host server 503 via an access point 502A. The HMD 501B is connected to the host server 503 via an access point 502B. The HMD 501A and HMD 501B are also collectively referred to as HMD 501. The access point 502A and access point 502B are also collectively referred to as access point 502.

[0039] The host server 503 receives information about users wearing HMDs 501 connected to the host server 503, position information about the HMDs 501, and the like, and manages the information in an integrated manner. The host server 503 transmits information about users wearing other HMDs 501, position information about the other HMDs 501, and the like to the HMD 501. Based on the information received from the host server 503, the HMD 501 can generate an image in which multiple users appear to exist in the same virtual space.

[0040] The HMD 501 is equipped with a detection unit that can detect (acquire) user information such as the user's gaze position, words spoken by the user, the user's head movement (including the rotation speed of the user's head), the user's pulse, etc. The detection unit of the HMD 501 is, for example, an inertial sensor that detects movement, a sound collection device such as a microphone that detects sound, a sound level meter, a pulse meter that measures pulse, a biosensor that detects various types of biometric information, and a gaze detection system that detects the user's gaze.

[0041] The HMD 501 can also be operated by a controller connected by wire or wirelessly to the HMD 501. The HMD 501 also detects messages (instructions) sent by operating buttons on the controller as user information.

[0042] When multiple users participate in a virtual tour, there may be a request from other users to share their own viewing area (to have other users view it). For example, if user B viewing a VR image provided by host server 503 issues a request (hereinafter referred to as a sharing request) to have user A share his or her viewing area, user B's sharing request is sent to host server 503.

[0043] The visible area is an area designated by user B as an object to be shared (hereinafter also referred to as a designated area), and is not limited to an area that user B is viewing (an area designated by user B's line of sight), but may also be an area located in the direction designated by user B. User B can designate the designated area by voice (verbal) operation or by operating a predetermined operating member. The predetermined operating member may be an operating member provided in the HMD 501B, or may be an operating member of a controller for operating the HMD 501B.

[0044] User B can transmit a request to share the designated area by, for example, uttering words such as "Look at me," "Look at this," or "Amazing" to user A. The HMD 501B can determine that a request to share has been transmitted by recognizing the predetermined words uttered by user B as described above. The HMD 501B can also determine that a request to share has been transmitted when user B specifies an area using a controller. The HMD 501B may also measure user B's pulse rate using a detection unit and determine that a request to share has been transmitted when the pulse rate increases. In this case, the designated area can be the area that user B is viewing.

[0045] The host server 503 acquires (receives) a sharing request from user B from the HMD 501B worn by user B. The host server 503 analyzes to which position in the virtual space user B's designated area is allocated. The host server 503 transmits information about the analyzed designated area (hereinafter referred to as designated area information) to the HMD 501A worn by user A. The designated area information includes information such as the position and size of the designated area.

[0046] Based on the designated area information and user information received from the host server 503, the HMD 501A of the user A sets a second area to which preparatory processing for drawing an image at high resolution is applied.

[0047] Referring to Fig. 5(B), a process of displaying to a user an image generated based on a first area viewed by user A and a second area to which preparatory processing for rendering an image at high resolution is applied will be described. Fig. 5(B) is a block diagram illustrating an example of the functional configuration of an HMD 501 according to the first embodiment. Note that the HMDs 501 (HMD 501A, HMD 501B) connected to the host server 503 all have the configuration shown in Fig. 5(B). The information acquisition unit 505, processing unit 506, image rendering unit 507, and display unit 508 of the HMD 501 are the same as the information acquisition unit 105, processing unit 106, image rendering unit 107, and display unit 108 of the HMD 101 shown in Fig. 1(B), respectively, and therefore description thereof will be omitted.

[0048] The image generation unit 504 of the HMD 501 has a determination unit 509 in addition to a processing unit 506 and an image rendering unit 507. The determination unit 509 determines a second region based on the designated region information received from the host server 503 and user information acquired by the detection unit of the HMD 501. The determination unit 509 transmits information about the determined second region to the processing unit 506. The processing unit 506 sets the area viewed by the user (central visual field) as the first region, and sets the second region based on the information received from the determination unit 509. The image rendering unit 507 generates a VR image to be displayed on the display unit 508 based on the first region and second region set by the processing unit 506.

[0049] The process of setting the second area will be described with reference to FIG. 6. FIG. 6 shows an example of the process of setting the second area when the HMD 501A worn by user A receives a sharing request from user B. The process shown in FIG. 6 starts when a service by the VR system is started and the HMD 501A connects to the host server 503. Note that in the following description, part of the process executed by the host server 503 may be executed by the HMD 501A, and part of the process executed by the HMD 501A may be executed by the host server 503.

[0050] In step S601, the processing unit 506 of the HMD 501A sets a first region, which is a region in the VR image that is rendered at a higher resolution than other regions, based on the user information of the user A. The processing unit 506 can set the first region using a method similar to foveated rendering. For example, the processing unit 506 sets the region that the user A is viewing as the first region based on information about the gaze position of the user A included in the user information.

[0051] In step S602, host server 503 acquires user information for user B. In step S603, host server 503 analyzes the user information acquired in step S603 and generates image generation information for creating an image in which user A and user B exist in the same virtual space. Host server 503 transmits the generated image generation information to information acquisition unit 505 of HMD 501A.

[0052] In step S604, the host server 503 determines whether or not user B has requested user A to share the designated area specified by user B. If user B has requested sharing, the host server 503 acquires designated area information from user B's HMD 501B and transmits it to the information acquisition unit 505 of HMD 501A, and proceeds to step S605. If user B has not requested sharing, the host server 503 returns to step S602 and continues analyzing user B's user information.

[0053] In step S605, the determination unit 509 of the HMD 501A determines whether user A has accepted the sharing request from user B. For example, if user A faces the direction of the designated area, the determination unit 509 can determine that user A has accepted the sharing request from user B. Furthermore, if user A does not face the direction of the designated area, the determination unit 509 can determine that user A has not accepted the sharing request from user B. The determination unit 509 can determine whether user A has faced the direction of the designated area by acquiring the position and orientation of the HMD 501A using the detection unit. If user A has accepted the sharing request from user B, the process proceeds to step S606. If user A has not accepted the sharing request from user B, the process returns to step S602, and analysis of user B's user information continues.

[0054] In step S606, the determination unit 509 determines whether to prioritize the designated area based on the sharing request of user B when determining the second area. The determination unit 509 determines whether to prioritize the designated area based on the user information of user A. For example, the determination unit 509 determines whether user A has moved their line of sight based on the speed of rotation of user A's head, and prioritizes the designated area if user A has moved their line of sight. If the designated area is prioritized, the process proceeds to step S607. If the designated area is not prioritized, the process proceeds to step S608.

[0055] In step S607, the determination unit 509 determines a second region in the VR image viewed by user A based on the designated region. Note that if the rotation speed of user A's head is faster than a predetermined speed threshold, user A may immediately move his / her gaze to the designated region, and therefore the determination unit 509 determines the designated region as the second region. On the other hand, if the rotation speed of user A's head is slower than the predetermined speed threshold, user A may stop moving his / her gaze halfway to the designated region or may gaze at a region on the way to the designated region. In this case, the determination unit 509 may determine the visual recognition area of ​​user B and the section up to the point where user B's visual recognition area is reached as the second region.

[0056] In step S608, the determination unit 509 determines a second region in the VR image viewed by user A based on the user information of user A. In step S608, for example, the user information is information on the user's gaze position, and the determination unit 509 can determine the peripheral vision determined by foveated rendering as the second region. The processing unit 506 sets the second region determined by the determination unit 509 in steps S607 and S608 as a region to which preparatory processing for rendering at high resolution is applied.

[0057] In the processing of Fig. 6, the HMD 501A sets the second area based on information about the gaze position of user A viewing the VR image and area information about an area set according to conditions unrelated to user A (for example, a designated area designated by user B). The HMD 501A generates a VR image based on the first area set in step S601 and the second area set in step S607 or step S608. That is, the HMD 501A can generate a VR image by drawing the first area at high resolution and applying preparatory processing for drawing the second area at high resolution.

[0058] 7, the determination of whether or not user A accepts a request to share a designated area from user B in step S605 of Fig. 6 will be described. Fig. 7 is a flowchart illustrating an example of a process for determining whether or not to accept a sharing request.

[0059] In step S701, the information acquisition unit 505 of the HMD 501A receives a request to share the designated area from user B. For example, the information acquisition unit 505 receives designated area information via the host server 503, thereby recognizing that a request to share the designated area has been received from user B. It can be recognized.

[0060] In step S702, the information acquisition unit 505 determines whether or not user A has responded in a manner that approves the request to share the designated area from user B. The information acquisition unit 505 is only required to detect any reaction from user A to the request to share that approves the request.

[0061] Reactions by user A indicating approval of the sharing request include a change in gaze position, vocalization, head movement (rotation), operation of a controller, and change in pulse rate. The information acquisition unit 505 acquires information on user A's reaction indicating approval of the sharing request as user information. For example, the information acquisition unit 505 may analyze information on words uttered by user A to determine whether user A has responded in approval of user B's sharing request. The information acquisition unit 505 may also receive user A's reaction from a controller for operating the HMD 501A and determine whether user A has responded in approval of the sharing request. Furthermore, the information acquisition unit 505 may determine that user A has responded in approval of the sharing request when at least one of a change in gaze position, head movement, and pulse rate of user A is detected. If user A has responded in approval of the sharing request, the process proceeds to step S703. If user A has not responded in approval of the sharing request, the process returns to step S701.

[0062] In step S703, the information acquisition unit 505 determines whether or not the detection unit has detected a head rotation of user A. If a head rotation of user A is detected, the information acquisition unit 505 determines that user A has accepted user B's request to share the designated region, and proceeds to step S705. If a head rotation of user A is not detected, the process proceeds to step S704.

[0063] In step S704, the information acquisition unit 505 determines whether or not the gaze position of user A has moved to the designated area based on the sharing request of user B. If the gaze position of user A has not moved to the designated area, the process returns to step S701. If the gaze position of user A has moved to the designated area, the information acquisition unit 505 determines that user A has accepted the sharing request from user B and intends to view the designated area, and proceeds to step S705.

[0064] In step S705, the information acquisition unit 505 determines that user A accepts user B's request to share the designated area, and the process proceeds to step S606 in Fig. 6. User A's acceptance of user B's request to share the designated area is notified to the processing unit 506. User A's acceptance of user B's request to share the designated area is also notified to the host server 503 and the HMD 501B worn by user B.

[0065] FIG. 8 is a diagram illustrating the effects of embodiment 1. User 801 corresponds to user A in the flowcharts of FIGS. 6 and 7. User 802 corresponds to user B in the flowcharts of FIGS. 6 and 7. A case will be described in which user 801's line of sight moves from visible area 803 to user B's visible area 807 (an area not visible to user 801). FIG. 8 is a plan view looking down on the virtual space, similar to FIG. 3(B). In FIG. 8, user 801 is viewing visible area 803. Visible area 803 includes first area 803a and second area 803b. User 802 is viewing visible area 807. Visible area 807 is the visible area of ​​user 802 and is not visible to user 801.

[0066] User 802 transmits a request to user 801 to share his / her viewing area 807. If user 801 accepts the sharing request from user 802, the processing unit 506 of HMD 501A sets user B's viewing area 807 as the second region. Image drawing The unit 507 applies preparation processing for drawing an image at high resolution in the viewing area 807 set in the second region. Therefore, when the user 801 moves his / her line of sight to the viewing area 807 in response to a sharing request from the user 802, the image is already prepared to be drawn at high resolution, so there is no delay in image generation. Furthermore, the HMD 501A does not need to prepare to draw an image at high resolution throughout the entire virtual space where the user A exists, so the computational resources used by the HMD 501A can be reduced.

[0067] In the above-described first embodiment, the HMD 501 sets the second area based on information about the gaze position of the user viewing the VR image and area information about an area set according to conditions unrelated to the user (for example, a designated area designated by another user). By applying preparatory processing for rendering the second area at high resolution and rendering it, the HMD 501 can generate an image without delay and with a low load, even if the user's gaze moves suddenly.

[0068] <Embodiment 2> Embodiment 1 shows an example of a VR system that assumes a situation in which user A and user B are participating in the same virtual tour. Embodiment 2 shows an example of a VR system that assumes a situation in which a tour guide is present in addition to multiple users participating in the virtual tour. The configuration of the VR system according to embodiment 2 and the functional configuration of the HMD 501 are the same as the configuration shown in FIGS. 5(A) and 5(B) described in embodiment 1, and therefore a description thereof will be omitted. Below, an image generation process when user A moves his / her line of sight using the HMD 501A worn by user A will be described.

[0069] Figure 9 shows a situation in which multiple users are participating in a virtual tour in the same virtual space. User A (user 901) and user B (user 902) are users wearing HMDs 501A and 501B, respectively, in the VR system described in Figure 5(A). A tour guide 911 also exists in the virtual space.

[0070] In the second embodiment, a second region to which preparatory processing for rendering at high resolution is applied is set based on region information set according to line of sight information and pointing direction of the tour guide 911. A difference from the first embodiment is that the HMD 501A sets the second region by prioritizing the tour guide 911's viewing area or the area pointed by the tour guide 911 over the viewing areas of other users 902. This is because, in a virtual tour where the tour guide 911 is present, each user including the user 901 normally views the VR image from their own viewpoint and focuses on the direction pointed by the tour guide 911 every time the tour guide 911 gives guidance.

[0071] In this way, the tour guide 911 is given priority over the user 902 as a user who issues a request to share a designated area. That is, in the second embodiment, the user who issues a request to share a designated area is selected from a plurality of users who view the VR image based on their respective attributes. The user attributes may be any attributes that can distinguish between a general user and a user who attracts attention, such as the tour guide 911.

[0072] The host server 503 sets priorities by taking into consideration attribute information (here, whether or not the user is a tour guide) of each user and tour guide 911 participating in the virtual tour. The host server 503 transmits information on the designated area of ​​the tour guide 911, who has a higher priority than each user participating in the tour, to the HMD 501 of each user as a sharing request.

[0073] The host server 503 transmits information on the designated area of ​​the tour guide 911 and information on the designated area of ​​each user to the HMD 501 of each user, with priority information attached. For example, the HMD 501A of the user A may select which user's (including the tour guide 911) designated area to set as the second area based on the priority set by the user A, regardless of the priority set by the host server 503.

[0074] In the above-described second embodiment, the priority of each user's designated area is set based on the attribute information of each user wearing an HMD 501 connected to the host server 503. The HMD 501 sets the second area based on area information set according to conditions unrelated to the user, such as the priority of other users' designated areas. By setting the priority based on user attributes among users existing in the same virtual space, the HMD 501 can set the second area in accordance with various types of VR services. The HMD 501 applies preparatory processing for rendering in the second area at high resolution, thereby suppressing delays in the image generation processing of VR images and reducing the processing load.

[0075] The HMD 501 may set multiple areas from the area information assigned with priority information as the second area. For example, in a situation where it is expected that the user will frequently move their line of sight, the HMD 501 can effectively suppress delays in the image generation process by selecting multiple areas with higher priorities and setting them as the second area.

[0076] <Embodiment 3> Embodiment 1 shows an example of a VR system assuming a situation in which user A and user B are participating in the same virtual tour. Embodiment 2 shows an example of a VR system assuming a situation in which, in addition to multiple users participating in the virtual tour, a tour guide 911 is also present. Embodiment 3 shows an example of a VR system assuming a situation in which multiple users are participating in the same virtual tour, but the tour guide 911 is not present.

[0077] The configuration of the VR system according to the third embodiment and the functional configuration of the HMD 501 are the same as those shown in Figures 5(A) and 5(B) described in the first embodiment, and therefore will not be described here. The following describes the image generation process when the user A moves his / her line of sight using the HMD 501A worn by the user A.

[0078] 10 shows a situation in which multiple users exist in the same virtual space, including user A (user 1001). Areas 1010 to 1015 in the virtual space are areas that each user views. Area 1010 is the area that user 1001 views. Areas 1011 to 1015 are areas that user 1001 does not view (out-of-view areas).

[0079] Areas 1011 and 1012 are located closer to user 1001 than areas 1013 to 1015. The number of users looking at area 1011 is greater than the number of users looking at area 1012. That is, more gaze positions are detected in area 1011 than in area 1012.

[0080] Furthermore, areas 1013 to 1015 are located farther from user 1001 than areas 1011 and 1012. The number of users looking at area 1013 is greater than the number of users looking at areas 1014 and 1015. That is, more gaze positions are detected in area 1013 than in areas 1014 and 1015.

[0081] In the situation shown in FIG. 10, the HMD 501A can set a second area to which a preparatory process for rendering an image at high resolution is applied based on information on the distance from the user 1001 to each area and information on the degree of concentration of the gaze positions of other users relative to each area. For example, the HMD 501A can set a second area to which a preparatory process for rendering an image at high resolution is applied based on information on the distance from the user 1001 to each area and information on the degree of concentration of the gaze positions of other users relative to each area. The HMD 501A sets the second area based on information about an area that is visually recognized by a larger number of users among the areas visually recognized by the user 1001. Furthermore, the HMD 501A sets the second area based on information about an area that is closer to the area visually recognized by the user 1001 among the areas visually recognized by each of a plurality of users different from the user 1001.

[0082] A difference from the second embodiment is that the HMD 501A can set the second region based on the positional relationship with the region gazed by other users and the degree of concentration of gaze positions in that region, even if the user's attribute information has not been acquired. The host server 503 may not have received the attribute information of each user from the HMD 501 of each user immediately after the provision of the VR service starts. Even in such a case, the HMD 501A can set the second region based on the region that is closest to user A among the regions gazed by other users. Furthermore, the HMD 501A can set the second region based on the region that is gazed by other users and where the gaze positions of the other users are most concentrated among the regions gazed by other users.

[0083] In the example of FIG. 10, for the HMD 501A of the user 1001, the first priority area, which is given the highest priority, is set to area 1011, which is closer to the user 1001 and where the gazes of other users are concentrated. The second priority area, which is given the second highest priority, is set to area 1012, which is closer to the user 1001 and where the gazes of other users are not concentrated. The third priority area, which is given the third highest priority, is set to area 1013, which is farther from the user 1001 and where the gazes of other users are concentrated.

[0084] The priority of each area is set for each HMD 501 (for each user). The priority of each area may be set by the host server 503 or may be set in each HMD 501.

[0085] When the host server 503 sets the priority order, the host server 503 may transmit information about the first priority area to the HMD 501. The host server 503 may also transmit information about the first to third priority areas together with priority order information to the HMD 501. The HMD 501 can appropriately set the second area using the received area information and priority order information.

[0086] Furthermore, the HMD 501 receives information used to set the priority from the host server 503, thereby setting the priority of each area and appropriately setting the second area.

[0087] In the above-described third embodiment, the processing unit 506 of the HMD 501 sets the second area based on information about an area that is set according to conditions unrelated to the user, such as the positional relationship with an area viewed by other users and the degree of gaze concentration in that area. The image rendering unit 507 of the HMD 501 generates a VR image based on the first area, which is the user's viewing area, and the set second area. The HMD 501 applies preparatory processing for rendering the second area at high resolution, thereby suppressing delays in the image generation processing of the VR image and reducing the processing load.

[0088] <Embodiment 4> In the first to third embodiments, a second area to which a preparatory process for rendering at high resolution is applied is set based on the information of an area set according to conditions caused by other users existing in the same virtual space. The fourth embodiment assumes a situation in which the host server 503 randomly generates an event in the virtual space represented by the VR image, not caused by other users. An example of a VR system is shown below.

[0089] The user's gaze position may move not only in response to a request for sharing from another user, but also when the host server 503 generates an event in the virtual space regardless of the user's will. If the event occurs in an area outside the user's line of sight, image generation of the event location may be delayed. Events generated by the host server 503 include, for example, a volcano erupting, a whale jumping in the ocean, fireworks being launched, a shooting star appearing, an animal appearing, and a clock tower bell ringing.

[0090] The configuration of the VR system according to the fourth embodiment and the functional configuration of the HMD 501 are the same as those shown in FIGS. 5(A) and 5(B) described in the first embodiment, and therefore a description thereof will be omitted.

[0091] The host server 503 of the VR system transmits event information about an event to each user's HMD 501 in advance. The event information includes information such as the area and time in which the event will occur. Based on the event information, the HMD 501 sets a second area to which preparatory processing for rendering at high resolution is applied.

[0092] In the above-described fourth embodiment, the HMD 501 sets the second area based on information about an area where an event occurs, as information about an area set according to conditions unrelated to the user. The HMD 501 generates a VR image based on the first area, which is the user's visual recognition area, and the set second area. The HMD 501 applies preparatory processing for rendering the second area at high resolution, thereby suppressing delays in the image generation processing of the VR image and reducing the processing load.

[0093] Note that each functional unit (FIGS. 1(B) and 5(B)) of the image processing device (HMD) according to the first to fourth embodiments may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware. For example, the image processing device may have a processor and a memory in which a control program is stored. Then, the functions of at least some of the functional units of the image processing device may be realized by the processor reading and executing the control program from the memory.

[0094] The above-described embodiment is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configuration of the above-described embodiment within the scope of the gist of the present invention.Furthermore, the present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0095] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0096] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a first setting means for setting a first region in the VR image, the first region being a region to be rendered at a higher resolution than other regions; User information about a first user who views the VR image, and a second setting means for setting a second region to which a preparatory process for rendering at the resolution of the first region is applied, based on region information relating to a region set according to conditions unrelated to the first region; a generating means for generating the VR image based on the first region and the second region; 1. An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein the area information is information about an area designated by a second user who views the VR image. (Configuration 3) 3. The image processing device according to configuration 2, wherein the area designated by the second user includes an area visually recognized by the second user. (Configuration 4) The area designated by the second user is designated by a voice operation by the second user or by an operation of a predetermined operating member by the second user. 4. The image processing device according to configuration 2 or 3. (Configuration 5) The second user is selected from a plurality of users who view the VR image based on the attributes of each of the plurality of users. 5. The image processing device according to any one of configurations 2 to 4. (Configuration 6) The area information is information on an area that is visually recognized by a larger number of users among areas visually recognized by a plurality of users different from the first user. 2. The image processing device according to configuration 1, (Configuration 7) The area information is information about an area that is closer to an area that the first user is viewing, among the areas that the multiple users are viewing. 7. The image processing device according to configuration 1 or 6, (Configuration 8) The area information is information about an area where an event occurs in the virtual space represented by the VR image. 2. The image processing device according to configuration 1, (Configuration 9) The first area is set based on an area that the first user is viewing. 9. The image processing device according to any one of configurations 1 to 8. (Configuration 10) The VR image is a panoramic image. 10. The image processing device according to any one of configurations 1 to 9, wherein: (Configuration 11) The device further includes an acquisition unit for acquiring the user information. 11. The image processing device according to any one of configurations 1 to 10. (Configuration 12) The user information includes at least one of information on the user's gaze position, words, head movement, and pulse rate. 12. The image processing device according to any one of configurations 1 to 11. (Configuration 13) an image processing device according to any one of configurations 1 to 12; a display means for displaying the VR image; A head-mounted display device comprising: (method) setting a first region in the VR image that is to be rendered at a higher resolution than other regions; A step of setting a second region to which a preparatory process for rendering at the resolution of the first region is applied based on user information about a first user who views the VR image and region information about a region set according to conditions unrelated to the first user; generating the VR image based on the first region and the second region; An image processing method comprising: (program) 13. A program for causing a computer to function as each means of the image processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0097] 501A: Image processing device (HMD), 506: Processing unit, 507: Image drawing unit, 509: Determination unit

Claims

1. a first setting means for setting a first region in the VR image, the first region being a region to be rendered at a higher resolution than other regions; a second setting means for setting a second region to which a preparatory process for rendering at the resolution of the first region is applied based on user information about a first user viewing the VR image and region information about a region set according to conditions unrelated to the first user; and a generating means for generating the VR image based on the first area and the second area; 1. An image processing device comprising:

2. The image processing device according to claim 1 , wherein the area information is information about an area designated by a second user who views the VR image.

3. The image processing device according to claim 2 , wherein the area designated by the second user includes an area visually recognized by the second user.

4. The area designated by the second user is designated by a voice operation by the second user or by an operation of a predetermined operating member by the second user.

3. The image processing device according to claim 2.

5. The second user is selected from a plurality of users who view the VR image based on the attributes of each of the plurality of users.

3. The image processing device according to claim 2.

6. The area information is information on an area that is visually recognized by a larger number of users among areas visually recognized by a plurality of users different from the first user.

2. The image processing device according to claim 1, wherein:

7. The area information is information about an area that is closer to an area that the first user is viewing, among the areas that the multiple users are viewing.

2. The image processing device according to claim 1, wherein:

8. The area information is information about an area where an event occurs in the virtual space represented by the VR image.

2. The image processing device according to claim 1, wherein:

9. The first area is set based on an area that the first user is viewing.

2. The image processing device according to claim 1, wherein:

10. The VR image is a panoramic image.

2. The image processing device according to claim 1, wherein:

11. The device further includes an acquisition unit for acquiring the user information.

2. The image processing device according to claim 1, wherein:

12. The user information includes at least one of information on the user's gaze position, words, head movement, and pulse rate.

2. The image processing device according to claim 1, wherein:

13. An image processing device according to any one of claims 1 to 12; a display means for displaying the VR image; A head-mounted display device comprising:

14. setting a first region in the VR image, the first region being a region to be rendered at a higher resolution than other regions; A step of setting a second region to which a preparatory process for rendering at the resolution of the first region is applied based on user information about a first user viewing the VR image and region information about a region set according to conditions unrelated to the first user; generating the VR image based on the first region and the second region; An image processing method comprising:

15. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 12.

Images