Virtual reality system and head-mounted display used therein

Abstract

The purpose of the present invention is to provide an imaging function in virtual space taking privacy protection measures into consideration in a non-anonymous virtual reality system.　In order to realize the abovementioned purpose, provided is a virtual reality system comprising a server that provides a virtual reality service, a head-mounted display that receives the virtual reality service, and a network connecting the server and the head-mounted display, wherein as user information, the server holds: first object data that is used to generate a first avatar image for display; second object data that is used to generate a second avatar image for imaging; and an attribute for being imaged that sets the imaging conditions when the user is imaged by another user, the server transmits first object data or second object data to the head-mounted display according to the attribute for being imaged, and the head-mounted display generates and displays a first avatar image or a second avatar image from the received first object data or second object data.

Description

Technical Field

[0001] The present invention relates to a virtual reality system that provides a photographing function considering the privacy protection of participants in the virtual reality system, and a head-mounted display used therefor.

Background Art

[0002] There is a virtual reality system that provides background data and object data in a virtual space to a participant (hereinafter also referred to as a user) in a virtual reality system wearing a head-mounted display (hereinafter referred to as HMD: Head Mounted Display) to experience virtual reality (VR: Virtual Reality).

[0003] Furthermore, as virtual reality systems, there are a virtual reality system where the virtual space is artificial like in games, etc., and the user uses an avatar image created with a nickname or CG (Computer Graphics) and has anonymity, and a virtual reality system where the virtual space mimics the real space, and the user generally uses an avatar image with their real name or an avatar image that allows self-association. Examples of the latter virtual space include a virtual administrative space where procedures that require identity verification are carried out while receiving guidance from a clerk at a window such as a government office, a virtual school space where interactions between students can be experienced during classes and breaks, a virtual social space that enables recognition of the other party, a virtual tourism space experienced with friends, etc.

[0004] In the real space, camera photography is generally carried out due to the spread of smartphones, but distributing a photo in which an individual who has not consented to the photography is captured in the net space has legal problems, and this has a certain deterrent effect on privacy protection. Also, by blurring or mosaicking the part in which an individual who has not consented to the photography is captured, it is made distributable.

[0005] In non-anonymous virtual reality systems, the ability to take photos within the virtual space creates an experience that feels as if the user is in the real world. Therefore, even in non-anonymous virtual reality systems, measures to protect privacy are necessary, just as in the real world.

[0006] An example of privacy protection in a virtual reality system is described in Patent Document 1. Patent Document 1 describes a system that detects a specific situation in which a predetermined specific action of an avatar corresponding to a first user placed in a virtual space is performed, which should be hidden from a second user who is different from the first user. When the specific situation is detected, a content image representing the virtual space in a manner in which the specific action cannot be seen is displayed on the second user's user terminal. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2021-56884 [Overview of the project] [Problems that the invention aims to solve]

[0008] The purpose of privacy protection in Patent Document 1 is to prevent other users from seeing specific actions that should be hidden, such as when a user enters protected information like a password, and does not take into consideration the shooting function that is the subject of the present invention.

[0009] This invention has been made in view of the above points, and its purpose is to provide a virtual space photography function that takes privacy protection measures into consideration in a non-anonymous virtual reality system. [Means for solving the problem]

[0010] The present invention, to give one example, is a virtual reality system comprising a server that provides virtual reality services, a head-mounted display that receives virtual reality services, and a network connecting the server and the head-mounted display, wherein the server holds, as user information, first object data that generates a first avatar image for display, second object data that generates a second avatar image for photography, and a subject attribute that sets the conditions for photography when the user is photographed by another user, the server transmits the first object data or the second object data to the head-mounted display according to the subject attribute, and the head-mounted display generates and displays the first avatar image or the second avatar image from the received first object data or the second object data. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide a virtual reality system that does not use anonymity and offers a virtual space photography function that takes into consideration the protection of user privacy. [Brief explanation of the drawing]

[0012] [Figure 1] This is a system configuration diagram of the virtual reality system in Example 1. [Figure 2] This is an external view of the HMD in Example 1. [Figure 3] This is a functional block diagram of the HMD in Example 1. [Figure 4] This is a hardware block diagram of the HMD in Example 1. [Figure 5] This is a sequence diagram between the HMD and the virtual reality service server in Example 1. [Figure 6] This diagram illustrates the virtual space of the HMD and the user's field of view in Example 1. [Figure 7] This is a display image for the HMD in Example 1, in which a captured image is superimposed on a portion of the display image. [Figure 8]This is a diagram showing an example of a virtual space to be photographed by the HMD in Example 1. [Figure 9A] This is an example of the display of a captured image of the HMD when the use of captured object data that enables identification of the user in Example 1 is permitted. [Figure 9B] This is an example of the display of a captured image of the HMD when the use of captured object data that enables identification of the user in Example 1 is not permitted. [Figure 9C] This is an example of the display of a captured image of the HMD when photography is not permitted in Example 1. [Figure 10] This is the user attribute table managed by the virtual reality service server in Example 1. [Figure 11] This is the virtual reality processing flowchart of the virtual reality service program of the HMD in Example 1. [Figure 12] This is the sequence diagram between the HMD and the virtual service providing server in Example 2. [Figure 13] This is the virtual reality processing flowchart of the virtual reality service program of the HMD in Example 2. [Figure 14] This is the virtual reality processing flowchart of the virtual reality service program of the HMD in Example 3.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described while referring to the drawings.

Examples

[0014] Figure 1 is a system configuration diagram of the virtual reality system in this embodiment. In Figure 1, 100 is the virtual reality service server (hereinafter sometimes referred to as Server 100), 200 is the network, 300 is an access point, 1 is an HMD, and 1A is a user. In Figure 1, access point 300 is installed at site A, and other access points are also installed at sites B and C. These access points have equivalent functions, and through each access point, users can receive virtual reality services from the virtual reality service server 100 via network 200 from different sites. User 1A is at site A wearing HMD 1, but there are other users (the numbers are omitted) who can also receive virtual reality services simultaneously.

[0015] Figure 2 is an external view of the HMD in this embodiment. In Figure 2, the HMD1 includes a camera 10, a distance measuring unit 11, a pair of left and right image projection units 12a and 12b, a screen 13, a group of position and motion sensors 14, a control unit 15, a pair of left and right speakers 16a and 16b, a microphone 17, and mounting parts 18a and 18b.

[0016] The user of HMD1 (the user of the virtual reality system) attaches HMD1 to their head using attachment parts 18a and 18b. Attachment part 18a supports the HMD at the nose area of ​​the face, and attachment part 18b secures the HMD around the head.

[0017] Camera 10 captures images in front of the HMD1. The control unit 15 captures the images from camera 10 and recognizes real-world objects from those images. Furthermore, it assigns depth data obtained from the distance measuring unit 11 to the real-world objects to recognize them in three dimensions. The control unit 15 also generates a background image from the background data of the virtual space and an avatar image from the object data, which are then projected onto the screen 13 by the projection units 12a and 12b as three-dimensional images of the virtual space. The control unit 15 also creates sound to be amplified by speakers 16a and 16b.

[0018] The projection units 12a and 12b and the screen 13 constitute the display unit of the HMD1. Projection unit 12a projects the image seen by the left eye of the virtual object onto the screen 13, and projection unit 12b projects the image seen by the right eye onto the screen 13, so that the virtual object appears as if it were at a predetermined distance in real space.

[0019] Figure 3 is a functional block diagram of the HMD in this embodiment, showing the details of the internal configuration of the control unit 15. Functions identical to those in Figure 2 are denoted by the same reference numerals. Also, the projection units 12a and 12b in Figure 2 are combined to form the projection unit 12. Furthermore, the microphone, speaker, screen, etc., are omitted.

[0020] In Figure 3, 20 is the image recognition operation unit, 21 is the communication unit, 22 is the shooting tool processing unit, 23 is the position and movement processing unit, 24 is the virtual reality image processing unit, 25 is the personal data storage unit, 26 is the display processing unit, and 27 is the data storage unit.

[0021] The image recognition operation unit 20 receives camera images from the camera 10 and distance data from the distance measuring unit 11. It recognizes real objects such as the user's fingers and arms from the real space captured by the camera images and assigns depth data to the feature points of these real objects. Furthermore, it recognizes the user's intended operation from the movements of the user's fingers and hands.

[0022] The communication unit 21 downloads object data, etc., from the virtual space via the network, or reads already saved object data, etc., from a storage device (not shown).

[0023] The shooting tool processing unit 22 generates a shooting image that captures a portion of the virtual space, by providing the shooting position, orientation, and field of view, just as if operating a drone or the like in real space to take a picture from any position.

[0024] The position and motion processing unit 23 uses the position information to determine the viewpoint and the orientation information to determine the line of sight from the GPS, compass, and gyro sensor signals output by the position and motion sensor group 14.

[0025] The virtual reality image processing unit 24 generates a display image from the background image of the virtual space background data and the avatar image of the object data obtained based on the viewpoint and gaze.

[0026] The personal data storage unit 25 stores user information such as names necessary for logging into virtual reality services, as well as attributes of the person being photographed. The data storage unit 27 stores the images to be photographed.

[0027] The display processing unit 26 sends the display image generated by the virtual reality image processing unit 24, or the capture image generated by the capture tool processing unit 22, to the projection unit 12.

[0028] Figure 4 is a hardware block diagram of the HMD in this embodiment. In Figure 4, the same reference numerals are used for functions identical to those in Figures 2 and 3, and their descriptions are omitted. In Figure 4, the difference from the functional block diagram in Figure 3 is that the control unit 15 is configured as an information processing device in which the CPU and other components interpret the operation program and execute various functions through software processing. An advantage is that a general-purpose device such as a smartphone can be used as the information processing device.

[0029] In Figure 4, the control unit 15 includes a communication unit 21, a CPU 30, RAM 31, flash ROM (FROM) 32, and an interface unit 36. The interface unit 36 ​​is connected to an interface unit 37 located in the HMD main body and is also responsible for external output.

[0030] The communication unit 21 of the control unit 15 selects an appropriate processing method from several communication processes such as mobile communication (4G, 5G, etc.) and wireless LAN, and connects the HMD1 to the network. Furthermore, it downloads virtual space object data and the like from an external server. FROM 32 includes a basic program 33, a virtual reality service program 34, and a data storage unit 35 as processing programs. These processing programs are loaded into RAM 31 and executed by the CPU 30. The data storage unit 35 temporarily stores intermediate data necessary for executing the processing programs and also performs the roles of the personal data storage unit 25 and data storage unit 27 in Figure 3. FROM 32 may be a single memory medium as shown in the figure, or it may be composed of multiple memory mediums. Furthermore, it may be a non-volatile memory medium other than flash ROM.

[0031] The interfaces implemented by interface units 36 and 37 may be wired, such as USB® or HDMI®, or wireless, such as wireless LAN.

[0032] Figure 5 is a sequence diagram between the HMD and the virtual reality service server in this embodiment. The left side of the figure shows the virtual reality service server 100, and the right side shows the virtual reality processing unit of the HMD1 (hereinafter sometimes referred to as HMD).

[0033] In Figure 5, first, in step S10, the login process is initiated on the HMD1. Then, in step S11, an authentication request is sent from the HMD1 to the server 100. The authentication request includes the user's ID, password (PW), and HMD1 profile. The user's ID and password (PW) are managed by the server 100 in association with the user's real name and personal image for authentication. The HMD1 profile refers to the hardware and software capability information of the HMD1, such as the ability to distinguish between images for virtual reality display and images for capture, and the ability to output display images externally.

[0034] Next, in step S12, if the user ID and password sent from HMD1 match the information registered in server 100, HMD1 is authenticated, and server 100 issues an authentication OK confirmation.

[0035] Next, in step S13, the HMD1 issues a user attribute update. The user attributes include the attributes to be photographed, which are applied when the user is photographed, and object data for displaying or photographing the user, such as avatar images. Note that the timing of issuance is not as shown in Figure 5, but can be at any time in the sequence. After receiving the user attribute update, the server 100 uses the new user attributes.

[0036] Steps S14 to S17 are a sequence for the HMD1 to obtain display images for the virtual reality system. In step S14, the HMD1 sends out viewpoint parameters such as the user's position and gaze direction in the virtual space. In step S15, the server 100 extracts objects that are within the range visible to the HMD1 based on the received position and gaze direction. Furthermore, in step S16, the server 100 sends out background data and extracted object data. Background data may be sent out in advance for a wide range of background data, and in S16 only data to complement it may be sent out. In step S17, the HMD1 uses the received data to generate and display a display image for the virtual reality.

[0037] Steps S18 to S24 are the shooting sequence. In step S18, the shooting tool on the HMD1 determines the shooting parameters such as the position, direction, and field of view of the shooting location, and in step S19, these parameters are sent to the server 100.

[0038] In step S20, the server 100 extracts objects within the shooting range based on the received shooting parameters. Furthermore, in step S21, it checks the shooting attributes of other users' objects among the extracted objects. In step S22, the server 100 sends out the shooting attributes of other users' objects. The shooting attributes of other users' objects are information indicating whether other users allow shooting, or whether they allow identification of themselves during shooting. This information is registered by other users as their own settings and is recorded and managed by the server 100. In step S23, the server 100 sends out background and object data.

[0039] In step S24, HMD1 generates and saves a capture image using the captured object attributes and background and object data of the other user that it has received. When generating the capture image, if the other user's captured attributes allow identification of the person, an avatar image that allows identification of the person is obtained from the capture object data. In this case, an image including an avatar that allows identification of the person is recorded as capture data during shooting. On the other hand, if the other user's captured attributes do not allow identification of the person, an avatar image that makes it difficult to identify the person is obtained from the capture object data, and a capture image is generated. In this case, an image including an avatar that makes it difficult to identify the person is recorded as capture data during shooting. Furthermore, if the other user's captured attributes do not allow shooting, no avatar image is used. In this case, an image that does not include the other user's avatar is recorded as capture data during shooting.

[0040] Subsequently, the sequence from steps S14 to S17, in which the HMD1 obtains images for displaying virtual reality, and the sequence from steps S18 to S24, in which the images are captured, are repeatedly executed.

[0041] Figure 6 illustrates the virtual space of the HMD and the user's field of view in this embodiment. The user's field of view corresponds to the image displayed on HMD1.

[0042] In Figure 6, the virtual space P10 is wider than the user's field of view P11. The HMD1 may receive background data of the wide virtual space all at once, or it may receive it in several stages. The user's field of view P11 is determined by the user's position and gaze direction within the virtual space. If another user's avatar exists within the user's field of view P11, the HMD1 obtains an avatar image P12 from the other user's display object data and overlays it onto the background of the virtual space within the field of view to generate a display image. If no image capture is being performed by the HMD1, an avatar image P12 that allows for the user's identification is displayed.

[0043] Figure 7 shows the display image for the HMD in this embodiment, an example in which the capture image P14 is superimposed on a portion of the display image P13. As a method for making the user see the capture image, in addition to the method of superimposing the capture image P14 on a portion of the display image P13 shown in Figure 7, a method of switching the display image to the capture image and displaying it for a certain period of time may also be used.

[0044] Next, the display and images to be captured when taking images with HMD1 will be explained using Figures 8 and 9A, 9B, and 9C. Figure 8 is a diagram showing an example of a virtual space that is the target of HMD capture in this embodiment. In Figure 8, the user's viewing range P11 exists within the virtual space P10. Within the viewing range P11, there are three other users' avatars P12, P20, and P21.

[0045] Figures 9A, 9B, and 9C illustrate the images captured by the HMD when shooting is performed in the state shown in Figure 8. It is assumed that the other users shown by the avatars on P20 and P21 have permitted the use of capture object data that allows for their identification. The differences in display based on the captured attributes of the other users shown by the avatar on P12 are then explained.

[0046] Figure 9A shows an example of display when the user of P12 has permitted the use of identifiable object data for photography. In this case, the avatar image P12 used for display, which allows for the user's identification, is used in the photographic image P14. Figure 9B shows an example of display when the user of P12 has not permitted the use of identifiable object data for photography. In this case, the avatar image P17 used for photography, which makes it difficult to identify the user, is used in the photographic image P14. Figure 9C shows an example of display when the user of P12 has not permitted photography. In this case, no other user's avatar image is superimposed on the photographic image P14.

[0047] Figure 10 shows the user attribute management table managed by server 100 in this embodiment. In Figure 10, the attribute items of the user attribute management table consist of user management number (USR#) T11, authentication data T12, display object data (abbreviated as display OBJ in the figure) T15, login status T16, and photographed attribute T17.

[0048] Authentication data T12 consists of name / password (Name / PW) T13 and identity verification image data T14. The photographed attribute T17 consists of the following items for photographed attributes: unconditional permission T18, permission limited to the photographer T19, object replacement instruction (abbreviated as OBJ replacement instruction in the diagram) T20, photographic object data (abbreviated as photographic OBJ in the diagram) T21, paid permission T22, and not permitted T23.

[0049] The identity verification image T14 is, for example, an encoded image of the user equivalent to an identification card issued by a public institution. Figure 10 shows an example of an identity image. The display object data T15 is data capable of generating a display avatar image with a level of detail that allows for the identification of each user, and is encoded as highly confidential data. Figure 10 shows an example of an avatar's appearance. The login status T16 indicates that the user is using the virtual reality service.

[0050] The object replacement instruction T20 has three defined states. If the value of object replacement instruction T20 is 0, the object is not replaced and the display object data T15 is used for display. If the value of object replacement instruction T20 is 1, the capture object data T21 is used. If the value of object replacement instruction T20 is 2, the corresponding object is not displayed. The capture object data T21 is used when the value of object replacement instruction T20 is 1, and is data that can generate a capture avatar image at a level where it is difficult to identify each user. Figure 10 shows that a simple humanoid character is registered.

[0051] For example, regarding a user with user management number T11 of 1, the username is A and the password is B. The photographed attribute T17 indicates that user management numbers 2 and 3 are registered in the photographer-only permission T19. Furthermore, the object replacement instruction T20 has a value of 0 for user management numbers 2 and 3, indicating that object replacement is not necessary, while the value for other users is 1, indicating that object replacement is necessary. Therefore, the display object data T15 is used for the captured image only when the photographer is user management number 2 or 3. In other words, when user management numbers 2 or 3 take the picture, an avatar that can be recognized as user management number 1 is recorded. For users other than management numbers 2 and 3, the object replacement instruction value is 1, and photography is possible by replacing it with the photograph object data T21. Therefore, when a user other than management numbers 2 or 3 takes the picture, an avatar that makes it difficult to identify the person is recorded.

[0052] For a user with username C and user management number T11 of 2, object replacement is not required for users with user management numbers 1 and 3. Therefore, when a user with user management numbers 1 or 3 takes a photo, an avatar that can be recognized as user 2 is recorded. For other users, since the value of object replacement instruction T20 is 2, the corresponding object is not displayed. Therefore, when a user other than user management numbers 1 or 3 takes a photo, the avatar of user 2 is not recorded.

[0053] For a user with username E and user management number T11 of 3, the photographer-only permission setting is not enabled, and photography is permitted if the object data T21 is replaced. Therefore, regardless of which user takes the photo, an avatar that makes it difficult to identify the person will be recorded.

[0054] For a user with username G and user management number T11 of 4, object replacement is required for users with user management numbers 1 and 3. Therefore, when users with user management numbers 1 and 3 take a photo, an avatar that makes it difficult to identify the user is recorded. For other users, the value of object replacement instruction T20 is 2, so the corresponding object is not displayed. Therefore, when a user other than management number 1 or 3 takes a photo, the avatar of user with user management number 4 is not recorded.

[0055] For a user with username "I" and user management number T11 of 5, the "Photographed" attribute T17 is set to "Not Allowed" (T23), meaning that photography is not permitted. Therefore, regardless of which user takes the photograph, the avatar of user management number 5 will not be recorded.

[0056] For a user with username K and user management number T11 of 6, the "photographed attribute T17" has "unconditional permission T18" set, meaning that photography is unconditionally permitted. In other words, regardless of which user takes the photo, an avatar that can be recognized as the user with user management number 6 will be recorded.

[0057] For a user with username M and user management number T11 of 7, the paid permission T22 is set in the subject attribute T17. In this case, by paying $1.2 as shown in Figure 10, it is possible to record an avatar that can be identified as the user with user management number 7. If no payment is made, since the value of the object replacement instruction is 1, it is possible to take a picture by replacing it with the object data T21 for taking a picture.

[0058] Furthermore, in Figure 10, permission / prohibition for filming may be set for specific locations within the metaverse. In addition, permission for filming may be defined not only by user ID such as name, but also by relationship, such as being registered as a friend. Moreover, recording conditions may be linked to avatars using NFTs (Non-Fungible Tokens).

[0059] Figure 11 is a virtual reality processing flowchart of the HMD's virtual reality service program 34 in this embodiment. In Figure 11, the same processes as in Figure 5 are denoted by the same reference numerals, and their explanations are omitted.

[0060] In Figure 11, processing starts in step S50, and login authentication is performed in step S51. In step S13, data for updating user attributes is sent to server 100. Note that step S13 does not necessarily have to be performed at this timing.

[0061] In step S14, viewpoint parameters are transmitted, and in step S54, background and object data are received based on the position information. In step S17, an avatar image is calculated from the object data as a virtual reality image. If there are multiple object data, an avatar image is calculated for all of them. Furthermore, a display image is generated from the background image and avatar image and displayed. Steps S14 to S17 are the process for creating the display image.

[0062] In step S56, it is determined whether shooting is in progress. If shooting is not in progress (NO), the process returns to step S14 and the display image process is repeated. If shooting is to be performed (YES), shooting parameters such as the shooting position are transmitted in step S19. Subsequently, in step S58, the target attributes are received for objects within the shooting range based on the shooting parameters, and in step S59, the background and object data are received. The received object data is display object data if shooting is permitted based on the target attributes, and shooting object data if replacement shooting of the object is permitted based on the target attributes. If the object data to be received has already been received in the display image process, the reception of object data may be omitted and temporarily stored object data may be used.

[0063] In step S24, an avatar image is calculated from the object data. If there are multiple object data, an avatar image is calculated for all of them. Furthermore, a capture image is generated from the background image and the avatar image and saved. Steps S19 to S24 are the process for creating the capture image.

[0064] In step S61, the program confirms whether to continue. If it continues (YES), it returns to step S14; otherwise, it terminates in step S62.

[0065] As described above, the virtual reality system in this embodiment consists of an HMD implementing virtual reality processing, a virtual reality service server, and a network. When a user is wearing the HMD and experiencing virtual reality, and wants to take a picture of a part of the virtual space, the user takes the picture using the virtual reality processing's shooting tool. At this time, the HMD's virtual reality processing transmits the shooting parameters to the virtual reality service server, and if another user's object is within the shooting range, the virtual reality service server transmits attributes related to permission to photograph the other user's object to the HMD. If permission to photograph is not granted, the HMD applies an avatar image that makes it difficult to identify personal information to the other user's object and generates an image for photographing the virtual space.

[0066] Furthermore, the HMD in this embodiment consists of a control unit that performs virtual reality processing, a communication unit, a position sensor unit, a display unit, and an image recognition operation unit. It may also include a data storage unit and an external output unit. The communication unit is connected to a network and communicates with a virtual reality service server via the network. Information from the position sensor unit is transmitted to the virtual reality service server, and the virtual reality service server receives virtual space background data based on the user's current position, gaze direction, etc., and object data of other user objects present within the user's field of view. Furthermore, the control unit generates a virtual reality display image using an avatar image that allows for user identification and displays it on the display unit. The image recognition operation unit may consist of, for example, a camera unit and an image recognition unit. The camera unit, which photographs the front of the HMD, photographs the user's hand movements, and the image recognition unit recognizes them to identify the user's operations. When the user wants to photograph a part of the virtual space, the control unit's photography tool is used. The photography tool is similar to drone photography in the real world. The user can take photographs as if they were in the real world. For example, this could be a snapshot of friends experiencing the virtual reality that serves as the background provided by the virtual space. The captured image is stored in the data storage unit or output to an external device from the external output unit. In this case, other users' avatar images may appear in the background. The control unit notifies the virtual reality service provider server that it is in shooting mode by communicating shooting parameters such as shooting position, direction, and field of view. The control unit obtains attributes from the virtual reality service server regarding permission to photograph other user objects that may appear in the shooting range. If permission to photograph is not granted, it generates a captured image using an avatar image that makes it difficult to identify the user.

[0067] As explained above, this embodiment makes it possible to provide a virtual space photography function that takes into consideration the protection of user privacy in a non-anonymous virtual reality system. [Examples]

[0068] This embodiment describes an example that addresses cases where the HMD does not have the ability to distinguish between images for virtual reality display and images for capturing, or where the display image is output externally. In this embodiment as well, the configuration of HMD1 shown in Figures 2, 3, and 4 applies.

[0069] Figure 12 is a sequence diagram between the HMD and the virtual reality service server in this embodiment. In Figure 12, components identical to those in Figure 5 are denoted by the same reference numerals, and redundant explanations are omitted.

[0070] In Figure 12, steps S14 to S17 are the same sequence as in Figure 5 of Embodiment 1, for HMD1 to obtain display images for the virtual reality system. After the login process, HMD1 sends a status flag to the server 100 in step S30. The status flag is a shooting notification information indicating whether HMD1 is in a non-shooting state or a shooting state. Normally, immediately after user login, HMD1 sends a value indicating a non-shooting state because the user has not started shooting. Also, if HMD1 does not have the ability to distinguish between virtual reality display images and shooting images, or does not have the ability to stop the external output of display images, it must send a value indicating a shooting state.

[0071] In step S31, the server 100 uses a status flag to determine the shooting status of the HMD1. If the status flag received by the server 100 is a value indicating a non-shooting state, in step S16, it sends the background and object data of the virtual reality object to the HMD1. In this case, in step S17, the HMD1 uses the received object data to generate and display an image for virtual reality. What is sent from the server 100 in step S16 is normal display object data that is not intended to be photographed, and in step S17, the display avatar is displayed. This completes the process when the status flag in step S30 is a value indicating a non-shooting state.

[0072] On the other hand, if the HMD1 transmits a value indicating the shooting status using a status flag in step S30, the server 100 determines in step S31 that the status flag is a value indicating the shooting status. In this case, the server 100 skips steps S16 through S20 and proceeds to step S23.

[0073] In HMD1, steps S18 to S33 constitute the image capture sequence. Step S18 determines the image capture parameters for HMD1, and step S19 transmits these parameters to the server 100. Furthermore, step S32 transmits the status flag of HMD1 to the server 100. This status flag is a value indicating the image capture status.

[0074] In step S20, when the server 100 receives the shooting parameters and a status flag indicating the shooting state, it extracts objects within the shooting range based on the shooting parameters. Alternatively, the HMD1 may not explicitly send the status flag, and the server 100 may process step S20 by considering the transmission and reception of the shooting parameters as the shooting state.

[0075] Next, in step S23, the server 100 sends out the extracted background and object data. In S17, the HMD1 generates and displays the virtual reality image as a capture image, and then in step S33, saves or outputs the virtual reality image as a capture image.

[0076] If the transmission of the status flag in step S30 is not confirmed, the server 100 considers the HMD1 to be in a shooting state and processes accordingly. That is, it selects the object data to send to the HMD1 according to the captured attribute T17 in Figure 10. Therefore, unless unconditional permission T18 is set, the server does not send the display object data T15 for other users in the virtual space to the HMD1, thereby protecting the user's privacy. For external output, since the capabilities of the connected external device are unknown, the protection of other users' privacy is the same as for captured images.

[0077] Figure 13 is a virtual reality processing flowchart of the HMD's virtual reality service program 34 in this embodiment. In Figure 13, the same reference numerals are used for steps that are the same as in Figure 11, and redundant explanations are omitted.

[0078] In Figure 13, step S56 determines the shooting status. If shooting is enabled, step S32 sends a value indicating the shooting status using a status flag, and step S19 sends shooting parameters such as the shooting position. If the shooting status in S56 is not shooting, step S30 sends a value indicating the not-shooting status using a status flag. Then, step S59 receives object data. The object data received in step S59 is sent from server 100 based on the status flag sent in step S32 or S30. For example, if the shooting attribute is enabled, it is display object data for generating a display avatar image that allows identification of the person. If the shooting attribute allows object replacement shooting, it is shooting object data for generating a shooting avatar image that does not allow identification of the person.

[0079] In step S73, an avatar image is generated from object data. In step S74, the generated virtual reality image becomes a display image, a capture image, and an external output image that is displayed or output externally.

[0080] As explained above, according to this embodiment, even in a non-anonymous virtual reality system where the HMD lacks the ability to distinguish between images for display and images for capturing virtual reality, or when outputting display images externally, sending a status flag in advance allows for the provision of a virtual space capture function that takes user privacy into consideration. Furthermore, unless the server receives a status flag, the avatar will be displayed according to the capture attributes of other users' objects, thus ensuring privacy protection. [Examples]

[0081] Figure 14 is a virtual reality processing flowchart of the HMD's virtual reality service program 34 in this embodiment. The configuration of HMD1 in Figures 2, 3, and 4 also applies to this embodiment. Furthermore, in Figure 14, the same reference numerals are used for steps identical to those in Figure 11, and redundant explanations are omitted.

[0082] In Figure 14, the step S17 for generating the display image in Figure 11 is replaced by steps S80 to S83. In step S80, a virtual reality image is generated, but before displaying the generated virtual reality image as a display image in step S83, step S81 checks whether there is a notification that the user is being photographed. If the user is being photographed in step S81 (YES), in step S82 a notification mark, which indicates that the user is being photographed, is superimposed on the virtual reality image. The notification mark can be a colored marker, such as red, and should make the user aware that they are being photographed. The virtual reality image with the notification mark superimposed is displayed as a display image in step S83.

[0083] Furthermore, methods other than notification marks may be used for notification. For example, since a user's avatar's hands are visible to the user, the avatar's hands may be displayed differently than usual. Examples of display changes include making the visible parts of the hands glow, changing their color, or making them semi-transparent.

[0084] As described above, this embodiment provides a virtual reality system HMD that offers a virtual space photography function while considering the protection of user privacy. Furthermore, it has the effect of allowing the subject to easily recognize that they are being photographed, just as in the real world.

[0085] Although embodiments have been described above, the present invention is not limited to the embodiments described above and includes various modifications. For example, in the embodiments described above, it was explained that a CPU and the like interpret an operating program and execute various functions through software processing, but some or all of the above configuration may be made up of hardware, or hardware and software may be used in combination. Furthermore, the embodiments described above are described in detail in order to explain the present invention in an easy-to-understand manner and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace some of the configurations of one embodiment with the configurations of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace some of the configurations of each embodiment with other configurations. [Explanation of Symbols]

[0086] 1: Head-mounted display (HMD), 1A: User, 100: Virtual reality service server (Server), 200: Network, 300: Access point, P10: Virtual space, P11: Viewing range, P12, P17, P20, P21: Avatar, P13: Display image, P14: Capture image, T11: User attribute data, T12: Authentication data, T15: Display object data, T17: Attributes of the object being captured, T21: Capture object data Data, 10: Camera, 11: Distance measuring unit, 12, 12a, 12b: Projection unit, 13: Screen, 14: Sensor group, 15: Control unit, 20: Image recognition operation unit, 21: Communication unit, 22: Shooting tool processing unit, 23: Position and movement processing unit, 24: Virtual reality image processing unit, 25: Personal data storage unit, 26: Display processing unit, 27: Data storage unit, 30: CPU, 32: FROM, 34: Virtual reality service program, 35: Data storage unit, 36: Interface unit.

Claims

1. A virtual reality system comprising a server that provides virtual reality services, a head-mounted display that receives virtual reality services, and a network connecting the server and the head-mounted display, The server holds, as user information, first object data for generating a first avatar image for display, second object data for generating a second avatar image for photography, and a photographed attribute that sets the conditions for when the user is photographed by another user. The server transmits the first object data or the second object data to the head-mounted display according to the subject attributes. The head-mounted display is characterized by generating and displaying the first avatar image or the second avatar image from the received first object data or the second object data.

2. A virtual reality system according to claim 1, A virtual reality system characterized in that the server transmits first object data for generating the first avatar image for display as display object data to the head-mounted display if the subject's attributes permit identification of the person, and transmits second object data for generating the second avatar image for shooting as shooting object data to the head-mounted display if the subject's attributes do not permit identification of the person.

3. A virtual reality system according to claim 2, When the head-mounted display captures a virtual space, it transmits to the server as capture information the user's position and gaze direction within the virtual reality space, or the position, direction, and field of view of the shooting point within the virtual reality space. The virtual reality system is characterized in that the server recognizes other users present within the shooting range based on the shooting information and sends the display object data or the shooting object data to the head-mounted display according to the shooting attributes of the other users.

4. A virtual reality system according to claim 3, The virtual reality system is characterized in that the server obtains capability information from the head-mounted display, further recognizes other users within its field of view, and sends the other users' display object data or capture object data to the head-mounted display according to the capability information and the other users' attributes of being captured.

5. A virtual reality system according to claim 4, The aforementioned capability information is characterized by being information about the capability to distinguish and generate images for display and images for capture, or the capability to output images for display externally.

6. A virtual reality system according to claim 2, The head-mounted display transmits a shooting notification to the server indicating whether or not it is capturing images in a virtual space. The virtual reality system is characterized in that the server sends the display object data or the shooting object data to the head-mounted display according to the shooting notification information and the attributes of other users being photographed.

7. A virtual reality system according to claim 6, The virtual reality system is characterized in that, if the server fails to receive the shooting notification information, it sends the object data for shooting to the head-mounted display.

8. A virtual reality system according to claim 1, The first avatar image for display is an avatar image that allows for the identification of the user. The virtual reality system is characterized in that the second avatar image used for photography is an avatar image that makes it difficult to identify the user.

9. A head-mounted display that communicates with a server providing a virtual reality service, It has a control unit that performs virtual reality processing, a communication unit, and a display unit. The communication unit receives from the server first object data or second object data corresponding to the subject attribute that sets the shooting conditions when a user is photographed by another user, The head-mounted display is characterized in that the control unit generates a first avatar image for display or a second avatar image for capture using the received first object data or the second object data, and displays it on the display unit.

10. A head-mounted display according to claim 9, The head-mounted display is characterized in that the first object data is display object data that generates the first avatar image for display when the subject's attributes permit identification of the person, and the second object data is shooting object data that generates the second avatar image for shooting when the subject's attributes do not permit identification of the person.

11. A head-mounted display according to claim 10, The head-mounted display is characterized in that, when capturing images of a virtual space, the control unit transmits to the server, via the communication unit, as capturing information, viewpoint parameters including the user's position and gaze direction within the virtual reality space, or capturing parameters including the position, direction, and field of view of the shooting point within the virtual reality space.

12. A head-mounted display according to claim 11, The control unit transmits the capability information of the head-mounted display to the server via the communication unit. A head-mounted display characterized by receiving from the server the capability information and the display object data or shooting object data of other users corresponding to the attributes of other users who are within the viewing range based on the viewpoint parameters.

13. A head-mounted display according to claim 12, The head-mounted display is characterized in that the capability information is information about the capability to distinguish and generate images for display and images for capture, or the capability to output images for display to an external source.

14. A head-mounted display according to claim 10, The control unit transmits a shooting notification information to the server via the communication unit indicating whether or not it is taking a picture of the virtual space. A head-mounted display characterized by receiving the display object data or the shooting object data of another user in accordance with the shooting notification information.

15. A head-mounted display according to claim 9, It includes a data storage unit that stores user attributes consisting of personal data and image data, The aforementioned personal data includes authentication data and photographed attributes, The attributes of the person being photographed include information on permission to photograph and information on conditional permission to photograph. The image data includes display object data and shooting object data used in conditional shooting. The head-mounted display is characterized in that the control unit transmits the user attributes to the server via the communication unit.

16. A head-mounted display according to claim 10, A head-mounted display characterized by displaying an indication of being photographed on the display image when the user is the subject of the photograph.

17. A head-mounted display according to claim 9, The first avatar image for display is an avatar image that allows for the identification of the user. The head-mounted display is characterized in that the second avatar image used for shooting is an avatar image that makes it difficult to identify the user.

Images