Virtual reality system and head mounted display therefor

Abstract

The purpose of the present invention is to provide, in a non-anonymous virtual reality system, an imaging function in a virtual space in consideration of privacy protection measures. In order to achieve the above purpose, a virtual reality system is configured from a server that provides a virtual reality service, a head-mounted display that accepts the provision of the virtual reality service, and a network that connects the server and the head-mounted display, the server holds, as user information, first object data for generating a first virtual image for display, second object data for generating a second virtual image for imaging, and imaging attributes that set imaging conditions when a user is imaged by another user. The server sends the first object data or the second object data to the head-mounted display according to the shot attribute, and the head-mounted display generates a first virtual image or a second virtual image according to the received first object data or the second object data and displays the first virtual image or the second virtual image.

Description

Technical Field

[0001] The present invention relates to a virtual reality system that provides a shooting function taking into account the privacy protection of participants of the virtual reality system and a head-mounted display used in the virtual reality system. Background Art

[0002] There is a virtual reality system that provides background data and object data of a virtual space to a participant (hereinafter also referred to as a user) of the virtual reality system wearing a head mounted display (hereinafter referred to as HMD: Head Mounted Display) to allow the participant to experience virtual reality (VR).

[0003] In addition, virtual reality systems include anonymous virtual reality systems, such as games, where the virtual space is an artificially constructed space and users use avatar images created using nicknames and CG (Computer Graphics); and non-anonymous virtual reality systems, where the virtual space is a space that imitates the real space and users use their real names and avatar images that can be associated with themselves in principle. Examples of the latter virtual space include virtual administrative spaces where procedures that require personal identification are performed while receiving guidance from agency staff at the window of government agencies, virtual school spaces where students can experience contact with each other during classes or breaks, virtual social spaces where each other can be identified, and virtual sightseeing spaces where one can experience experience with companions.

[0004] In the real world, camera photography is common due to the popularity of smartphones, but posting photos of individuals who do not consent to being photographed in cyberspace has legal issues, which has a certain inhibitory effect on privacy protection. In addition, it is possible to publish photos of individuals who do not consent to being photographed by blurring or mosaicing the part of the photo.

[0005] In the shooting function in the virtual space of the non-anonymous virtual reality system, the user experiences as if he is in the real space. Therefore, in the non-anonymous virtual reality system, the same privacy protection measures are required as in the real space.

[0006] An example of privacy protection in a virtual reality system is described in Patent Document 1. Patent Document 1 describes the following: a specific situation in which a predetermined specific action of an avatar corresponding to a first user arranged in a virtual space is performed and the specific action should be hidden from a second user different from the first user is detected, and when the specific situation is detected, a content image representing the virtual space is displayed on a user terminal of the second user in a form in which the specific action cannot be visually recognized.

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1: Japanese Patent Application Publication No. 2021-56884 Summary of the invention

[0010] Problems to be solved by the invention

[0011] The purpose of the privacy protection in Patent Document 1 is to prevent other users from visually recognizing a specific action of a user inputting protected information such as a password, etc., which should be hidden, and does not take into account the photographing function that is the subject of the present invention.

[0012] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a shooting function in a virtual space taking into account privacy protection measures in a non-anonymous virtual reality system.

[0013] Means for solving problems

[0014] As an example, the present invention is a virtual reality system consisting of a server that provides virtual reality services, a head-mounted display that receives the virtual reality services, and a network connecting the server and the head-mounted display. The server maintains first object data for generating a first virtual image for display, second object data for generating a second virtual image for shooting, and a subject attribute for setting shooting conditions when the user is photographed by other users as user information. The server sends the first object data or the second object data to the head-mounted display based on the photographed attributes, and the head-mounted display generates a first virtual image or a second virtual image based on the received first object data or second object data and displays it.

[0015] Effects of the Invention

[0016] According to the present invention, in a non-anonymous virtual reality system, a shooting function in a virtual space can be provided in consideration of user privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a system structure diagram of the virtual reality system in Example 1.

[0018] Figure 2 This is an external view of the HMD in Example 1.

[0019] Figure 3 This is a functional block diagram of the HMD in the first embodiment.

[0020] Figure 4 This is a hardware block diagram of the HMD in Example 1.

[0021] Figure 5 This is a sequence diagram between the HMD and the virtual reality service server in Example 1.

[0022] Figure 6 This is a diagram for explaining the virtual space of the HMD and the visual recognition range of the user in the first embodiment.

[0023] Figure 7 This is a display image of the HMD in Example 1, in which a photographing image is superimposed on a part of the display image.

[0024] Figure 8 1 is a diagram showing an example of a virtual space that is an imaging target of the HMD in the first embodiment.

[0025] Fig. 9A This is an example of displaying a photographing image of the HMD in the case where use of photographing target data capable of identifying the user is permitted in the first embodiment.

[0026] Fig. 9B This is an example of displaying a photographing image of the HMD when use of photographing target data that can identify the person is not permitted in the first embodiment.

[0027] Fig. 9C This is an example of display of an image for photographing by the HMD when photographing is not permitted in the first embodiment.

[0028] Fig.10 This is the user attribute table managed by the virtual reality service server in Example 1.

[0029] Fig.11 This is a virtual reality processing flow chart of the virtual reality service program of the HMD in the first embodiment.

[0030] Fig.12 This is a sequence diagram between the HMD and the virtual service providing server in the second embodiment.

[0031] Fig.13 This is a virtual reality processing flow chart of the virtual reality service program of the HMD in the second embodiment.

[0032] Fig.14 This is a virtual reality processing flow chart of the virtual reality service program of the HMD in the third embodiment. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0034] Example 1

[0035] Figure 1 is a system structure diagram of the virtual reality system in this embodiment. Figure 1 In FIG. 1 , 100 is a virtual reality service server (hereinafter sometimes referred to as server 100), 200 is a network, 300 is an access point, 1 is an HMD, and 1A is a user. Figure 1 In FIG. 1 , the access point 300 is set at the base A, and the access points are also set at the base B and the base C. Since the functions of these access points are the same, the user can receive the provision of virtual reality services from the virtual reality service server 100 via the network 200 from different bases via each access point. The user 1A wears the HMD 1 and is located at the base A. Although the symbols are omitted, there are other users, and they can receive the provision of virtual reality services at the same time.

[0036] Figure 2 : is an appearance diagram of the HMD in this embodiment. Figure 2 In the figure, HMD 1 includes a camera 10, a distance measuring unit 11, a pair of left and right image projection units 12a, 12b, a screen 13, a sensor group 14 for position and movement, a control unit 15, a pair of left and right speakers 16a, 16b, a microphone 17, and wearing units 18a, 18b.

[0037] The user of the HMD 1 (user of the virtual reality system) wears the HMD 1 on his / her head using the wearing parts 18a and 18b. The wearing part 18a supports the HMD via the nose of the face, and the wearing part 18b fixes the HMD around the head.

[0038] The camera 10 captures the front of the HMD 1. The control unit 15 takes in the image of the camera 10 and recognizes real objects from the image. In addition, the depth data obtained from the distance measuring unit 11 is given to the real object to recognize the real object in three dimensions. In addition, the control unit 15 generates a background image generated based on the background data of the virtual space and an avatar image generated based on the object data as a three-dimensional image of the virtual space projected by the projection units 12a and 12b onto the screen 13. In addition, the control unit 15 generates sound amplified by the speakers 16a and 16b.

[0039] The display unit of the HMD 1 is composed of the projection units 12a, 12b and the screen 13. The projection unit 12a projects an image of a virtual object for visual recognition with the left eye onto the screen 13, and the projection unit 12b projects an image for visual recognition with the right eye onto the screen 13, so that the virtual object is displayed as if it is at a predetermined distance in the real space.

[0040] Figure 3 1 is a functional block diagram of the HMD in this embodiment, showing the details of the internal structure of the control unit 15. Figure 2 The same symbols are used to designate the same functions. Figure 2The projection units 12a and 12b are combined as the projection unit 12. In addition, a microphone, a speaker, a screen, etc. are omitted.

[0041] exist Figure 3 Among them, 20 is an image recognition operation unit, 21 is a communication unit, 22 is a shooting tool processing unit, 23 is a position movement processing unit, 24 is a virtual reality image processing unit, 25 is a personal data holding unit, 26 is a display processing unit, and 27 is a data storage unit.

[0042] The image recognition operation unit 20 receives the camera image from the camera 10 and the distance data from the distance measuring unit 11, recognizes real objects such as the user's fingers or arms from the real space captured by the camera image, and assigns depth data to the feature points of the real objects. In addition, the operation intended by the user is recognized based on the movement of the user's fingers or hands.

[0043] The communication unit 21 downloads object data of the virtual space via the network, or reads already stored object data from a storage device (not shown).

[0044] The imaging tool processing unit 22 generates an image for capturing a portion of a virtual space by providing the position, direction, and angle of view of the image capture, so that the camera captures the image from an arbitrary position as if operating a drone or the like in the real space.

[0045] The position and motion processing unit 23 obtains the viewpoint from the position information and obtains the line of sight from the direction information based on the sensor signals of the GPS, direction, and gyro sensors outputted by the position and motion sensor group 14 .

[0046] The virtual reality image processing unit 24 generates a display image based on a background image of background data of a virtual space and an avatar image of object data obtained based on a viewpoint and a line of sight.

[0047] The personal data storage unit 25 stores user information such as name and subject attributes required for logging into a virtual reality service, etc. In addition, the data storage unit 27 stores images for shooting.

[0048] The display processing unit 26 transmits the display image generated by the virtual reality image processing unit 24 or the photographing image generated by the photographing tool processing unit 22 to the projection unit 12 .

[0049] Figure 4 : is a hardware block diagram of the HMD in this embodiment. Figure 4 In, with Figure 2 , Figure 3 The same functions are denoted by the same symbols and their descriptions are omitted. Figure 4 In, with Figure 3The difference of the functional block diagram is that the control unit 15 is constituted as an information processing device such as a CPU that interprets the operation program and executes various functions by software processing. As an information processing device, there is an advantage that a general-purpose device such as a smartphone can be used.

[0050] exist Figure 4 In FIG. 1 , the control unit 15 includes a communication unit 21, a CPU 30, a RAM 31, a flash ROM (FROM) 32, and an interface unit 36. The interface unit 36 ​​is connected to an interface unit 37 located in the HMD body, and is also responsible for external output.

[0051] The communication unit 21 of the control unit 15 selects an appropriate process from several communication processes such as mobile communication such as 4G and 5G, wireless LAN, etc., and connects the HMD 1 to the network. In addition, object data of the virtual space is downloaded from an external server. In FROM 32, as a processing program, a basic program 33, a virtual reality service program 34, and a data storage unit 35 are included. These processing programs are expanded in RAM 31 and executed by CPU 30. In the data storage unit 35, in addition to temporarily storing intermediate data required for executing the processing program, it is also responsible for Figure 3 The personal data holding unit 25 and the data storage unit 27 are used as the personal data holding unit 25 and the data storage unit 27. As shown in the figure, FROM 32 can be a single storage medium or a plurality of storage media. Moreover, it can also be a non-volatile storage medium other than a flash ROM.

[0052] The interfaces implemented by the interface units 36 and 37 may be wired interfaces such as USB (registered trademark) and HDMI (registered trademark), or wireless interfaces such as wireless LAN.

[0053] Figure 5 1 is a sequence diagram between the HMD and the virtual reality service server in this embodiment. The left side of the figure is the virtual reality service server 100, and the right side is the virtual reality processing unit of the HMD 1 (hereinafter sometimes referred to as HMD).

[0054] exist Figure 5 In the process, first, in step S10, login is started in HMD1. Then, in step S11, an authentication request is sent from HMD1 to server 100. The authentication request includes the user's ID, password (PW), and the profile of HMD1. The user's ID and password (PW) are associated with the user's real name and the user's image for authentication, and are managed by server 100. The profile of HMD1 is the capability information of the hardware and software of HMD1, for example, the capability to distinguish between the display images and the shooting images of virtual reality, the capability to output the display images externally, etc.

[0055] Next, in step S12 , if the ID and password of the user transmitted from the HMD 1 match the contents registered in the server 100 , the HMD 1 is authenticated, and confirmation of authentication OK is issued from the server 100 .

[0056] Next, in step S13, the HMD 1 issues a user attribute update. User attributes include subject attributes applied when photographing the user, user display and photographing object data such as avatar images, etc. In addition, the timing of issuance may not be Figure 5 In that case, it is issued at an arbitrary timing in the time series. After receiving the user attribute update, the server 100 uses the new user attribute.

[0057] Steps S14 to S17 are a sequence for HMD1 to obtain display images for the virtual reality system. In step S14, HMD1 sends viewpoint parameters such as the user's position and sight direction in the virtual space. In step S15, the server 100 extracts objects that exist within the range that HMD1 can visually recognize based on the received position and sight direction. Moreover, in step S16, the server 100 sends background data and the extracted object data. Regarding background data, a large range of background data can also be sent in advance, and only data to supplement it is sent in S16. In step S17, HMD1 uses the received data to generate a display image for virtual reality and displays it.

[0058] Steps S18 to S24 are a sequence of shooting. In step S18, shooting parameters such as the position, direction, and viewing angle of the shooting location are determined by the shooting tool of the HMD 1, and sent to the server 100 in step S19.

[0059] In step S20, the server 100 extracts objects existing in the shooting range based on the received shooting parameters. Furthermore, in step S21, the shooting attributes related to the objects of other users among the extracted objects are confirmed. In step S22, the server 100 sends the shooting attributes of the objects of other users. The shooting attributes of the objects of other users are information indicating whether other users permit shooting, or information indicating whether the person is permitted to be identified when shooting. This information is registered by other users as their own settings, and is recorded and managed by the server 100. In step S23, the server 100 sends the background and object data.

[0060] In step S24, HMD1 generates and saves a photographic image by using the received photographed attributes, background, and object data of the object of the other user. At this time, in the generation of the photographic image, if the photographed attributes of the other user allow the identification of the person, a virtual image image that can identify the person is obtained based on the photographic object data. In this case, an image containing a virtual image that can identify the person in the photographing is recorded as photographing data. On the other hand, if the photographed attributes of the other user do not allow the identification of the person, a virtual image image that is difficult to identify the person is obtained based on the photographic object data, and a photographic image is generated. In this case, an image containing a virtual image that is difficult to identify the person in the photographing is recorded as photographing data. Moreover, if the photographed attributes of the other user do not allow the photographing, the virtual image image is not used. In this case, an image that does not contain the virtual image of the other user in the photographing is recorded as photographing data.

[0061] Thereafter, the sequence of steps S14 to S17 for the HMD 1 to obtain an image for displaying virtual reality and the sequence of steps S18 to S24 for photographing are repeatedly executed.

[0062] Figure 6 1 is a diagram for explaining the virtual space of the HMD and the visual recognition range of the user in this embodiment. The visual recognition range of the user is the display image in the HMD 1.

[0063] exist Figure 6 In the example, the virtual space P10 is wider than the user's visual recognition range P11. HMD1 can receive the background data of the wide virtual space at one time or in several times. The user's visual recognition range P11 is determined by the user's position and line of sight in the virtual space. When the virtual image of another user exists in the user's visual recognition range P11, HMD1 obtains the virtual image image P12 from the display object data of the other user, multiplexes it with the background of the virtual space in the visual recognition range, and generates a display image. When HMD1 is not shooting, the virtual image image P12 that can identify the user is displayed.

[0064] Figure 7 is a display image of the HMD in this embodiment, and is an example in which the photographic image P14 is superimposed on a portion of the display image P13. Figure 7 In addition to the method of superimposing a part of the display image P13 on the captured image P14 as shown, a method of switching the display image to the captured image at a constant time and displaying the same may be used.

[0065] Next, use Figure 8 and Fig. 9A , Fig. 9B , Fig. 9C The display and the photographic image when photographing is performed by the HMD 1 will be described. Figure 8 : is a diagram showing an example of a virtual space that is a photographing object of the HMD in this embodiment. Figure 8 In the virtual space P10, there is a visual recognition range P11 of the user. In the visual recognition range P11, there are three virtual images P12, P20, and P21 of other users.

[0066] Fig. 9A , Fig. 9B , Fig. 9C It is explained in Figure 8 . It is assumed that the other users shown by the avatars P20 and P21 allow the use of the photographing object data that can identify the user. Next, the difference in display caused by the photographed attributes of the other users shown by the avatars P12 is explained.

[0067] Fig. 9A This is a display example when the user of P12 has permitted the use of the photographing target data that can identify the user. In this case, the photographing image P14 uses the avatar image P12 for display that can identify the user. Fig. 9B This is a display example when the user of P12 does not permit the use of the photographing target data that can identify the user. In this case, the photographing image P14 uses an avatar image P17 for photographing that is difficult to identify the user. Fig. 9C This is a display example when the user of P12 does not permit photography. In this case, the avatar images of other users are not superimposed on the photographic image P14.

[0068] Fig.10 is a user attribute management table managed by the server 100 in this embodiment. Fig.10 , the attribute items of the user attribute management table include a user management number (USR#) T11, authentication data T12, display object data (abbreviated as display OBJ in the figure) T15, login status T16, and photographed attribute T17.

[0069] The authentication data T12 is composed of name / password (Name / PW) T13 and image data for personal identification T14. The subject attribute T17 is composed of the subject attribute items of unconditional permission T18, photographer-limited permission T19, object replacement instruction (abbreviated as OBJ replacement instruction in the figure) T20, object data for shooting (abbreviated as OBJ for shooting in the figure) T21, charging permission T22, and non-permission T23.

[0070] The personal authentication image T14 is, for example, an image of the person that is equivalent to an ID card issued by a public institution and is encoded and registered. Fig.10 The display object data T15 is data that can generate a detailed display avatar image that can identify the level of each user, and is encoded as highly confidential data. Fig.10 An example of the appearance of the avatar is shown in . The logged-in state T16 indicates that the virtual reality service is being used.

[0071] The object replacement indication T20 is defined in three states. If the value of the object replacement indication T20 is 0, the object is not replaced and the display object data T15 is used for display. If the value of the object replacement indication T20 is 1, the shooting object data T21 is used. If the value of the object replacement indication T20 is 2, the corresponding object is not displayed. The shooting object data T21 is used when the value of the object replacement indication T20 is 1, and is data that can generate a shooting virtual avatar image that makes it difficult to determine the level of each user. Fig.10 2 shows a case where a simple humanoid character is registered.

[0072] For example, if a user whose user management number T11 is 1 is described, the user name is A and the password is B. The photographed attribute T17 has user management numbers 2 and 3 registered in the photographer-limited permission T19. In addition, the object replacement indication T20 indicates that the user management numbers 2 and 3 have a value of 0 and do not require object replacement, and the other users have a value of 1 and require object replacement. Therefore, only when the photographer is a user with user management numbers 2 and 3, the display object data T15 is used in the captured image. That is, when the user management numbers 2 and 3 shoot, a virtual image that can be identified as the user with user management number 1 is recorded. For users other than management numbers 2 and 3, the value of the object replacement indication is 1, and if it is replaced with the shooting object data T21, shooting can be performed. Therefore, when the user other than management numbers 2 and 3 shoots, a virtual image that is difficult to identify is recorded.

[0073] The user whose user name is C and whose user management number T11 is 2 does not need to replace the object with the users of user management numbers 1 and 3. Therefore, when the users of user management numbers 1 and 3 take the photo, the avatar that can be identified as the user of user management number 2 is recorded. In the case of other users, the value of the object replacement instruction T20 is 2, so the corresponding object is not displayed. Therefore, when the users other than user management numbers 1 and 3 take the photo, the avatar of the user of user management number 2 is not recorded.

[0074] The user whose user name is E and whose user management number T11 is 3 has not set the photographer-limited permission, but if the shooting object data T21 is replaced, shooting is permitted. Therefore, an avatar that is difficult to identify no matter which user shoots is recorded.

[0075] The user whose user name is G and whose user management number T11 is 4 needs to replace the object with the user management number 1 or 3. Therefore, when the user with the user management number 1 or 3 takes a photo, the avatar whose identity is difficult to identify is recorded. In the case of other users, the value of the object replacement instruction T20 is 2, so the corresponding object is not displayed. Therefore, when the user other than the management number 1 or 3 takes a photo, the avatar of the user with the user management number 4 is not recorded.

[0076] The user whose user name is 1 and whose user management number T11 is 5 sets T23 in the subject attribute T17 to disallow photographing. Therefore, no matter which user takes a photograph, the avatar of the user with user management number 5 is not recorded.

[0077] The user whose user name is K and whose user management number T11 is 6 has unconditional permission T18 set in the subject attribute T17, and photography is unconditionally permitted. That is, an avatar that can be identified as the user with user management number 6 is recorded regardless of which user takes the photo.

[0078] A user whose user name is M and whose user management number T11 is 7 sets a paid license T22 in the photographed attribute T17. In this case, if Fig.10 As shown in FIG. 1, $1.2 can record an avatar that can be identified as the user of user management number 7. If no payment is made, the value of the object replacement instruction is 1, so if it is replaced with the shooting object data T21, shooting can be performed.

[0079] In addition, Fig.10 In the Metaverse, it is also possible to set shooting permission / prohibition in specific places in the Metaverse. In addition, the photographer-limited permission can be specified not only by user ID such as name, but also by relationship, such as friend registration. In addition, the recording conditions can be associated with the avatar through NFT (Non-Fungible Token: non-homogeneous token).

[0080] Fig.11 : is a virtual reality processing flow chart of the virtual reality service program 34 of the HMD in this embodiment. Fig.11 In, with Figure 5 The same processes are denoted by the same reference numerals, and their description is omitted.

[0081] exist Fig.11 In step S50, the process starts, and the login authentication is performed in step S51. In step S13, the data for updating the user attributes is sent to the server 100. In addition, step S13 may not be performed at this timing.

[0082] In step S14, the viewpoint parameters are sent, and in step S54, the background and object data are received according to the position information. In step S17, the virtual image is calculated as the virtual reality image according to the object data. In the case of multiple object data, the virtual image is calculated for all the object data. Moreover, the display image is generated and displayed according to the background image and the virtual image. Steps S14 to S17 are the processing of the display image.

[0083] In step S56, it is determined whether the state is shooting. If it is not shooting (No), the process returns to step S14 and the process of displaying the image is repeated. If shooting is to be performed (Yes), shooting parameters such as the shooting position are sent in step S19. Next, in step S58, the shooting attributes of the object within the shooting range are received based on the shooting parameters, and in step S59, the background and object data are received. The received object data is display object data when the shooting attributes indicate that the object can be photographed, and becomes shooting object data when the shooting attributes indicate that the object has replacement shooting permission. In the case where the object data to be received has been received during the process of displaying the image, the reception of the object data can be omitted and the temporarily stored object data can be used.

[0084] In step S24, a virtual avatar image is calculated based on the object data. In the case of multiple object data, the virtual avatar image is calculated for all the object data. In addition, a photographic image is generated and saved based on the background image and the virtual avatar image. Steps S19 to S24 are processing of the photographic image.

[0085] In step S61, the continuation of the program is confirmed. If it is continued (yes), the process returns to step S14. If it is ended (no), the process ends in step S62.

[0086] As described above, the virtual reality system in this embodiment is composed of an HMD with virtual reality processing installed, a virtual reality service server, and a network. When a user wears the HMD to experience virtual reality and wants to shoot a part of the virtual space, the shooting tool of the virtual reality processing is used to perform the shooting. At this time, the virtual reality processing of the HMD transmits the shooting parameters to the virtual reality service server. When the object of other users is within the shooting range, the virtual reality service server transmits the attributes related to the shooting permission of the object of other users to the HMD. In the absence of shooting permission, a virtual image image whose personal information is difficult to determine is applied to the object of other users in the HMD to generate an image for shooting in the virtual space.

[0087] In addition, the HMD in this embodiment is composed of a control unit that performs virtual reality processing, a communication unit, a position sensor unit, a display unit, and an image recognition operation unit. Moreover, it can also be equipped with a data storage unit and an external output unit. The communication unit is connected to the network and communicates with the virtual reality service server via the network. The information of the position sensor unit is sent to the virtual reality service server, and the background data of the virtual space based on the user's current position, line of sight, etc., and the object data of other user objects existing in the user's visual recognition range are received from the virtual reality service server. In addition, in the control unit, a virtual image image that can be recognized by the user is used to generate a display image for virtual reality, and the display unit is displayed. The image recognition operation unit can be composed of a camera unit and an image recognition unit, for example. The user's hand movement is photographed by the camera unit in front of the HMD, and the image recognition unit recognizes it, thereby determining the user's operation. When the user wants to shoot a part of the virtual space, the shooting tool of the control unit is used. The shooting tool is similar to drone shooting in the real space. The user can shoot as in the real space. For example, it is a snapshot with a partner who is experiencing the virtual reality as the background provided by the virtual space. The captured image for shooting is stored in the data storage unit, or output from the external output unit to an external device. At this time, sometimes the virtual image of another user is projected in the background. The control unit transmits shooting parameters such as shooting position, direction, and viewing angle to the virtual reality service providing server so that it recognizes it as a shooting mode, obtains the attributes related to the shooting permission of other user objects projected in the shooting range from the virtual reality service server, and generates the image for shooting using a virtual image of a user that is difficult to identify, etc., in the absence of shooting permission.

[0088] As described above, according to the present embodiment, in a non-anonymous virtual reality system, a shooting function in a virtual space can be provided in consideration of user privacy protection.

[0089] Example 2

[0090] This embodiment also applies to the case where the HMD does not have the ability to distinguish between the display image and the captured image for processing virtual reality, or the case where the display image is output externally. Figure 2 , Figure 3 , Figure 4 The structure of the HMD1.

[0091] Fig.12 is a timing diagram between the HMD and the virtual reality service server in this embodiment. Fig.12 In, with Figure 5 The same parts are denoted by the same symbols, and repeated descriptions are omitted.

[0092] exist Fig.12 In the embodiment 1, steps S14 to S17 are the same as those in the embodiment 1. Figure 5 The same is the timing for HMD1 to obtain the display image of the virtual reality system. After the login process, HMD1 sends the status flag to the server 100 in step S30. The status flag is shooting notification information indicating whether HMD1 is in a non-shooting state or a shooting state. Usually, immediately after the user logs in, the user does not start shooting, so HMD1 sends a value indicating a non-shooting state. In addition, in the case where HMD1 does not have the ability to distinguish between the display image and the shooting image of the virtual reality, or does not have the ability to stop the external output of the display image, it is necessary to send a value indicating the shooting state.

[0093] In step S31, the server 100 determines the shooting state of the HMD 1 using the state flag. When the state flag received by the server 100 is a value indicating a non-shooting state, the background and object data of the virtual reality object are sent to the HMD 1 in step S16. In this case, the HMD 1 generates and displays a display image of the virtual reality using the object data received in step S17. In step S16, the normal display object data that is not assumed to be shot is sent from the server 100, and the display virtual image is displayed in step S17. So far, this is the processing when the state flag is sent in step S30. The value indicating the non-shooting state.

[0094] On the other hand, when the HMD 1 transmits a value indicating the shooting state through the state flag in step S30, the server 100 determines that the state flag is a value indicating the shooting state in step S31. In this case, the server 100 skips step S20 from step S16 and proceeds to step S23.

[0095] In HMD 1, steps S18 to S33 are a sequence of shooting. In step S18, shooting parameters of HMD 1 are determined, and in step S19, the shooting parameters are sent to server 100. In step S32, the state flag of HMD 1 is sent to server 100. The state flag at this time is a value indicating the shooting state.

[0096] In step S20, when the server 100 receives the shooting parameters and the status flag indicating the shooting status value, it extracts the object existing in the shooting range based on the shooting parameters. Alternatively, the HMD 1 may not explicitly send the status flag, and the server 100 may regard the sending and receiving of the shooting parameters as the shooting status and process step S20.

[0097] Next, in step S23, the server 100 transmits the extracted background and object data. In step S17, the HMD 1 generates and displays a virtual reality image as a photographing image, and in step S33, stores or outputs the virtual reality image as a photographing image.

[0098] If the transmission of the status flag is not confirmed in step S30, the server 100 regards the HMD 1 as being in the shooting state and performs processing. Fig.10 The object data to be sent to HMD1 is selected by the photographed attribute T17. Therefore, unless unconditional permission T18 is set, the display object data T15 of other users in the virtual space is not sent to HMD1, thereby protecting the privacy of the user. The external output does not know what capabilities the connected external device has, so the protection of the privacy of other users is based on the photographed image.

[0099] Fig.13 : is a virtual reality processing flow chart of the virtual reality service program 34 of the HMD in this embodiment. Fig.13 In, with Fig.11 The same steps are marked with the same symbols, and repeated descriptions are omitted.

[0100] exist Fig.13 In step S56, the shooting state is determined. If it is the shooting state, a value indicating the shooting state is sent through a state flag in step S32, and shooting parameters such as the shooting position are sent in step S19. If the shooting state is the non-shooting state in S56, a value indicating the non-shooting state is sent through a state flag in step S30. Then, object data is received in step S59. The object data received in step S59 is data sent from the server 100 based on the state flag sent in step S32 or S30. For example, in the case where the subject's attribute is that the subject can be photographed, it is display object data for generating a display virtual avatar image that can identify the subject. In the case where the subject's attribute is that the subject's replacement shooting permission is allowed, it becomes shooting object data for generating a shooting virtual avatar image that cannot identify the subject.

[0101] In step S73, a virtual avatar image is generated based on the object data, and the generated virtual reality image becomes a display image and a photographing image to be displayed in step S74, and becomes an external output image to be displayed or output to the outside.

[0102] As described above, according to this embodiment, in a non-anonymous virtual reality system, even when the HMD does not have the ability to distinguish between the display image and the photographed image of the virtual reality, or when the display image is output externally, by sending the state flag in advance, it is possible to provide a photographing function in a virtual space that takes into account the privacy protection of the user. In addition, as long as the server does not receive the state flag, the virtual image is displayed according to the photographed attribute of the object of other users, so that privacy can be reliably protected.

[0103] Example 3

[0104] Fig.14 3 is a virtual reality processing flow chart of the virtual reality service program 34 of the HMD in this embodiment. Figure 2 , Figure 3 , Figure 4 The structure of HMD1. Fig.14 In, with Fig.11 The same steps are marked with the same symbols, and repeated descriptions are omitted.

[0105] exist Fig.14 middle, Fig.11 The step S17 of generating the display image is replaced by steps S80 to S83. In step S80, a virtual reality image is generated, but before the generated virtual reality image is displayed as a display image in step S83, it is confirmed in step S81 whether there is a notification of whether the user has become the subject. In the case of becoming the subject in step S81 (yes), a display indicating the photographed state, i.e., a notification mark, is superimposed on the virtual reality image in step S82. The notification mark can be a colored mark such as red, for example, and it is a mark that makes the user aware that they are being photographed. The virtual reality image superimposed with the notification mark is displayed as the display image in step S83.

[0106] In addition, as a notification method, a method other than a notification mark may be used. For example, since the user's avatar's hand can be visually recognized by the user, the avatar's hand may be changed to a display different from the usual one. As an example of display change, it is possible to perform processing such as making the visually recognizable part of the hand glow, changing the color, or making it semi-transparent.

[0107] As described above, according to this embodiment, in the HMD of the non-anonymous virtual reality system, a shooting function in a virtual space can be provided that takes the privacy protection of the user into consideration, and the effect of being able to easily identify the person being shot as in the real space can be achieved.

[0108] The above embodiments are described, but the present invention is not limited to the above embodiments, and includes various modifications. For example, in the above embodiments, it is described that the action program is interpreted by a CPU or the like and various functions are executed through software processing, but part or all of the above structures may also be composed of hardware, or hardware and software may be used in combination. In addition, the above embodiments are examples described in detail in order to easily understand the present invention, and are not limited to all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can be added to the structure of a certain embodiment. In addition, for a part of the structure of each embodiment, other structures can also be added, deleted, or replaced.

[0109] Explanation of symbols

[0110] 1: Head mounted display (HMD), 1A: User, 100: Virtual reality service server (server), 200: Network, 300: Access point, P10: Virtual space, P11: Visual recognition range, P12, P17, P20, P21: Avatar, P13: Display image, P14: Photographing image, T11: User attribute data, T12: Authentication data, T15: Display object data, T17: Photographed attribute, T21: Photographing object data, 10: Camera, 11: Distance measuring unit, 12, 12a, 12b: Projection unit, 13: Screen, 14: Sensor group, 15: Control unit, 20: Image recognition operation unit, 21: Communication unit, 22: Shooting tool processing unit, 23: Position motion processing unit, 24: Virtual reality image processing unit, 25: Personal data holding unit, 26: Display processing unit, 27: Data storage unit, 30: CPU, 32: FROM, 34: Virtual reality service program, 35: Data storage unit, 36: Interface unit.

Claims

1. A virtual reality system, comprising a server providing virtual reality services, a head mounted display receiving the virtual reality services, and a network connecting the server and the head mounted display, characterized in that: The server stores, as user information, first object data for generating a first virtual avatar image for display, second object data for generating a second virtual avatar image for photographing, and photographed attributes for setting photographing conditions when the user is photographed by another user. The server sends the first object data or the second object data to the head mounted display according to the photographed attribute, The head mounted display generates and displays the first virtual avatar image or the second virtual avatar image according to the received first object data or the second object data.

2. The virtual reality system according to claim 1, characterized in that: When the attributes of the subject being photographed permit identification of the individual, the server generates first object data for the first virtual image for display and sends it to the head-mounted display as object data for display; and when the attributes of the subject being photographed do not permit identification of the individual, the server generates second object data for the second virtual image for shooting and sends it to the head-mounted display as object data for shooting.

3. The virtual reality system according to claim 2, characterized in that: When shooting the virtual space, the head mounted display sends viewpoint parameters such as the position and sight direction of the user in the virtual space, or shooting parameters such as the position, direction, and viewing angle of the shooting location in the virtual space as shooting information to the server. The server identifies other users existing in a photographing range based on the photographing information, and transmits the display object data or the photographing object data to the head mounted display according to the photographed attributes of the other users.

4. The virtual reality system according to claim 3, characterized in that: The server obtains capability information of the head mounted display and identifies other users within a visual recognition range, and sends the display object data or the shooting object data of other users to the head mounted display based on the capability information and the photographed attributes of the other users.

5. The virtual reality system according to claim 4, characterized in that: The capability information is information for distinguishing the capability of generating a display image and a photographing image, or the capability of outputting a display image to the outside.

6. The virtual reality system according to claim 2, characterized in that: The head mounted display sends shooting notification information indicating whether the virtual space is being shot to the server, The server transmits the display object data or the photography object data to the head mounted display according to the photography notification information and the photographed attributes of other users.

7. The virtual reality system according to claim 6, characterized in that: When the server fails to receive the photographing notification information, the server transmits the photographing object data to the head mounted display.

8. The virtual reality system according to claim 1, characterized in that: The first virtual image for display is a virtual image capable of identifying a user, The second avatar image for shooting is an avatar image in which it is difficult to identify the user.

9. A head mounted display that communicates with a server providing virtual reality services, characterized in that: The head mounted display comprises a control unit, a communication unit, and a display unit for executing virtual reality processing. The communication unit receives, from the server, first object data or second object data corresponding to a subject attribute that sets a photographing condition when a user is photographed by another user, The control unit generates a first virtual avatar image for display or a second virtual avatar image for photography using the received first object data or the received second object data, and displays the image on the display unit.

10. The head mounted display according to claim 9, characterized in that: The first object data is display object data for generating the first virtual image for display when the attribute of the subject permits identification of the person concerned, and the second object data is shooting object data for generating the second virtual image for shooting when the attribute of the subject permits not identification of the person concerned.

11. The head mounted display according to claim 10, wherein: The control unit sends the viewpoint parameters such as the user's position and sight direction in the virtual space, or the shooting parameters such as the position, direction, and viewing angle of the shooting location in the virtual space as shooting information to the server via the communication unit when shooting the virtual space.

12. The head mounted display according to claim 11, characterized in that: The control unit sends capability information of the head mounted display to the server via the communication unit, and receives from the server the display object data or the shooting object data of other users corresponding to the capability information and the subject attributes of other users existing in the visual recognition range based on the viewpoint parameter.

13. The head mounted display according to claim 12, wherein: The capability information is information for distinguishing the capability of generating a display image and a photographing image, or the capability of outputting a display image to the outside.

14. The head mounted display according to claim 10, characterized in that: The control unit transmits shooting notification information indicating whether the virtual space is being shot to the server via the communication unit, and receives the display object data or the shooting object data of another user corresponding to the shooting notification information.

15. The head mounted display according to claim 9, characterized in that: The head mounted display includes a data storage unit for storing user attributes composed of personal data and image data. The personal data includes authentication data and photographed attributes, The photographed attributes include photographing permission information and conditional photographing permission information. The image data includes display object data and shooting object data used in conditional shooting. The control unit transmits the user attributes to the server via the communication unit.

16. The head mounted display according to claim 10, wherein: When the user becomes the subject of the photograph, a display showing the photographed state is performed in the display image.

17. The head mounted display according to claim 9, characterized in that: The first avatar image for display is an avatar image that can identify the user, and the second avatar image for shooting is an avatar image that is difficult to identify the user.

Images