Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program which enable a user to experience a content displayed in a virtual space safely and comfortably.SOLUTION: An HMD which is worn by a user and displays a virtual space determines whether a content displayed in the virtual space from obstacle information indicating a positional relation between an obstacle within a play area of the user and the play area and user information acquired from the user can be transited into the next state, and performs reporting corresponding to a result of the determination.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program, and more particularly to an information processing system, an information processing method, and a program for controlling state transitions of content displayed in a virtual space. [Background technology]

[0002] Recently, a technology called VR (Virtual Reality), which allows users to experience immersion in a computer-generated virtual space that differs from reality, has become widely used. There is also a wide range of content (hereafter referred to as VR content) that is displayed in a virtual space that utilizes VR technology. For example, a user wears a head-mounted display (HMD) on their head and plays a game by operating a controller while viewing a screen displayed on the HMD. Wearing an HMD in this way obscures the real world, enhancing the user's sense of immersion in the virtual space. In addition, in VR content where remote users or non-player characters (NPCs) are displayed within the HMD and a sports match is played, the user wearing the HMD can also play the game within a play area set up in the virtual space.

[0003] For example, Patent Document 1 discloses a technology in which, when a user is wearing an HMD and enjoying a virtual space, the HMD transitions the game to a paused state upon receiving a game interruption signal, and then transitions the game to a resumed state upon receiving a game resume signal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-029968 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, the game state transition is performed in response to the reception of signals such as a game interrupt signal or a game resume signal, which poses a problem that the game state transition is performed without ensuring the safety of the user.

[0006] The present invention has been made in consideration of such problems, and aims to provide an information processing device, an information processing method, and a program that allow users to safely and comfortably experience content displayed in a virtual space. [Means for solving the problem]

[0007] In order to solve the above problem, the information processing device of claim 1 of the present invention is an information processing device that displays a virtual space on a head-mounted display worn by a user, and is characterized by comprising: a play area acquisition means that acquires the user's play area as a first play area; an obstacle detection means that detects obstacles in real space within the first play area and generates obstacle information indicating the positional relationship between the detected obstacles and the first play area; a user information acquisition means that acquires user information from the user; a state transition determination means that determines whether the content displayed in the virtual space can transition to the next state from the obstacle information and the user information; and a notification means that issues a notification according to the determination result of the state transition determination means. [Effects of the Invention]

[0008] According to the present invention, a user can safely and comfortably experience content displayed in a virtual space. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of an information processing device according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing a functional configuration of an information processing device according to a first embodiment of the present invention. [Figure 3]FIG. 2 is a diagram illustrating an example of state transition control of VR content according to the first embodiment of the present invention. [Figure 4] 4 is a flowchart of a state transition control process of VR content according to the first embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing a modified example of state transition control of VR content according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram showing an example of an HMD display according to the first embodiment of the present invention. [Figure 7] FIG. 1 is a diagram showing an example of obstacles and a play area in a three-dimensional space. [Figure 8] FIG. 10 is a block diagram showing a functional configuration of an information processing device according to a second embodiment of the present invention. [Figure 9] FIG. 10 is a diagram illustrating an example of state transition control of VR content according to the second embodiment of the present invention. [Figure 10] 10 is a flowchart of a state transition control process of VR content according to the second embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing an example of an HMD display according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, a configuration may be achieved by appropriately combining parts of each of the embodiments described below. In the following embodiments, the same components will be described with the same reference symbols.

[0011] First Embodiment Fig. 1 is a block diagram showing an example of the hardware configuration of an information processing device 101 according to a first embodiment of the present invention. Fig. 2 is a block diagram showing a functional configuration of the information processing device 101 according to the first embodiment of the present invention. Fig. 3 is a diagram showing an example of state transition control of VR content according to the first embodiment of the present invention. Note that content displayed in a virtual space is referred to as VR content here.

[0012] First, an example of state transition control of VR content in a virtual space according to this embodiment based on the state of the real space and the state of the user will be described with reference to Fig. 3. Here, as an example, a scene in which a user plays VR table tennis at home will be described. However, any scene in which a competitive VR content (sports, games, training) is started at home or the like will suffice, and the VR content to which the present invention is applied is not limited to VR table tennis.

[0013] Image 300 in FIG. 3 is an image of the user viewed from above.

[0014] FIG. 3 shows a trash can 301 which is an obstacle in the real world, a user 302 experiencing the VR content, a virtual ping-pong table 303, a play area 304, an opponent's avatar 305, and a line of sight 306 of the user 302.

[0015] The play area 304 is the play area of ​​the user 302, and its size is predefined depending on the type of VR content and the number of players. In Figure 3, the play area 304 indicates the user's movement range in a singles game of VR table tennis.

[0016] Although the user 302 cannot see the trash can 301 in the real space because he is wearing an HMD (head mounted display), the HMD worn by the user 302 has a function that can acquire position and orientation information of the trash can 301 and the user 302. Therefore, the HMD can obtain obstacle information that indicates the positional relationship between the trash can 301 and the play area 304. Furthermore, the HMD can reflect movements that are similar to those of the user 302 in the real space in an avatar in the virtual space.

[0017] The information processing device 101 determines whether or not a state transition of the VR content is possible based on information about the user 302 (position, posture, line of sight, and voice) acquired by the HMD and information about obstacles in the real space that exist in the play area 304, and displays the result on the HMD. More specifically, this will be explained in the VR content state transition control process described later with reference to the flowchart in FIG.

[0018] In order for the user 302 to transition to a state where he or she can start VR content, it is necessary to confirm in advance whether the safety of the user 302 is ensured and whether the user 302 is ready to start the VR content. However, because the user 302 wearing the HMD cannot see the real space, it is difficult for the user to transition to the next state of the VR content after recognizing an obstacle in the real space (trash can 301) while experiencing the VR content.

[0019] Furthermore, if the opponent is an NPC, the NPC may cause the state of the VR content to transition regardless of the state of the user 302 or the state of the real space, which may result in an uncomfortable user experience.

[0020] 3, there is a possibility that the user 302 may come into contact with the trash can 301 while playing the game, since the trash can 301 is present in the play area 304. Existing technology pauses the game, which is VR content, when a text message is received or when a person speaking near the user 302 is detected, but does not pause the game based on the state of the user 302 or the state of obstacles in the real world.

[0021] Therefore, in this embodiment, the information processing device 101 determines whether or not a state transition of the VR content is possible based on user information such as position information, posture information, and line of sight information of the user 302 experiencing the VR content, and obstacle information in the real space, and presents the result to the user 302. This enables the user 302 to experience the VR content safely and comfortably.

[0022] Next, the hardware configuration of the information processing device 101 will be described with reference to FIG.

[0023] The information processing device 101 is an HMD worn by a user 302 and is connected to an external PC. However, the information processing device 101 may be any electronic device that can communicate with and control the HMD worn by the user 302, and may be, for example, an electronic device such as a PC to which an HMD is connected. The information processing device 101 has, as components, a CPU 102, a ROM 103, a RAM 104, a sensing unit 105, an imaging unit 106, a display unit 107, an operation unit 108, and a communication unit 109. The components are connected to each other via a bus 110.

[0024] The CPU 102 is an arithmetic processing unit that performs overall control of the information processing device 101. The CPU 102 executes various programs stored in the ROM 103 or the like to perform various processes.

[0025] The ROM 103 is a read-only nonvolatile memory device that stores programs (information processing programs, initial data, etc.) and parameters that do not require modification.

[0026] The RAM 104 temporarily stores input information, calculation results in information processing, etc. The RAM 104 is also a memory device that provides a working area for the CPU 102.

[0027] The sensing unit 105 is a device such as a sensor, and acquires information on the position and orientation and gaze direction of the user 302 by detecting the rotation, tilt, amount of movement, and gaze of the head of the user 302 wearing the information processing device 101, which is an HMD. The sensing unit 105 also includes a sound collection device such as a microphone that acquires the voice of the user 302.

[0028] The photographing unit 106 is a photographing device consisting of a built-in camera of the information processing device 101, which is an HMD, and performs photographing processing. If the information processing device 101 is a PC connected to an HMD worn by a user, an externally connected web camera or the like functions as the photographing unit 106.

[0029] The display unit 107 is configured with a liquid crystal display or the like, and displays captured images, virtual objects, characters, and / or items (icons; VR content) to a user wearing the information processing device 101, which is an HMD.

[0030] The operation unit 108 is an operation unit including operation members such as a power button or a dial.

[0031] The communication unit 109 (communication means) transmits and receives data to and from an external device via wired communication or wireless communication (wireless LAN, local 5G, etc.). For example, the external device may be an external controller (not shown) held by a user wearing the information processing device 101, which is an HMD, another HMD, an external operation panel, etc. If the information processing device 101 is a PC connected to an HMD worn by a user, the communication unit 109 receives position and orientation information detected by the information processing device 101, which is an HMD, etc., via a network (not shown).

[0032] FIG. 2 is a block diagram showing the functional configuration of an information processing device 101 (hereinafter simply referred to as HMD 101) according to this embodiment, which is an HMD.

[0033] The HMD 101 has, as its functional configuration, a play area acquisition unit 201, an obstacle detection unit 202, a user position detection unit 203, a user posture detection unit 204, a user gaze detection unit 205, a state transition determination unit 206, and a notification unit 207.

[0034] The play area acquisition unit 201 (play area acquisition means) acquires information about a play area (first play area) in which a user can experience VR content from the ROM 103. The size of the play area is predefined depending on the type of VR content and the number of players.

[0035] The obstacle detection unit 202 (obstacle detection means) detects obstacles in the real space within the play area and generates obstacle information indicating the positional relationship between the detected obstacles and the play area. There are various types of obstacles, such as walls, chairs, desks, trash cans, people, and pets, but the method for identifying obstacles will not be described here. In this embodiment, obstacles may be detected by the imaging unit 106, which is a camera attached to the HMD 101, or by using a second external camera via the communication unit 109. Alternatively, obstacles may be detected by the sensing unit 105, which is a sensor.

[0036] A user position detection unit 203 (user information acquisition means) acquires position information in the virtual space of the user wearing the HMD 101 using the sensing unit 105, and stores the information as user position information.

[0037] A user posture detection unit 204 (user information acquisition means) acquires posture information in the virtual space of the user wearing the HMD 101 using the sensing unit 105, and stores the information as posture information of the user.

[0038] A user line-of-sight detection unit 205 (user information acquisition means) acquires information on the line-of-sight direction of the user wearing the HMD 101 in the virtual space using the sensing unit 105, and stores it.

[0039] The state transition determination unit 206 (state transition determination means) determines whether to transition the VR content state to the next state based on the obstacle information generated by the obstacle detection unit 202 and user information such as the user's position, posture, and line of sight.

[0040] The notification unit 207 (notification means) displays a notification screen on the display unit 107 according to the determination result of the state transition determination unit 206 .

[0041] 4 is a flowchart of the state transition control process of VR content according to this embodiment. The process shown in this flowchart is realized by the CPU 102 reading a program stored in the ROM 103 into the RAM 1104 and executing it. The execution timing of this process is not limited. For example, it may be executed when a user wearing the HMD 101 (hereinafter simply referred to as the user) starts the VR content, or when the operation unit 108 detects a predetermined user operation. The following description will be given using VR table tennis as an example, assuming that the VR content is in a paused state and the user is in the play area when the VR content starts.

[0042] First, in step S401, the play area acquisition unit 201 acquires from the ROM 103 information on the play area of ​​the currently suspended VR content.

[0043] Next, in step S402, the obstacle detection unit 202 detects obstacles in the real space within the play area based on the play area information acquired in step S401. At this time, the notification unit 207 may notify the user to look around so that the area around the user can be widely searched. Thereafter, the obstacle detection unit 202 generates obstacle information indicating the positional relationship between the detected obstacles and the play area.

[0044] Next, in step S403, the user position detection unit 203 acquires the user's position information.

[0045] Next, in step S404, the user posture detection unit 204 acquires posture information of the user.

[0046] Next, in step S405, the user line-of-sight detection unit 205 acquires information about the user's line-of-sight direction.

[0047] Note that the generation of obstacle information and the acquisition of information on the user's position, posture, and gaze direction in steps S402 to S405 may be performed after step S401 and before step S406, which will be described later, and the order in which these steps are performed is not limited to the order shown in FIG.

[0048] Next, in step S406, the state transition determination unit 206 determines whether or not a state transition of the VR content is possible based on the obstacle information generated in step S402 and the information on the user's position, posture, and gaze direction acquired in steps S403 to S405. Specifically, when playing table tennis in real space, the referee checks whether the receiver is ready before resuming a suspended game and before the opponent on the server side hits the ball in his hand to start serving, a similar operation is performed in the VR environment. If it is determined that a state transition of the VR content is not possible (NO in step S406), the process proceeds to step S408, and if it is determined that a state transition of the VR content is possible (YES in step S406), the process proceeds to step S407.

[0049] Here, a method for determining whether or not a state transition of VR content according to this embodiment is possible will be described with reference to Figures 3 and 5. Image 300 in Figure 3 and images 500 and 510 in Figure 5 are images of the user viewed from above.

[0050] 3, if a trash can 301 exists in the play area 304, there is a possibility that the user 302 may collide with the trash can 301 during the game. In such a case, in order to avoid a collision between the user 302 and the trash can 301, the state transition determination unit 206 determines that the state transition of the VR content is not possible, that is, that the current suspended state is to be maintained and that the state will not transition to the play state.

[0051] FIG. 5 is a diagram for explaining a method for determining whether or not a state transition of VR content is possible based on the obstacle information generated in step S402 and the information on the line of sight direction of the user 302 acquired in step S405.

[0052] 5, in image 500, there are no obstacles in the real space in play area 304, but user 302's line of sight 501 is not directed toward opponent avatar 305. Therefore, state transition determination unit 206 determines that the VR content cannot transition to a state of play, i.e., to maintain the current suspended state and not to transition to a play state.

[0053] 5, there are no real-space obstacles in the play area 304, and the line of sight 511 of the user 302 is directed toward the opponent's avatar 305. Therefore, the state transition determination unit 206 determines that the state of the VR content can be transitioned, that is, that the current paused state will be transitioned to the play state.

[0054] In this embodiment, if the angle between the user's line of sight acquired in step S405 and the direction of the opponent's avatar as seen from the user in the virtual space is equal to or smaller than a predetermined angle, it is determined that the user's line of sight is directed toward the opponent's avatar. On the other hand, if the angle between the user's line of sight and the direction of the opponent's avatar as seen from the user in the virtual space is greater than the predetermined angle, it is determined that the user's line of sight is not directed toward the opponent's avatar.

[0055] In this embodiment, the user's gaze direction toward the opponent is used to determine whether a state transition is possible, but this is not limited to this. The user's gaze direction used to determine whether a state transition is possible may be specified for each VR content, or may be set according to the type and number of opponents. In addition to the user's gaze direction, the gaze duration may also be used. Furthermore, the target at which the user's gaze is directed may be a target specified in advance. For example, it may not be the opponent's avatar, but may be a predetermined area in the virtual space.

[0056] In addition, if the user is not present in the play area or if either the user's position or posture is not stationary within the play area, it may be determined that the user is not ready and the state of the VR content cannot be transitioned.

[0057] Furthermore, if the user does not conform to a pre-specified posture, such as a receiving form in table tennis, it may be determined that the user is not ready and the state of the VR content cannot be transitioned.

[0058] The information processing device 101 may also have a user voice detection unit (FIG. 8) described later. In this case, when the user utters a pre-specified word, for example, "WAIT!", it may be determined that the user is not ready and that the state of the VR content cannot be transitioned.

[0059] In other words, the state transition determination unit 206 determines that the state of the VR content can be transitioned if it determines that there are no obstacles in the play area 304 and that the user is ready based on at least one of the user's position, line of sight, posture, and voice.

[0060] Next, in step S407, the state transition determination unit 206 transitions the state of the VR content in accordance with the result of the determination in step S406. For example, the state transition is from the paused state to the play state.

[0061] Next, in step S408, the notification unit 207 performs HMD display to display a notification screen according to the state of the VR content on the user's HMD 101 (display unit 107). After that, this process ends.

[0062] Figure 6 shows an example of an HMD display depending on the state of the VR content.

[0063] In FIG. 6, the user on the receiver side during the VR table tennis match is called the first user, and the user on the server side is called the second user.

[0064] In FIG. 6, an HMD display 600 is an example of an HMD display when the VR content used by the user wearing the HMD 101, who is the first user, is in a play state.

[0065] An HMD display 600 displayed on the display unit 107 of the first user displays a play area 601, a virtual ping-pong table 602, an avatar 603 of the second user who is the opponent, and text 604.

[0066] Text 604 is a user interface that presents a message to the first user according to the current state, and displays "PLAY" to indicate that the VR content is in a play state. In this state, a ball is displayed in the hand of the second user's avatar 603, so the second user can hit the ball and start serving.

[0067] An HMD display 610 in Fig. 6 is an example of an HMD display when the VR content used by a first user wearing the HMD 101 is in a suspended state. An HMD display 620 in Fig. 6 is an example of an HMD display when the VR content used by a second user wearing the HMD 101 is in a suspended state.

[0068] An HMD display 610 displayed on the display unit 107 of the first user displays a play area 611, a virtual ping-pong table 612, an avatar 613 of the second user who is the opponent, and text 614.

[0069] Text 614 is a user interface that presents a message to the first user according to the current state, and displays "TIME!!" indicating that the VR content is in a suspended state.

[0070] Also, an HMD display 620 displayed on the display unit 107 of the second user displays a play area 621, a virtual ping-pong table 622, an avatar 623 of the first user who is the opponent, and text 624.

[0071] Text 624 is a user interface that presents a message to the second user according to the current state, and displays "TIME!!" to indicate that the VR content is in a suspended state. In this state, the ball is not displayed in the hand of the second user's avatar 613, and the second user cannot start serving.

[0072] In this embodiment, the method for determining whether a state transition is possible in a horizontal two-dimensional space has been described using Figures 3 and 5, but a three-dimensional space including the vertical direction may also be used. Also, the shape of the play area may be changed depending on the type and situation of the VR content.

[0073] FIG. 7 is a diagram for explaining a method for determining whether a state transition is possible or not, using obstacle detection and posture information in a three-dimensional space including the vertical direction.

[0074] In FIG. 7, image 700 is an image of the user viewed from above, and image 710 is an image of the user viewed from the side.

[0075] User 701 and user 711 are the same person. Lights 702 and 712 are the same obstacle that exists in real space, and similarly, trash cans 703 and 713 are the same obstacle that exists in real space. Furthermore, play area 704 and play area 714 represent the same area, and ping-pong table 705 and ping-pong table 715 are the same virtual object. It is preferable that the height of play area 714 is set so that even if user 711 raises his or her hand directly above while holding a racket or external controller, the tip of the racket or external controller that he or she is holding will be within play area 714. Furthermore, since user 711 normally does not enter space 714u below virtual table 715 during gameplay, space 714u is excluded from the space of play area 714. Therefore, when viewed from above as in image 700, the trash can 703 appears to be within the play area 704. However, when viewed from the side as in image 710, the trash can 713 is not within the play area 714. Therefore, in the case of FIG. 7, there are no obstacles within the play area, and the user's safety can be ensured even if the game is resumed. Therefore, the state transition determination unit 206 determines that the state of the VR content can be transitioned, that is, the current paused state will be transitioned to the play state. Furthermore, the user posture detection unit 204 may acquire posture information of the user 711 in three dimensions and determine whether the user 711 is crouching below the ping-pong table 715 based on the posture information. In this case, if it is determined that the user 711 is crouching, the state transition determination unit 206 may determine that the user 711 is not ready for a serve from the opponent on the server side and that the state of the VR content cannot be transitioned.

[0076] As described above, according to this embodiment, by using real-space obstacle information and user information, the user can safely and comfortably experience VR content.

[0077] <Second embodiment> An HMD 101 according to the second embodiment will be described with reference to FIGS.

[0078] The HMD 101 according to the second embodiment has the same hardware configuration as the HMD 101 according to the first embodiment, and therefore, a duplicated description will be omitted.

[0079] The HMD 101 according to the first embodiment determines whether a state transition of VR content is possible based on obstacle information in the real space, position and orientation information of the user, and line of sight information, and presents the determination result to the user. In contrast, the HMD 101 according to the second embodiment determines whether a state transition of the VR content is possible based on obstacle information in the real space of each user, and position and orientation information, line of sight information, and audio information of each user, and presents the determination result to the user. Here, as an example, the operation of the HMD 101 according to this embodiment will be described using a scene in which multiple people work together to operate a working device.

[0080] FIG. 8 is a block diagram showing the functional configuration of the HMD 101 according to this embodiment.

[0081] The HMD 101 has, as its functional configuration, a play area acquisition unit 201, an obstacle detection unit 202, a user position detection unit 203, a user posture detection unit 204, a user line of sight detection unit 205, a user voice detection unit 806, a state transition determination unit 206, and a notification unit 207. That is, the HMD 101 in this embodiment has the functions of the HMD 101 according to the first embodiment (FIG. 2) plus the user voice detection unit 806.

[0082] The user voice detection unit 806 (user information acquisition means) acquires voice using a sound collection device such as a microphone attached to the HMD 101 worn by the user. The user voice detection unit 806 may also acquire voice using a sound collection device external to the HMD 101, such as an intercom.

[0083] The operation of the HMD 101 when multiple people work together in this embodiment will be described with reference to FIG. Image 900 in FIG. 9 is an image of the device installation work viewed from above.

[0084] In FIG. 9, users 903, 905, and 908 and a work device 901 exist in the real space.

[0085] The work device 901 is a device in real space that is provided with an operation panel 902 and on which users 903, 905, and 908 can simultaneously perform work.

[0086] 9, users 905 and 908 are users who operate the work device 901. Although not shown in Fig. 9, users 905 and 908 wear HMDs 101 and 101a, respectively, and perform work while receiving work support from the system. User 903 is not wearing an HMD. Note that the HMD 101a is an information processing device that has the same hardware configuration and functional configuration as the HMD 101, and therefore corresponding components are numbered with an "a" at the end, and duplicated explanations will be omitted.

[0087] Play area 906 (first play area) is the play area of ​​user 905 wearing HMD 101 and is acquired by play area acquisition unit 201. Play area 909 (second play area) is the play area of ​​user 908 wearing HMD 101a and is acquired by play area acquisition unit 201a (second play area acquisition means). Work area 904 (first work area) is the area where user 905 works, and work area 911 (second work area) is the area where user 908 works. HMD 101 displays an image of the real space seen by user 905 superimposed on the virtual space. Similarly, HMD 101a displays an image of the real space seen by user 908 superimposed on the virtual space. User 903 operates the entire working device 901 using operation panel 902.

[0088] In this embodiment, the working device 901, working area 904, working area 911, and operation panel 902 are described as objects / areas existing in real space, but they may also be objects / areas in virtual space.

[0089] Gaze direction 907 is the gaze direction of user 905 and is detected by user gaze detection unit 205. Similarly, gaze direction 910 is the gaze direction of user 908 and is detected by user gaze detection unit 205a (second user information acquisition means).

[0090] A task on the task device 901 must be performed in cooperation with users 903, 905, and 908, and once users 905 and 908 are ready, user 903 (third user) operates the operation panel 902. The HMD 101 worn by user 905, the HMD 101a worn by user 908, and the operation panel 902 are connected to each other so that they can communicate with each other. As a result, when the state transition determination unit 206 determines that users 905 and 908 are ready, the result of the determination is reflected in the interface of the operation panel 902.

[0091] Here, the state in which the user 905 is ready means that all conditions are met for the user 905 to safely start working. Specifically, this means that there are no obstacles detected by the obstacle detection unit 202 in the play area 906, the user 905 is in the play area 906, and the line of sight 907 is directed toward the work area 904. In addition to this, the conditions may also include whether the user 905 has uttered a pre-specified word, such as "I'm ready."

[0092] Similarly, the state in which the user 908 is ready means that all conditions are met for the user 908 to safely start work. Specifically, this means that there are no obstacles detected by the obstacle detection unit 202a (second obstacle detection means) in the play area 909, the user 908 is in the play area 909, and the line of sight 910 is directed toward the work area 911. In addition to this, the conditions may include whether or not the user 908 utters a pre-specified word, such as "I'm ready." Note that the state transition determination unit 206 acquires, via the communication unit 109, obstacle information (second obstacle information) generated by the obstacle detection unit 202a of the HMD 101a and user information (second user information) such as the position, posture, and line of sight of the user 908.

[0093] 10 is a flowchart of a state transition control process for VR content according to the second embodiment of the present invention. The process shown in this flowchart is realized by the CPU 102 executing the information processing program according to this embodiment. The execution timing of this process is not limited. For example, it may be executed when the user starts the VR content, or when the operation unit 108 detects a predetermined user operation. Below, an explanation will be given assuming that the VR content is in an unworkable state, taking the installation work of the working device 901 as an example.

[0094] The processing from step S1001 to step S1005 is the same as the processing from step S401 to step S405 shown in FIG. 4, and therefore a description thereof will be omitted.

[0095] In step S1006, the user voice detection unit 806 acquires the user's voice.

[0096] Note that the generation of obstacle information and the acquisition of user position information, posture information, line of sight information, and user voice in steps S1002 to S1006 may be performed after step S1001 and before step S1007, which will be described later. That is, the order in which these steps are performed is not limited to the order shown in Fig. 10. Although not shown in the flowchart of Fig. 10, the CPU 102a of the HMD 101a executes processing similar to steps S1002 to S1006, and acquires obstacle information and various user information in the real environment on the user 908 side.

[0097] Next, in step S1007, the state transition determination unit 206 determines whether or not the state transition of the VR content is possible based on the obstacle information in the real environment acquired in step S1002 and the various user information acquired in steps S1003 to S1006. If it is determined that the state transition of the VR content is not possible (NO in step S1007), the process proceeds to step S1009, and if it is determined that the state transition of the VR content is possible (YES in step S1007), the process proceeds to step S1008.

[0098] Here, a method for determining whether a state transition of VR content according to this embodiment is possible will be described with reference to FIG. 9. As described above, the state transition determination unit 206 determines that a state transition of VR content is possible when the user 905 and the user 908 are ready. Specifically, when all of the following conditions are met: there are no obstacles in the play area 906, the user 905 is in the play area 906, the line of sight 907 is directed toward the work area 904, and the user 905 has uttered a designated word, it is determined that the user 905 is ready. Here, the designated word is not particularly limited, but an example is "ready."

[0099] The conditions for determining that the user 905 is ready may include the user assuming a pre-specified posture, such as a posture similar to the pointing posture for safety confirmation. The conditions for determination may be stored in the ROM 103 in advance, or may be added or deleted by the user. Furthermore, the conditions for determination may be stored in the ROM 103 with a priority, or may be selectable by the user. That is, the state transition determination unit 206 determines that the user 905 is ready when there are no obstacles in the play area 906 and at least one of the user's position, line of sight, posture, and voice satisfies the conditions for determining that the user 905 is ready.

[0100] In this embodiment, when both users 905, 908 are ready, the state transition determination unit 206 determines that the state of the VR content can be transitioned, and the process proceeds to step S1008. On the other hand, when either one of the users 905, 908 is not ready, the state transition determination unit 206 determines that the state of the VR content cannot be transitioned, and the process proceeds to step S1009. That is, the state transition determination unit 206 acquires information from the HMD 101a as to whether the user 908 is ready, and determines whether the state of the VR content can be transitioned based on the acquired information.

[0101] Next, in step S1008, the state transition determination unit 206 transitions the state of the VR content in accordance with the result of the determination in step S1007. For example, the state transition is from an unworkable state to a workable state.

[0102] Next, in step S1009, the notification unit 207 transmits information indicating the state of the VR content to the operation panel 902, and causes the operation panel 902 to display a notification screen for the user 903 according to the received information. Hereinafter, the operations of the notification unit 207 and the operation panel 902 in step S1009 will be described with reference to Figs. 9 and 11.

[0103] FIG. 11 shows an example of a notification screen displayed on the operation panel 902 visually recognized by the user 903. In FIG.

[0104] As shown in FIG. 11, a notification screen including a start button 1103 and a stop button 1104 is displayed on the operation panel 902 of the maintenance device 901 .

[0105] The start button 1103 is a user interface for starting the maintenance device 901 , and the stop button 1104 is a user interface for stopping the maintenance device 901 .

[0106] If the state transition determination unit 206 transitions the state of the VR content to a workable state in step S1008, the notification unit 207 transmits information indicating that the VR content has entered a workable state to the operation panel 902 in step S1009. Upon receiving this information, the operation panel 902 activates the start button 1103 (a state selectable by the user).

[0107] On the other hand, if the state transition determination unit 206 keeps the state of the VR content as the work unavailable state in step S1008, the notification unit 207 sends information to the effect that the work unavailable state remains to the operation panel 902 in step S1009. When the operation panel 902 receives this information, it makes the start button 1103 inactive (user selectable state).

[0108] At this time, the operation panel 902 may change the display method of the start button 1103 depending on whether the operation panel 902 is in an active state or an inactive state. Also, when the start button 1103 is in an active state, the user 903 can start the maintenance device 901 by pressing the start button 1103.

[0109] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The present invention can also be realized by multiple processors working together to perform processing.

[0110] Furthermore, all of the above-described embodiments are illustrative of the present invention and are not limiting, and the present invention can be implemented in various other modified and altered forms.

[0111] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0112] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An information processing device that displays a virtual space on a head-mounted display worn by a user, comprising: a play area acquisition means that acquires the user's play area as a first play area; an obstacle detection means that detects obstacles in real space within the first play area and generates obstacle information indicating the positional relationship between the detected obstacles and the first play area; a user information acquisition means that acquires user information from the user; a state transition determination means that determines whether the content displayed in the virtual space can transition to the next state based on the obstacle information and the user information; and a notification means that issues a notification based on the determination result of the state transition determination means. (Configuration 2) The information processing device according to configuration 1, wherein the user information includes at least one of the user's position, line of sight, posture, and voice. (Configuration 3) The information processing device described in Configuration 2, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the position of the obstacle is present within the first play area. (Configuration 4) The information processing device described in configuration 2 or 3, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the user's position is included in the first play area. (Configuration 5) An information processing device described in any one of configurations 2 to 4, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether at least one of the user's position and posture is stationary in the first play area. (Configuration 6) An information processing device described in any one of configurations 2 to 5, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the user's gaze is directed toward a pre-specified target in the virtual space. (Configuration 7) The information processing device according to configuration 6, wherein the state transition determination means determines an object at which the user's gaze is directed based on the direction of the user's gaze. (Configuration 8) The information processing device according to Configuration 6, wherein the state transition determination means determines an object at which the user's gaze is directed based on the user's gaze direction and gaze duration. (Configuration 9) An information processing device described in any one of configurations 2 to 8, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the user's posture conforms to a pre-specified posture. (Configuration 10) An information processing device described in any one of configurations 2 to 9, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the user's voice contains a pre-specified word. (Configuration 11) The information processing device according to any one of configurations 1 to 10, wherein the notification means displays a message for the user in the virtual space depending on the determination result of the state transition determination means. (Configuration 12) An information processing device described in any one of configurations 2 to 10, further comprising a communication means for communicating with another information processing device that displays the virtual space on another head-mounted display worn by another user, wherein the other information processing device comprises a second play area acquisition means for acquiring the play area of ​​the other user as a second play area, a second obstacle detection means for detecting obstacles in real space within the second play area and generating second obstacle information indicating the positional relationship between the detected obstacles and the play area, and a second user information acquisition means for acquiring second user information from the other user, wherein the state transition determination means acquires the second obstacle information and the second user information from the other information processing device via the communication means, and determines whether the content can transition to the next state from the obstacle information, the second obstacle information, the user information, and the second user information. (Configuration 13) The information processing device according to configuration 12, wherein the second user information includes at least one of the position, gaze direction, posture, and voice of the other user. (Configuration 14) The information processing device described in Configuration 13, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the user's voice and the other user's voice contain pre-specified words. (Configuration 15) An information processing device described in any one of configurations 12 to 14, characterized in that the communication means is further capable of communicating with an operation panel of a work device in real space operated by a third user, and the notification means sets a start button displayed on the operation panel for the third user to start the work device to either an active state or an inactive state depending on the judgment result of the state transition judgment means. (Configuration 16) The information processing device described in Configuration 15, characterized in that the head-mounted display displays an image of the real space as seen by the user superimposed on the virtual space, the other head-mounted display displays an image of the real space as seen by the other user superimposed on the virtual space, and the state transition determination means determines whether or not a state transition of the content is possible depending on whether or not the user's line of sight is directed toward a first work area on the work device and whether or not the other user's line of sight is directed toward a second work area on the work device. (Configuration 17) The information processing device described in Configuration 16, characterized in that the state transition determination means determines the object at which the gaze of the user and the other user is directed based on the gaze direction of the user and the other user. (Configuration 18) The information processing device described in Configuration 16, characterized in that the state transition determination means determines the target at which the gaze of the user and the other users is directed based on the gaze direction and gaze time of the user and the other users. (Configuration 19) The information processing device according to any one of configurations 1 to 18, wherein the obstacle detection means detects the obstacle in a two-dimensional space in the horizontal direction. (Configuration 20) The information processing device according to any one of configurations 1 to 18, wherein the obstacle detection means detects the obstacle in a three-dimensional space in the horizontal and vertical directions. (Configuration 21) The information processing device according to configuration 20, wherein the obstacle detection means excludes spaces that the user does not enter from the play area. (Method 1) An information processing method for an information processing device that displays a virtual space on a head-mounted display worn by a user, comprising: a play area acquisition step of acquiring the user's play area as a first play area; an obstacle detection step of detecting obstacles in real space within the first play area and generating obstacle information indicating the positional relationship between the detected obstacles and the first play area; a user information acquisition step of acquiring user information from the user; a state transition determination step of determining whether the content displayed in the virtual space can transition to the next state based on the obstacle information and the user information; and a notification step of providing a notification based on the determination result in the state transition determination step. (Program 1) A program for causing a computer to function as each means of the information processing device described in any one of configurations 1 to 21. [Explanation of symbols]

[0113] 101,101a HMD 109 Communications Department 201 Play Area Acquisition Department 202 Obstacle detection unit 203 User position detection unit 204 User posture detection unit 205 User gaze detection unit 206 State transition determination unit 207 Notification Department 806 User voice detection unit

Claims

1. An information processing device that displays a virtual space on a head-mounted display worn by a user, a play area acquisition means for acquiring the user's play area as a first play area; an obstacle detection means for detecting an obstacle in real space within the first play area and generating obstacle information indicating a positional relationship between the detected obstacle and the first play area; user information acquisition means for acquiring user information from the user; a state transition determination means for determining whether the content displayed in the virtual space can transition to a next state based on the obstacle information and the user information; a notification means for issuing a notification according to a determination result of the state transition determination means; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the user information includes at least one of the user's position, line of sight, posture, and voice.

3. The information processing device according to claim 2, wherein the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the position of the obstacle is present within the first play area.

4. The information processing device according to claim 2, wherein the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the user's position is included in the first play area.

5. The information processing device described in claim 2, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether at least one of the user's position and posture is stationary in the first play area.

6. The information processing device according to claim 2, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the user's line of sight is directed toward a pre-specified target in the virtual space.

7. 7. The information processing apparatus according to claim 6, wherein the state transition determining means determines an object at which the user's gaze is directed based on the direction of the user's gaze.

8. 7. The information processing apparatus according to claim 6, wherein the state transition determining means determines an object at which the user's gaze is directed based on the user's gaze direction and gaze duration.

9. 3. The information processing apparatus according to claim 2, wherein the state transition determining means determines whether or not the state transition of the content is possible depending on whether or not the user's posture conforms to a pre-specified posture.

10. 3. The information processing apparatus according to claim 2, wherein the state transition determining means determines whether or not the state transition of the content is possible depending on whether or not a pre-specified word is included in the user's voice.

11. 2. The information processing apparatus according to claim 1, wherein said notification means displays a message for said user in said virtual space in accordance with the result of the determination made by said state transition determination means.

12. further comprising a communication means for communicating with another information processing device that displays the virtual space on another head-mounted display worn by another user; The other information processing device is a second play area acquisition means for acquiring the play area of ​​the other user as a second play area; a second obstacle detection means for detecting an obstacle in real space within the second play area and generating second obstacle information indicating a positional relationship between the detected obstacle and the play area; second user information acquisition means for acquiring second user information from the other user; Equipped with The state transition determination means acquiring the second obstacle information and the second user information from the other information processing device by the communication means; The information processing apparatus according to claim 2 , wherein it is determined whether the content can transition to the next state based on the obstacle information, the second obstacle information, the user information, and the second user information.

13. The information processing apparatus according to claim 12 , wherein the second user information includes at least one of the position, line of sight, posture, and voice of the other user.

14. The information processing device according to claim 13, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not a pre-specified word is included in the voice of the user and the voice of the other user.

15. the communication means is further capable of communicating with an operation panel of a work device that is in real space and is operated by a third user; 13. The information processing device according to claim 12, wherein the notification means sets a start button displayed on the operation panel for the third user to start the operating device to either an active state or an inactive state depending on the judgment result of the state transition judgment means.

16. the head-mounted display displays an image of the real space seen by the user superimposed on the virtual space; the other head-mounted display displays an image of the real space as seen by the other user in a superimposed manner on the virtual space; The information processing device according to claim 15, characterized in that the state transition determination means determines whether or not the state transition of the content is possible depending on whether or not the user's line of sight is directed toward a first work area on the work device and the other user's line of sight is directed toward a second work area on the work device.

17. 17. The information processing apparatus according to claim 16, wherein the state transition determination means determines an object at which the gazes of the user and the other user are directed based on the gaze directions of the user and the other user.

18. 17. The information processing apparatus according to claim 16, wherein the state transition determination means determines an object at which the gazes of the user and the other user are directed based on the gaze directions and gaze durations of the user and the other user.

19. 2. The information processing apparatus according to claim 1, wherein said obstacle detecting means detects said obstacle in a horizontal two-dimensional space.

20. 2. The information processing apparatus according to claim 1, wherein said obstacle detecting means detects said obstacle in a three-dimensional space in horizontal and vertical directions.

21. 21. The information processing apparatus according to claim 20, wherein the obstacle detection means excludes from the play area any space that the user does not enter.

22. An information processing method of an information processing device that displays a virtual space on a head-mounted display worn by a user, comprising: a play area acquisition step of acquiring the user's play area as a first play area; an obstacle detection step of detecting an obstacle in real space within the first play area and generating obstacle information indicating a positional relationship between the detected obstacle and the first play area; a user information acquisition step of acquiring user information from the user; a state transition determination step of determining whether the content displayed in the virtual space can transition to a next state based on the obstacle information and the user information; a notification step of issuing a notification according to a determination result in the state transition determination step; An information processing method comprising:

23. A program for causing a computer to function as each of the means of the information processing device according to claim 1.

Images