Display device, display control method, and display system

Abstract

This display device is provided with a sensor for acquiring direction information related to a direction in which a person different from a user of the display device moves, a storage device for storing activity information related to movement of the user, and a calculation device for performing a process on the basis of the direction information and information related to the activity information. The calculation device determines whether the user and the person interfere with each other on the basis of the direction information and the activity information, and outputs movability information related to whether or not the person is able to move.

Description

Display device, display control method, and display system

[0001] The present invention relates to a display device, a display control method, and a display system.

[0002] There is known a display device that allows a user to experience VR (Virtual Reality) content using a head-mounted display (also referred to as a head-mounted display or the like; hereinafter, referred to as an HMD (Head Mounted Display)).

[0003] Patent Document 1 discloses a game device including an input processing unit that acquires position information of a player moving in a play area or the like, a game processing unit that performs game processing, and a display processing unit that performs processing for displaying a game image on a head-mounted display device. When the game processing unit detects that the player approaches or reaches the boundary of an area set as the movement range of the player in the play area or within the game device housing, the game processing unit generates a first event that performs a suppression process to suppress the movement of the player in the direction toward the boundary. When it is detected that the player has moved beyond the boundary of the area, the game processing unit generates a second event that performs at least one of a process of changing to a different image display from the game image, a process of changing the game progress mode, and a process of ending the game (summary of Patent Document 1, etc.). [[ID=)]

[0004] Patent Document 2 discloses an obstacle avoidance device, focusing on "an obstacle detection unit that detects an obstacle for a user wearing a head-mounted display from an image of the outside world, a distance calculation unit that calculates the distance from the detected obstacle to the wearer of the head-mounted display, an obstacle replacement unit that replaces the detected obstacle with a virtual object, and a virtual object synthesis unit that synthesizes the virtual object at a position corresponding to the distance to the obstacle within the virtual space displayed on the head-mounted display (summary of Patent Document 2, etc.).

[0005] Japanese Unexamined Patent Application Publication No. 2017 - 119031, Japanese Unexamined Patent Application Publication No. 2013 - 257716

[0006] However, in known examples, when a user is experiencing VR content, the system displays virtual objects unrelated to the VR content or interrupts the VR content, leaving the user's VR experience hindered.

[0007] The present invention aims to provide a display device, etc., that provides movable information to an external person (hereinafter referred to as "external person") in the relationship between the user and the external person, thereby avoiding interference with the external person without interfering with the user's VR content experience.

[0008] To solve the above problems, the present invention comprises the configuration described in the claims. For example, a display device comprises a sensor that acquires direction information relating to the direction of movement of a person other than the user of the display device; a storage device that stores activity information relating to the user's movement; and a computing device that processes information based on the direction information and the activity information. The computing device determines whether the user and the person interfere with each other based on the direction information and the activity information, and outputs movable information relating to whether the person can move.

[0009] According to the present invention, it is possible to avoid interference with external individuals without interfering with the user's VR content experience. Further objectives, configurations, and effects will be revealed in the following embodiments.

[0010] This is a diagram illustrating an example of the hardware configuration of HMD1. This is a diagram showing an example of the external appearance of HMD1. This is a diagram schematically showing an example of adding a network and a VR service server to a diagram of the three-dimensional spatial relationship information between user 2 and external person 3. This is a two-dimensional plane view of Figure 3. This is a diagram showing an example of the activity level of user 2. This is a diagram showing an example of processing based on movement. This is a diagram showing a first output example of movable information. This is a diagram showing a second output example of movable information. This is a diagram showing an example of processing by the movable information determination program 26. This is a diagram showing an example of setting multiple areas in the second embodiment. This is a diagram showing an example of output of movable information in the second embodiment. This is a diagram showing an example of processing by the movable information determination program 26 in the second embodiment. This is a diagram showing an example of the third embodiment. This is a diagram showing an example of the third embodiment in a two-dimensional plane. This is a diagram showing an example of processing based on movement in the third embodiment. This is a diagram showing an example of output of movable information in the third embodiment. This is a diagram showing an example of processing by the movable information determination program 26 in the third embodiment. This is a diagram showing an example of the functional configuration.

[0011] The display device, display control method, and display system according to this embodiment aim to avoid interference with external individuals without interfering with the user's VR content experience. Therefore, since the present invention can enhance the commercial value of information processing equipment to which the present invention is applied, it is expected to contribute to Sustainable Development Goal 8.2 (Increase economic productivity through diversification, technological advancement, and innovation, particularly in industries that enhance the value of goods and services and in labor-intensive industries).

[0012] Embodiments of the present invention will be described below with reference to the drawings. The same components are denoted by the same reference numerals throughout the drawings, and redundant explanations are omitted.

[0013] In the following explanation, the display device will be assumed to be "HMD1".

[0014] <First Embodiment> Figure 1 is a diagram illustrating an example of the hardware configuration of HMD1. For example, HMD1 includes a camera 10, a distance measuring sensor 11, a display 12, a speaker 13, a microphone 14, a sensor group 15, a communication device 16, and a bus 17. Furthermore, HMD1 includes a communication device 20, a processing unit 21, a memory 22, and a storage 23, etc.

[0015] Camera 10 captures images of the user's surroundings and generates image data.

[0016] The distance measuring sensor 11 is a sensor that measures the distance to people or objects such as furniture in the vicinity of the user, using the user as the reference point.

[0017] Display 12 is an example of an output device that displays image data.

[0018] Speaker 13 is an example of an output device that outputs sound.

[0019] Microphone 14 is an example of an input device for receiving audio.

[0020] The sensor group 15 is a collection of various sensors. For example, the sensor group 15 senses acceleration, position, or light, and converts it into data.

[0021] Bus 17 transmits and receives data on the HMD1.

[0022] The communication device 16 communicates with an external device. For example, the communication device 16 sends and receives data to and from an external device using a Local Area Network (LAN) via wired or wireless connection. The communication device 16 also performs short-range communication with an external device using infrared light or Bluetooth®, etc.

[0023] The arithmetic unit 21 executes processing based on a program or the like. For example, the arithmetic unit 21 is a Central Processing Unit (CPU), etc.

[0024] Memory 22 is the main memory. For example, memory 22 stores data or programs used by the arithmetic unit 21 for processing. For example, memory 22 is Random Access Memory (RAM), etc.

[0025] Storage 23 is an auxiliary storage device. Specifically, storage 23 is a Read Only Memory (ROM) or a hard disk, etc. For example, storage 23 stores the basic operation program 24, the VR experience program 25, the movable information determination program 26, etc.

[0026] The hardware of HMD1 is not limited to the configuration described above. For example, HMD1 may be equipped with an external or internal arithmetic unit, control unit, memory device, input device, output device, or auxiliary device in addition to those described above.

[0027] The basic operation program 24, the VR experience program 25, and the movable information determination program 26 are loaded into memory 22 and executed by the arithmetic unit 21.

[0028] The basic operation program 24, the VR experience program 25, and the mobility information determination program 26 are, in principle, separate programs, but are not limited to this. For example, one program may include the basic operation function, the VR experience function, and the mobility information function, or they may be separate programs.

[0029] The user puts on the HMD1. After putting it on, when the user experiences VR content using the VR experience program 25, the camera 10 captures the real space in front of the HMD1.

[0030] If camera 10 consists of multiple cameras, or if it is a 360-degree camera, camera 10 may capture images of the 360-degree surroundings of HMD1. In this case, the computing unit 21 detects real objects such as furniture or external people from the image data captured by camera 10.

[0031] The distance measuring sensor 11 obtains the distance to the detected real object and the external person, and obtains three-dimensional spatial relationship information of the real space. Furthermore, the camera 10 and the distance measuring sensor 11 obtain the movement of the external person based on the amount of change in the external person's position over time.

[0032] The display 12 displays images of the VR content, and the speaker 13 outputs the sound of the VR content. If necessary, the microphone 14 acquires the user's voice.

[0033] The sensor group 15 senses, for example, the user's up, down, forward, backward, left, and right movements, as well as rotational movements that involve changes in the direction of their gaze. The user's movements are recorded as historical information, etc., measured by the sensors, within the HMD 1 or on a VR service server (Figure 3), etc.

[0034] Furthermore, the camera 10 may be equipped with infrared illumination and a device for detecting infrared light, taking into consideration shooting in dark areas. In addition, the distance measuring sensor 11 may measure the real space in three dimensions using laser light, such as with LiDAR, in order to improve detection accuracy.

[0035] The communication device 20 is equipped with multiple communication protocols, such as Internet communication or proximity communication, and is used according to the purpose.

[0036] The VR experience program 25 uses internet communication to receive VR content and allows the user to experience the VR content. Furthermore, the communication device 20 communicates with information terminals carried by external individuals via proximity communication, etc., and sends and receives data such as mobile information.

[0037] Furthermore, the processing may be partially or entirely performed on a server on the Internet via the communication device 20, or on an information processing device connected by communication.

[0038] Figure 2 shows an example of the appearance of HMD1. Although HMD1 is described as an immersive type, it may also be a transparent type. With an immersive type, the user is less likely to notice external people or other objects in their surroundings compared to a transparent type, so measures to avoid interference become even more effective.

[0039] In the transparent type, for example, the display 12 is transparent, allowing the user to see the outside world, and the VR display is superimposed on the external scenery. On the other hand, in the immersive type, the transparency of the display 12 is low, and the user cannot see the outside world while wearing the HMD 1.

[0040] Hereinafter, components identical to those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted.

[0041] The display 12 includes a plurality of display elements for the left eye and the right eye as display elements, and displays a left-eye image and a right-eye image. Through such display, the display 12 enables the user to recognize a three-dimensional image. Note that the display may adopt other methods as long as it can perform three-dimensional display.

[0042] The wearing cases 18a and 18b are cases for the user to wear the HMD 1. Note that the wearing cases 18a and 18b are not limited to the illustrated shapes, and any shape that allows the user to wear and includes hardware is acceptable.

[0043] FIG. 3 is a diagram schematically showing an example in which a network and a VR service server are added to a diagram of three-dimensional space relationship information between the user 2 and an external person 3 (hereinafter referred to as "three-dimensional space relationship diagram"). Hereinafter, components identical to those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted.

[0044] In the example shown in FIG. 3, an example is shown in which the external person 3 is around the user 2.

[0045] The information processing terminal 4 is a smartphone or a wearable terminal owned by the external person 3. The information processing terminal 4 outputs to earphones 5 or the like worn by the external person >3.

[0046] The HMD 1 captures the surrounding environment such as the room or facility during play with the camera 10 to obtain a three-dimensional space relationship diagram.

[0047] The access point 6a is used for the surrounding information processing devices to perform Internet communication.

[0048] The network 6b is a communication network to which the information processing device can be connected via the access point >6a. For example, via the network 6b, the information processing device can be connected to an external device such as the VR service server 6c.

[0049] The VR service server 6c provides VR content and other information processing devices connected via the network 6b. Specifically, as shown in the 3D spatial relationship diagram, the VR service server 6c provides VR content services to the HMD 1 and other devices.

[0050] On the other hand, let's consider an example where user 2 has physical objects around him, such as a bookshelf 30, a television 31, and a smart speaker 32.

[0051] User 2 receives VR content from the VR service server 6c via access point 6a and network 6b using the HMD 1 they are wearing, and then experiences the VR content.

[0052] Network 6b may include an intranet, etc. Furthermore, the VR service server 6c may be an external device such as a PC (Personal Computer) connected via direct communication. Also, HMD1 may be configured to allow user 2 to experience VR content that has been stored in HMD1 in advance.

[0053] The bookshelf 30, television 31, and smart speaker 32 are examples of fixed real objects in real space. For example, other objects such as a sofa or a table may be around user 2.

[0054] These real-world objects are those that should avoid interfering with User 2 while User 2 is experiencing the VR content.

[0055] The play area 33 is set as an area to avoid these real-world objects. If user 2 moves outside the play area 33, the HMD 1 interrupts the VR content experience and prompts user 2 to avoid going outside the play area 33 or to avoid colliding with real-world objects such as walls. In addition, if user 2 has gone outside the play area 33, the HMD 1 prompts user 2 to return to the area.

[0056] Furthermore, in the area immediately preceding the user to leave the play area 33, the HMD1 may encourage the user to avoid contact by superimposing a warning VR image onto the VR content.

[0057] Figure 4 is a two-dimensional plane view of Figure 3. The two-dimensional schematic diagram shown in Figure 4 represents the same space as the entire three-dimensional space shown in Figure 3, but from a different viewpoint. Hereafter, the same reference numerals are used for the same components as in Figure 3, and their explanations are omitted.

[0058] Living room 34 is an example of the overall space. Note that the use of HMD1 is not limited to living room 34; it can be used in other spaces as well.

[0059] Furthermore, the two-dimensional schematic diagram may be a two-dimensional schematic diagram of a partial region, which includes external figures and the user.

[0060] For example, a two-dimensional schematic diagram like the one shown in Figure 4 is displayed on an HMD1 or similar device.

[0061] <Example of User 2's Activity Level> Figure 5 shows an example of User 2's activity level. For example, when VR content is displayed by HMD1, User 2 performs various movements according to the VR content, such as moving, stepping in place, moving hands or arms, or moving their head. For example, User 2's activity level changes over time, as shown in Figure 5.

[0062] In Figure 5, the vertical axis represents the activity level of user 2 experiencing the VR content, and the horizontal axis represents time.

[0063] The point at which the VR content starts is defined as "Current Time T0". In Figure 5, the time from Current Time T0 onward represents the activity level of User 2 predicted from the content of the VR content.

[0064] For example, if VR content is a game where user 2 adventures in a virtual world, and user 2 virtually experiences the adventure according to the game's scenario, then if the scenario prompts user 2 to move, then user 2's activity level is expected to be high. On the other hand, if the scenario involves scenes where user 2 stands still and converses, then user 2's activity level is expected to be low.

[0065] As another example, if the VR content is sports or a fighting game, it is predicted that User 2's activity level will be high during a match or while playing. However, it is predicted that User 2's activity level will be low during breaks such as halftime, between sets, or between rounds.

[0066] Threshold 40a is a reference value set to determine whether external person 3 is able to move. For example, if user 2's activity level is greater than threshold 40a, it is expected that user 2 will move around or move their limbs during that period. Therefore, if external person 3 moves during this period, there is a high possibility of interference with user 2. For this reason, it is not a suitable period for external person 3 to move.

[0067] On the other hand, if User 2's activity level is low compared to threshold 40a, it is predicted that User 2 will not move or move their limbs during that period, or if they do, the movements will not be large. During this period, even if External Person 3 moves, the likelihood of interference with User 2 is low. Therefore, this is a suitable period for External Person 3 to move.

[0068] The threshold 40a is initially set by storing its initial value in storage 23 or storage located on network 6b. Furthermore, the threshold 40a may be adjusted to any value by user 2 or external person 3.

[0069] However, if the threshold 40a is set too high, interference will be more likely to occur, so for example, the adjustable range of the threshold 40a may be limited.

[0070] Furthermore, the threshold 40a may be set to a different value for each piece of content being played on the HMD1. In other words, different values ​​for the threshold 40a may be prepared separately depending on the type of content, and the threshold 40a may be set after determining the type of content.

[0071] In the example shown in Figure 5, the period from the first time point T1 to the second time point T2 is a period in which the activity level of user 2 is low compared to the threshold 40a. Therefore, the period from the first time point T1 to the second time point T2 is determined to be a period in which external person 3 can move or is suitable for moving. For example, in a game, in quiet scenes with little user movement, such as during breaks, the activity level of user 2 tends to be low, as in the period from the first time point T1 to the second time point T2.

[0072] On the other hand, the period before the first time point T1 is a period in which user 2's activity level is high compared to the threshold 40a. Similarly, the period from the second time point T2 onward is a period in which user 2's activity level is high compared to the threshold 40a. Therefore, both the period before the first time point T1 and the period from the second time point T2 onward are determined to be periods in which external person 3 is unable to move or is unsuitable to move. For example, in a game, in intense scenes involving user movement, such as combat scenes, user 2's activity level tends to be high, as in the period before the first time point T1 or from the second time point T2 onward.

[0073] User 2's activity level is a value stored as history when User 2 experiences the same or similar VR content. Alternatively, User 2's activity level may be calculated from the amount of movement or speed when User 2 previously experienced the same VR content.

[0074] Furthermore, User 2's activity level may be estimated from data obtained from the movements of a third party. For example, if the activity levels of a third party (who has the same or similar user parameters as User 2, such as gender, age group, height, or proficiency in the content) are selected, User 2's activity level can be set with high accuracy.

[0075] User 2's activity level is stored, for example, in the VR service server 6c, associated with content.

[0076] Figure 6 shows an example of motion-based processing. For example, the movements of user 2 are measured by a sensor and stored in advance as historical information. Specifically, the historical information is stored as shown in Figure 6(a) or Figure 6(b). The following explanation assumes that user 2 moved in the X-Y plane over time (the position in the Z-axis direction is assumed to be constant). However, the historical information may also be stored in three dimensions.

[0077] The trajectory information 41 is information that records the positions where user 2 moved, corresponding to time. For example, the trajectory information 41 is recorded in a format that associates (X, Y) coordinates with time. In this way, movement is used as data in a format such as the trajectory information 41.

[0078] Furthermore, the activity level of user 2 may be analyzed based on the movements indicated by the trajectory information 41, etc. For example, if the analysis based on the trajectory information 41 indicates that user 2 moved quickly or performed movements with a high load, it will be determined that user 2's activity level is high. Note that the trajectory information 41 is not limited to location information; for example, biometric information measured by the HMD 1 with sensors or acquired from an external device may be associated with and stored. In other words, the activity level may be determined by considering, for example, the acceleration, vibration, heart rate, pulse, or heat generation of a part of user 2's body.

[0079] Range 42 is the range that encompasses the movement indicated by the trajectory information 41. For example, range 42 is generated by adding a margin to the movement indicated by the trajectory information 41.

[0080] Specifically, the range 42 is generated by calculating the average, maximum, minimum, or standard deviation of the movement indicated by the trajectory information 41 to identify the position where user 2 moves, and then adding a margin. However, as shown in the figure, the range 42 may be replaced with a simple shape such as a rectangle, circle, or square, as long as it encompasses the movement. In this way, setting the range 42 with a simple shape reduces the processing burden.

[0081] Furthermore, the range 42 may be an estimate. For example, if user parameters such as gender, age group, height, or proficiency level in the content are entered, the range 42 may be set to an area within a radius of a predetermined distance (for example, set to 1 meter) centered on the position of user 2, according to the user parameters.

[0082] The range 42 may be set according to the results of an analysis of user 2's movements over a certain period. For example, during periods of low activity, such as the period from the first time point T1 to the second time point T2, the range 42 is set to be narrow. On the other hand, during periods of high activity, such as periods other than the first time point T1 to the second time point T2, the range 42 is set to be narrow. The width of the ranges for periods of low activity and periods of high activity can be set in advance, for example.

[0083] By identifying periods of low activity and narrowing the range 42 compared to periods of high activity, it can be determined that the narrower range 42 is suitable for external person 3 to move near user 2, or that the possibility of external person 3 and user 2 interfering with each other is relatively low.

[0084] Directional information 40 is information indicating the direction in which the external person 3 is moving. For example, directional information 40 is generated based on the movement of the external person 3 over time. In other words, assuming that the external person 3 will continue to move in the direction in which it moved within the most recent predetermined time, the direction in which the external person will move is estimated and used as directional information 40.

[0085] Furthermore, the direction information 40 may include not only the direction but also information on the speed at which the external person 3 is moving. For example, the speed may be calculated by dividing the distance traveled by the external person 3 in the past by the time. Also, the speed is not limited to measured data, but may also be statistical values ​​from a third party.

[0086] The speed and direction of movement may have a margin of error. For example, the speed of movement may be calculated at a slower speed, assuming that it will take longer than the actual measurement result. Also, the direction may have a range of angles, taking into account that it may deviate from the direction of movement in the past.

[0087] The trajectory information 41 and the direction information 40 may be generated, for example, by optical flow.

[0088] The range 42 is set according to the movements of User 2 that occur during the time it takes for the external person 3 to pass through the play area 33 from their current location, or for the external person 3 to overtake User 2, as shown in Figure 6.

[0089] The time spent by external character 3 is often short compared to the total time spent displaying the VR content. Therefore, the range 42 is often set to be part of the play area 33.

[0090] In Figure 6(a), the range 42 lies on the extension of the direction information 40 indicating the direction of movement of the external person 3 (for example, in Figure 6(a), it is the dotted line extending from the arrow indicated by the direction information 40). Therefore, if the external person 3 continues to move according to the direction information 40, the movement of user 2 and the external person 3 will overlap, and there is a high possibility that user 2 and the external person 3 will interfere with each other.

[0091] On the other hand, in Figure 6(b), the range 42 is small and does not lie on the extension of the direction information 40 indicating the direction of movement of the external person 3 (for example, the dotted line extending from the arrow indicated by the direction information 40 in Figure 6(b)). Therefore, even if the external person 3 continues to move, the possibility of interference between the user 2 and the external person 3 is low.

[0092] As described above, in the case of Figure 6(a), there is a high possibility of interference between User 2 and External Person 3, so External Person 3 is judged to be "unable to move" or "not a suitable time to move". On the other hand, in the case of Figure 6(b), User 2's movement and External Person 3 do not overlap, and there is a low possibility of interference between User 2 and External Person 3, so External Person 3 is judged to be "able to move" or "suitable time to move".

[0093] <Example of Outputting Movable Information> Figure 7 shows a first example of outputting movable information. The following explanation will describe an example of outputting movable information on the display of the information processing terminal 4. However, the output destination may be another output device provided by the information processing terminal 4, or an output device other than the information processing terminal 4, as long as it can be transmitted to an external person 3.

[0094] Figure 7(a) shows an example of mobility information indicating that movement is permitted. Specifically, when it is determined that an external person 3 is mobile, the information processing terminal 4, etc., outputs information to the external person 3 indicating that it is permitted to move, using a graphic symbol such as "GO" (hereinafter referred to as "first graphic symbol 50").

[0095] Furthermore, the movable information indicated by the symbol 50 in Figure 1 is a suggestion to the external person 3 to move, not a sign that compels or instructs them to move, nor is it intended to completely guarantee the safety of their movement. Therefore, even if movable information is displayed, it is desirable for the external person 3 to confirm their own safety before moving.

[0096] Figure 7(b) shows an example of movable information indicating that movement is impossible. Specifically, when it is determined that the external person 3 is immobile, the information processing terminal 4, etc., outputs information to the external person 3 using a graphic symbol such as "STOP" (hereinafter referred to as "second graphic symbol 51"), indicating that movement is not recommended and that the person should stop.

[0097] Furthermore, it is desirable that the mobility information be output not based on an inquiry from the information processing terminal 4, i.e., an operation by an external person 3, but rather when conditions are met, such as when an external person 3 approaches user 2 or the play area 33.

[0098] Furthermore, the movable information may be output in ways other than the first graphic symbol 50 and the second graphic symbol 51. For example, the movable information may be further displayed as supplementary information in the form of text or icons. Specifically, for example, Figure 7(a) may include the message, "After confirming that it is safe, proceed carefully." On the other hand, Figure 7(b) may display supplementary information such as, "Stop and carefully check your surroundings."

[0099] Figure 8 shows a second example of the output of movable information. Hereafter, as with Figure 7, the explanation will describe an example of outputting movable information on the display of the information processing terminal 4. Components similar to those in Figure 7 are denoted by the same reference numerals, and their explanations are omitted.

[0100] Figures 8(a) and 8(b) differ from Figures 7(a) and 7(b) in that a time display has been added.

[0101] The first time display 53 indicates that there are approximately 10 seconds remaining before the vehicle can be moved.

[0102] The second time display 54 indicates that there are approximately 15 seconds remaining before the object becomes immobile.

[0103] The first time display 53 and the second time display 54 may also be in a countdown format, where the value decreases as time progresses. When there is no time remaining, i.e., 0 seconds, the graphic symbol indicating either movable or immovable switches. From then on, the first time display 53 and the second time display 54 are redisplayed in the switched state.

[0104] Furthermore, audio associated with the state may be output. Also, when the remaining time is running low, an emphasis display may be shown. For example, when there are three seconds or less remaining, the display color of the first time display 53 and the second time display 54 may be changed, the display size may be increased, or decorations such as flashing may be added, so that the information is accurately conveyed to the external person 3.

[0105] Figure 8(c) shows a progress bar 57 in addition to the first time display 53. That is, the progress bar 57 displays the remaining time for the movable state. Hereafter, Figure 8(c) will be used as an example of a movable state, similar to Figure 8(a).

[0106] For example, suppose the first hour display 53 is set to start at 10 seconds. At the start, the progress bar 57 is not filled in. Next, as time passes, the filled portion of the progress bar 57 increases. Finally, when the first hour display 53 reaches 0 seconds, the progress bar 57 is completely filled in.

[0107] Alternatively, the progress bar 57 may start with a fully filled display. In this case, as time passes, the filled portion of the progress bar 57 increases. Finally, when the first time display 53 reaches 0 seconds, the progress bar 57 will be completely empty.

[0108] Note that the only movable information displayed may be the progress bar 57.

[0109] Figure 8(d) shows a message displayed separately from the second time display 54. Hereafter, Figure 8(d) will be used as an example to show a state where movement is impossible, similar to Figure 8(b).

[0110] Specifically, the situation may be notified to the external person 3 by a message that includes text, such as the first message 55 and the second message 56.

[0111] If the device is immobile, and this immobile state lasts for a predetermined period of time (set to 3 minutes in the diagram), information indicating that communication is taking place between the information processing terminal 4 and the HMD1 may be displayed.

[0112] For example, in the example shown in Figure 8(d), external person 3 is unable to move for more than three minutes. In this case, external person 3 may not be able to wait for the length of time. In such a case, for example, external person 3 notifies HMD1 via information processing terminal 4 that they will move. After this notification, external person 3 begins to move. Such notifications may be sent and received.

[0113] The second message 56 is a message indicating that the information processing terminal 4 is sending a notification to the HMD 1. However, the HMD 1 may be able to know the status even without a notification. In such cases, it is not necessary for the information processing terminal 4 to send a notification to the HMD 1. The HMD 1 then instructs the information processing terminal 4 to output the second message 56, such as "Sending notice". Based on this instruction, the information processing terminal 4 may display the second message 56.

[0114] On the other hand, the HMD1 displays a message in a place easily noticeable to user 2 (for example, a display on the display 12) indicating that an external person 3 wishes to move.

[0115] The second message 56 displays a message such as "Wait a moment" to the external person 3 if user 2 continues the VR content experience. On the other hand, if user 2 pauses the VR content experience, the information processing terminal 4 switches to a movable display such as Figure 8(a).

[0116] Furthermore, there are cases where user 2 remains unable to move without interrupting the VR content experience, or where it is unclear how long this state will last, and external person 3 moves. In such cases, the second message 56 may output advice such as "Move Carefully".

[0117] For example, the second time display 54 is set based on the activity level, etc. Specifically, the second time display 54 corresponds to the time between the first and second time points in Figure 5.

[0118] As described above, by outputting detailed information such as time, external person 3 can decide whether or not to move based on the time and other information. In addition, if time is displayed, external person 3 can understand the waiting time.

[0119] <Example of a program for determining whether a program is mobile> Figure 9 shows an example of processing by the mobile information determination program 26.

[0120] In step S10, the HMD1 starts processing by the movable information determination program 26.

[0121] In step S11, the HMD1 generates a three-dimensional spatial relationship diagram of the real space around user 2 based on the captured image data and the measured distance data.

[0122] In step S12, HMD1 generates a two-dimensional schematic diagram from the three-dimensional spatial relationship diagram.

[0123] In step S13, the HMD1 sets the play area 33. For example, the play area 33 is set to avoid objects and other things that exist in the real world.

[0124] The setting of play area 33 is notified to the VR experience program 25. Then, user 2 begins experiencing the VR content through the VR experience program 25.

[0125] Note that the setting of play area 33 is not limited to the timing described above. For example, after generating a spatial relationship diagram of the area around user 2, user 2 begins experiencing the VR content. After starting, the 3D spatial relationship diagram is generated again in accordance with user 2's movements. In this way, a method of gradually generating a spatial relationship diagram over a large area is also acceptable.

[0126] In step S14, the HMD1 confirms that it is experiencing VR content. For example, the VR experience program 25 and the mobility information determination program 26 are independent processes and are executed in parallel.

[0127] In step S15, the HMD 1 determines whether or not an external person 3 is in the vicinity of user 2. For example, the HMD 1 detects the external person 3 using a sensor such as a camera 10, or by short-range communication.

[0128] Furthermore, when detecting an external person 3 using image data, it is desirable for the HMD 1 to register image data of people who are likely to be in the vicinity in advance. People who are likely to be in the vicinity include, for example, people living together or family members. In addition, it is desirable for the HMD 1 to associate the information processing terminal 4 that the person being registered possesses with the registered image data. With such registration, external person 3 can be detected with high accuracy. Also, by associating and registering the information processing terminal 4, it becomes possible to efficiently establish communication between the HMD 1 and the information processing terminal 4.

[0129] In step S16, HMD1 evaluates the content of the VR content. Then, HMD1 identifies the activity level of user 2. For example, the activity level of user 2 is identified in the format shown in Figure 5.

[0130] In step S17, HMD1 determines whether user 2's activity level is high or low. If it determines that user 2's activity level is high (high in step S17), HMD1 proceeds to step S18. On the other hand, if it determines that user 2's activity level is low (low in step S17), HMD1 proceeds to step S21.

[0131] Whether User 2's activity level is high or low can be determined, for example, by a pre-set threshold. Alternatively, User 2's activity level may also be determined by, for example, the content of the VR content.

[0132] In step S18, HMD1 confirms the movements of user 2.

[0133] In step S19, HMD1 sets the range 42.

[0134] For example, steps S18 and S19 set the range 42 as shown in Figure 6. Preferably, the range 42 is set based on the movements of user 2.

[0135] In step S20, the HMD1 makes a "GO" or "STOP" decision. For example, as shown in Figure 6, if it is determined that the range 42 lies on the extension of the direction in which the external person 3 is moving, the HMD1 decides to "STOP". On the other hand, if it is determined that the range 42 does not lie on the extension of the direction, the HMD1 decides to "GO". It is desirable that the extension line be extended by a distance equal to the estimated travel time based on the speed at which the external person 3 is moving. In this way, the HMD1 determines whether the movement of user 2 and the external person 3 overlap.

[0136] In step S21, the HMD1 outputs mobility information. Specifically, the HMD1 outputs the "GO" or "STOP" determination result, which is determined in step S20, to the information processing terminal 4 of the external person 3. For example, the output is performed as shown in Figure 7 or Figure 8.

[0137] In step S22, the HMD1 determines whether or not the external person 3 has passed by. That is, the HMD1 determines whether or not the external person 3 has left the vicinity of user 2.

[0138] Next, if it is determined that external person 3 has passed through (YES in step S22), HMD1 proceeds to step S23. On the other hand, if it is determined that external person 3 has not passed through (NO in step S22), HMD1 proceeds to step S15.

[0139] In step S22, the HMD1 may be treated the same as if it had "passed through" (YES in step S22) if, for example, even if the external person 3 has passed through, it can be determined that the person has not approached the play area 33 for a certain period of time or has not moved for a certain period of time (for example, sitting or performing an action other than moving).

[0140] In step S23, HMD1 determines whether the termination condition has been met. For example, the termination condition is set in advance.

[0141] Next, if it is determined that the termination condition is met (YES in step S23), HMD1 proceeds to step S24. On the other hand, if it is determined that the termination condition is not met (NO in step S23), HMD1 proceeds to step S15.

[0142] The above example describes a case where the HMD1 detects the movement of an external person 3, but detection may be performed by other information processing devices or sensors. For example, the information processing terminal 4 held by the external person 3 may detect the movement of the external person 3 using a sensor that detects movement (e.g., an accelerometer). The HMD1 may then acquire the detection result from the information processing terminal 4 via proximity communication.

[0143] Similarly, the sensor may be installed in the living room 34, and the HMD 1 may acquire the detection result from the sensor via proximity communication. Furthermore, multiple data sets may be used for motion detection.

[0144] Other forms of mobile information may include sound, light, vibration, notifications other than those mentioned above, or combinations thereof.

[0145] Furthermore, the movable information may be output to a device other than the information processing terminal 4 held by the external person 3. For example, the output destination of the movable information may be the earphones 5, television 31, or smart speaker 32 worn by the external person 3.

[0146] As explained above, according to the HMD1 of the first embodiment, movement information can be output to the information processing terminal 4 held by the external person 3 even without a request from the external person 3. This reduces the possibility of the external person 3 interfering with user 2 without interfering with user 2's VR content experience. Therefore, the external person 3 can move without interfering with user 2.

[0147] <Second Embodiment> Figure 10 is a diagram showing an example of setting multiple regions in the second embodiment. The second embodiment differs from the first embodiment in that the region to be judged is divided and set into three regions: the first region 43, the second region 44, and the third region 45. Hereafter, the same components as in the first embodiment are denoted by the same reference numerals and their descriptions are omitted.

[0148] The first region 43 is set, for example, on the extension of the direction in which the external person 3 moves (the region is defined as the area within a certain angle to the left and right of the extension line).

[0149] The second region 44 and the third region 45 are both regions adjacent to the first region 43, with the second region 44 being the region set to the left as viewed from the perspective of the external person 3. Similarly, the third region 45 is the region set to the right as viewed from the perspective of the external person 3.

[0150] In the example shown in Figure 10, of the three regions, the first region 43 and the third region 45 are determined to be regions that overlap with the movements of user 2. On the other hand, the second region 44 is determined to be a region that does not overlap with the movements of user 2.

[0151] Therefore, if it is determined based on the direction information 40 that the external person 3 is moving in the direction of the first region 43 or the third region 45 (i.e., maintaining the current direction or moving to the right relative to the current direction of movement), it is determined that there is a high probability of interference between the user 2 and the external person 3.

[0152] Therefore, when it is determined that the external person 3 will move towards the first area 43 or the third area 45, the HMD 1 outputs the movement-permitted information "STOP". Here, "STOP" means "Stop" or "Don't move", but this is just one example of output. In order to avoid interference with user 2, the movement-permitted information will output characters, diagrams, or symbols that inform the external person 3 that they should not move at this time.

[0153] On the other hand, if the external person 3 is moving towards the second area 44, that is, if user 2 is moving to the left of its current direction, the possibility of interference between user 2 and external person 3 is low. Therefore, if it is determined that external person 3 is moving towards the second area 44, the HMD1 outputs the movement-enabled information "GO". "GO" means "move forward" or "go," but this is just one example of output. The movement-enabled information will output characters, diagrams, or symbols that indicate that now is a suitable time for external person 3 to move while avoiding interference with user 2.

[0154] <Example of output of movable information in the second embodiment> Figure 11 shows an example of output of movable information in the second embodiment. For example, in the second embodiment, movable information is output on the display of the information processing terminal 4, similar to the first embodiment. The setting of the area will be explained below using an example of setting the three areas shown in Figure 10.

[0155] For example, on the display of the information processing terminal 4, movable information is output for each area based on the respective judgment result.

[0156] In the case of the area settings shown in Figure 10, the external person 3 can find out the "GO" or "STOP" judgment result for each area using the information processing terminal 4, as shown in Figure 11.

[0157] In Figure 11, movable information for the second area 44 is output to "LEFT" (hereinafter referred to as "second area display 62"). That is, movable information for when external person 3 moves to the left from its current position is displayed in the second area display 62.

[0158] Mobility information for the first area 43 is output to "Current Direction" (hereinafter referred to as "first area display 61"). In other words, the mobility information for when external person 3 moves from its current position while maintaining its current direction of movement is displayed in the first area display 61.

[0159] Mobility information for the third area 45 is output to "RIGHT" (hereinafter referred to as "third area display 63"). In other words, mobility information for when external person 3 moves to the right from their current position is displayed in the third area display 63.

[0160] In the example shown in Figure 11, the second area display 62 shows "GO". On the other hand, the first area display 61 and the third area display 63 show "STOP". Therefore, by looking at the second area display 62, the first area display 61, and the third area display 63, we can see that if external person 3 moves to the left, the possibility of interference with user 2 is low.

[0161] In addition, in the second embodiment, as in the first embodiment, the movable information may include messages, etc. Specifically, in the second embodiment as well, as shown in Figure 8, the movable information may include graphic symbols, messages, images, etc.

[0162] Furthermore, in the example shown in Figure 11, even if the external person 3 is unlikely to interfere with user 2 if they move from the left side (the second area 44), if the destination is on the right side (the third area 45, or further beyond), moving from the left side may result in a longer route.

[0163] For example, in the case described above, if the time for "STOP" is displayed in the third area display 63, similar to Figure 8, the external person 3 will stop for the time being and can resume movement only after it becomes possible to move. In this way, if messages or other information are included in the information on when movement is possible, inconveniences such as taking detours can be reduced.

[0164] <Example of a movable information determination program in the second embodiment> Figure 12 shows an example of processing by the movable information determination program 26 in the second embodiment. Hereinafter, the same processing as in the first embodiment will be denoted by the same reference numerals, and redundant explanations will be omitted. Compared with Figure 9, Figure 12 differs in that steps S30 to S32 are added.

[0165] In step S19, the HMD1 sets up regions. However, in the second embodiment, for example, multiple regions are set up as shown in Figure 10.

[0166] In step S30, the HMD1 makes a determination regarding the first region 43.

[0167] In step S31, the HMD1 makes a determination regarding the second region 44.

[0168] In step S32, the HMD1 makes a determination regarding the third region 45.

[0169] For example, step S30 determines whether the extension of the direction in which the external person 3 indicated by the direction information 40 is moving overlaps with a certain range 42 of user 2's movement.

[0170] Step S31 determines whether the area to the left of the extension of the direction in which the external person 3 indicated by the direction information 40 is moving overlaps with the range 42 in which user 2 is moving.

[0171] Step S32 determines whether the area to the right of the extension of the direction in which the external person 3 indicated by the direction information 40 is moving overlaps with the range 42 of the user 2's movement.

[0172] Based on the determinations in steps S30 to S32, in step S20, any area determined to overlap with the range 42 where user 2's movement is located is judged as "STOP". On the other hand, based on the determinations in steps S30 to S32, in step S20, any area determined not to overlap with the range 42 where user 2's movement is located is judged as "GO".

[0173] Steps S30 to S32 are not limited to the order described above. For example, steps S30 to S32 may be performed in an order other than the one described above, or in parallel.

[0174] In step S32, the HMD1 outputs movable information based on the determination results from steps S30 to S32, for example, as shown in Figure 11.

[0175] Steps S15 to S21 and steps S30 to S32 are repeatedly executed until an external person passes through (YES in step S22). The latest information is output for the determination results of steps S30 to S32.

[0176] Note that while the above example uses three domains, the number and configuration of domains are not limited to this. For example, two or four or more domains may be set. Also, each domain is pre-configured, but can be changed.

[0177] The movable information may be changed depending on the number of areas, etc. Also, if the movable information cannot be displayed on a single screen (for example, if there are many areas), the screen may be scrolled and output on multiple screens, etc.

[0178] Furthermore, the movable information does not have to be output in the same way for each region. For example, the region corresponding to the case where external person 3 maintains its current direction of movement from its current position (the first region 43 in Figure 10) may be displayed larger or emphasized compared to the other regions.

[0179] Additionally, movable information may be output in a format such as displaying the movable area (the area determined to be "GO") larger than other areas, or highlighting it.

[0180] As explained above, the HMD1 of the second embodiment, similar to the first embodiment, can output movability information to the information processing terminal 4 held by the external person 3, even without a request from the external person 3. This reduces the possibility of the external person 3 interfering with user 2 without disrupting user 2's VR content experience, similar to the first embodiment. Therefore, the external person 3 can move without interfering with user 2.

[0181] Furthermore, with the HMD1 of the second embodiment, for example, even if the external person 3 changes their direction of movement to the left or right, not just their current direction of movement, the possibility of interference with the user 2 can be known through the movement information. Therefore, the possibility of interference between the external person 3 and the user 2 by changing their direction of movement can be reduced.

[0182] <Third Embodiment> Figure 13 shows an example of the third embodiment. Compared to the first embodiment, the third embodiment differs in that there are multiple users. Also, in the third embodiment, each user wears a separate HMD, so there are multiple HMDs. For example, this could be a game tournament in a VR experience facility where multiple users experience the same VR content.

[0183] Hereafter, components similar to those in the first embodiment will be denoted by the same reference numerals, and redundant explanations will be omitted. In the following example, we will assume that there are two users. And, as in the first embodiment, we will assume that there is one external person 3 for the first user 2a and the second user 2b.

[0184] In the following example, the first user 2a and the second user 2b each wear one HMD. Hereafter, the HMD worn by the first user 2a will be referred to as "first HMD 1a," and the HMD worn by the second user 2b will be referred to as "second HMD 1b." Furthermore, external person 3 will wear an HMD as an information processing terminal. Hereafter, the HMD worn by external person 3 will be referred to as "third HMD 8." Note that external person 3 may also have an information processing terminal other than third HMD 8.

[0185] A smart speaker 32, an information display 35, and a rest area 36 are installed in the vicinity of the first user 2a and the second user 2b, i.e., in the real space.

[0186] The following explanation will use an example where an external person 3 attempts to move to the rest area 36. However, the third embodiment may be implemented in other situations.

[0187] Movable information is transmitted from, for example, the first HMD 1a or the second HMD 1b, and output to an HMD or information processing device other than the source. However, the source and destination information processing devices may be other than those listed above.

[0188] Figure 14 is a diagram showing an example of the third embodiment in a two-dimensional plane. For example, if there are two users, such as a first user 2a and a second user 2b, the first user 2a and the second user 2b share the real space to experience the VR content. Specifically, a first play area 33a is set for the first user 2a. Similarly, a second play area 33b is set for the second user 2b.

[0189] As shown in the diagram, it is desirable to have a blank area at the boundary between the first play area 33a and the second play area 33b that is not set in either play area. Having such a blank area reduces the possibility of a user interfering with an object or another user if either user moves beyond the play area for any reason.

[0190] Figure 15 shows an example of motion-based processing in the third embodiment. For example, in the third embodiment, similar to the second embodiment, the region to be judged is divided into three regions: the first region 43, the second region 44, and the third region 45.

[0191] The first range 42a is the range in which the first user 2a is likely to move. Similarly, the second range 42b is the range in which the second user 2b is likely to move. For example, the first range 42a and the second range 42b are set by analyzing their respective movements.

[0192] For example, if the movement of the first user 2a is as indicated by the first trajectory information 41a, a range in which the first user 2a is likely to move is set, as shown in the first range 42a. Similarly, if the movement of the second user 2b is as indicated by the second trajectory information 41b, a range in which the second user 2b is likely to move is set, as shown in the second range 42b.

[0193] In the example shown in Figure 15, the third region 45 is a region that includes a part of the first play area 33a. Similarly, the second region 44 is a region that includes a part of the second play area 33b. Furthermore, the first region 43 includes both a part of the first play area 33a and a part of the second play area 33b, and its main area is the area between the first play area 33a and the second play area 33b (including the blank area between the first play area 33a and the second play area 33b).

[0194] In the example shown in Figure 15, the second region 44 and the second range 42b mainly overlap. Similarly, the third region 45 and the first range 42a mainly overlap.

[0195] On the other hand, the third region 45 does not overlap with the movements of any user. Note that the third region 45 overlaps with a portion of the second region 42b, but based on the second trajectory information 41b, it is determined that the possibility of interference is low. Such tolerances may be adjusted by pre-set thresholds or the like.

[0196] In the above-described state, if it is determined that the external person 3 is moving towards the second area 44 or the third area 45, the HMD outputs the movement-possible information "STOP".

[0197] On the other hand, if the external person 3 is moving towards the first region 43, that is, if the user 2 is moving in the direction that maintains its current direction, the possibility of interference between the user 2 and the external person 3 is low. Therefore, if it is determined that the external person 3 is moving towards the first region 43, the HMD outputs the movement-possible information "GO".

[0198] <Example of output of movable information in the third embodiment> Figure 16 shows an example of output of movable information in the third embodiment. In the third embodiment, it is desirable that the movable information be output by integrating the movable information for each region, as shown in Figure 16(a).

[0199] In the third embodiment, determination is first made for each region and each user. For example, in the example shown in Figure 15, firstly, determination is made based on the first user 2a, and determination is made by the first HMD 1a as shown in "Determination of the first user" in Figure 16(a).

[0200] In Figure 16(a), the "determination of the first user" is the determination of the external person 3 to "GO" or "STOP" for the first user 2a. For example, the determination of "GO" or "STOP" is performed in the same manner as in the first embodiment.

[0201] In the example shown in Figure 15, the "first user's judgment" excludes the "second area" from judgment because the second area 44 is outside the first play area 33a. Therefore, the "first user's judgment" does not perform a judgment on the "second area" and sets it to "NO JUDGE," which is neither "GO" nor "STOP." However, a judgment may also be performed on areas outside the first play area 33a.

[0202] On the other hand, the "determination of the first user" is "GO" for the "first region" because the first region 43 does not overlap with the first range 42a.

[0203] Furthermore, the "determination of the first user" for the "third region" is "STOP" because the third region 45 overlaps with the first range 42a.

[0204] Secondly, the determination is made based on the second user 2b, and the determination is made by the second HMD 1b as shown in "Determination of the second user" in Figure 16(a). For example, "Determination of the second user" is determined in the same way as "Determination of the first user".

[0205] In the example shown in Figure 15, the "determination of the second user" is "STOP" for the "second region" because the second region 44 overlaps with the second range 42b.

[0206] On the other hand, the "determination of the second user" is "GO" for the "first region" because the first region 43 does not overlap with the second region 42b.

[0207] Furthermore, in the "first user's judgment," the "third area" is excluded from the judgment because the third area 45 is outside the second play area 33b. Therefore, the "second user's judgment" does not perform a judgment on the "third area," and the result is "NO JUDGE," which is neither "GO" nor "STOP." However, a judgment may also be performed on areas outside the second play area 33b.

[0208] For example, once the determination for each area and each user is completed as described above, the determination results for each user and each area are integrated to generate movable information, such as the "integrated determination result" in Figure 16(a).

[0209] Integration is a process that involves collecting the judgment result information generated by each HMD, for example, into one of the HMDs. The following example assumes that the first HMD 1a performs the integration. However, the integration may also be performed by an information processing device other than the first HMD 1a (for example, the second HMD 1b, or a server for executing integration and other processing). Which information processing device performs the integration is determined, for example, in advance.

[0210] Integration is a process in which, if at least one of the judgment results for each of the multiple domains is "STOP", the "integration judgment result" is set to "STOP". In other words, if all of the judgment results for each of the multiple domains are "GO", the "integration judgment result" will be "GO".

[0211] Specifically, in the example shown in Figure 16(a), the "second domain" is determined to be "STOP" by the "second user determination". Therefore, the "integrated determination result" becomes "STOP" after integration.

[0212] In the example shown in Figure 16(a), the "first area" has determined that both the "first user determination" and the "second user determination" are "GO," and there are no "STOP" results. Therefore, the "integrated determination result" becomes "GO" through integration.

[0213] In the example shown in Figure 16(a), the "third domain" is determined to be "STOP" by the "first user's judgment". Therefore, the "integrated judgment result" becomes "STOP" after integration.

[0214] If even one of the judgments based on different users is "STOP," then external person 3 is highly likely to interfere with one of the users. Therefore, if even one of the multiple judgment results is "STOP," it is desirable to merge them and set the judgment result to "STOP."

[0215] "NO JUDGE" is treated similarly to, for example, "STOP". However, whether or not "NO JUDGE" is treated the same as "STOP" may be set in advance. For example, if external person 3 has an urgent matter and wants to move immediately, it may be treated the same as "GO", but in that case, it is desirable to inform external person 3 that this is different from a normal "GO" judgment, that there is a higher possibility of interference with another user compared to a normal "GO" judgment, or that they should move with even greater consideration for safety.

[0216] Figure 16(b) shows an example of outputting movable information based on the "integration judgment result" shown in Figure 16(a). For example, the movable information shown in Figure 16(b) is output to the external person 3 by the third HMD8 after integration.

[0217] In the third embodiment, the movable information is output in the same format as in the second embodiment, for example.

[0218] As shown in Figure 16(b), the movable information is displayed for each area, i.e., in the first area display 61, the second area display 62, and the third area display 63. In the example shown in Figure 16(b), only the integrated judgment result is displayed as movable information, but the movable information may also include the judgment results for each user.

[0219] In addition, in the third embodiment, as in the first embodiment, the movable information may include messages, etc.

[0220] <Example of a Movable Information Determination Program in the Third Embodiment> Figure 17 shows an example of processing by the movable information determination program 26 in the third embodiment. Hereinafter, the same processes as in the first and second embodiments are denoted by the same reference numerals, and redundant explanations are omitted. Compared with Figure 12, Figure 17 differs in that steps S40 to S50 are added. Also, the difference between the second and third embodiments is due to the use of multiple HMDs.

[0221] The following explanation of the overall processing will be based on an example where the first HMD1a is the primary processor. However, each process may be primarily performed by information processing devices other than the first HMD1a, or it may be performed in a distributed manner.

[0222] In step S40, each HMD detects other HMDs. If other HMDs are detected (YES in step S40), the first HMD 1a proceeds to step S41. On the other hand, if no other HMDs are detected (NO in step S40), the first HMD 1a proceeds to step S14.

[0223] In step S41, the first HMD 1a adjusts the play area. For example, as shown in Figure 14, step S41 is the process of setting the first play area 33a and the second play area 33b. Note that when setting each play area, adjustments to the play areas may occur between multiple HMDs.

[0224] In step S42, each HMD transmits and receives location information and other data with other HMDs. Note that each HMD may also transmit and receive location information and other data with other HMDs at steps other than S42. Furthermore, each HMD may also transmit and receive information other than location information with other HMDs.

[0225] In step S43, the first HMD 1a determines whether or not it is responsible for outputting the movable information. For example, whether or not it is responsible for outputting the movable information is set in advance. Next, if it is responsible for outputting the movable information (YES in step S43), the first HMD 1a proceeds to step S44. On the other hand, if it is not responsible for outputting the movable information (NO in step S43), the first HMD 1a proceeds to step S23. The following explanation will use an example where the first HMD 1a is responsible for outputting the movable information (YES in step S43).

[0226] In step S44, the first HMD 1a notifies the other HMDs that it is responsible for outputting movable information.

[0227] In step S45, the first HMD 1a shares information about the external person 3 with other HMDs.

[0228] In step S46, the first HMD 1a shares region information with other HMDs.

[0229] In step S47, the first HMD 1a determines whether or not it is responsible for outputting the movable information. If it is responsible for outputting the movable information (YES in step S47), the first HMD 1a proceeds to step S48. On the other hand, if it is not responsible for outputting the movable information (NO in step S47), the HMDs other than the first HMD 1a proceed to step S49.

[0230] In step S48, the first HMD 1a receives information such as the judgment result from the other HMD.

[0231] In step S49, HMDs other than the first HMD 1a transmit information such as the judgment result to the first HMD 1a.

[0232] Step S20 is performed, for example, in the same manner as in the second embodiment. The determination results performed separately on each HMD (the "determination for the first user" and the "determination for the second user" in Figure 16(a)) are collected in an information processing device that outputs movable information, i.e., performs integration.

[0233] In this example, the output of movable information is handled by the first HMD 1a, so HMDs other than the first HMD 1a transmit information such as the determination result to the first HMD 1a (step S49). Meanwhile, the first HMD 1a receives the information transmitted in step S49 (step S50).

[0234] In step S50, the first HMD 1a integrates the respective judgment results. For example, once the integration is complete, an integrated result like the "integrated judgment result" shown in Figure 16(a) is generated.

[0235] Once the integration result is generated, in step S50, the first HMD 1a outputs it as movable information. For example, as shown in Figure 16(b), the first HMD 1a transmits the movable information to the third HMD 8, causing the third HMD 8 to output it on a display or the like. Note that the output destination may be other than the third HMD 8.

[0236] In the example above, three regions were set, but the number and configuration of regions are not limited to this. For example, two or four or more regions may be set, similar to the second embodiment. Also, each region is set in advance, but may be changed.

[0237] Furthermore, if there are multiple external individuals 3, such as in a VR experience facility, the destination for transmitting the movement information may be a guidance display 35 that can be viewed by many people simultaneously. By using a guidance display 35 as the destination, movement information can be provided to many people with a small amount of data.

[0238] The information displayed on the guidance display 35 should preferably be common to all individuals. For example, as shown in Figure 16(b), a display based on the current movement direction of a specific individual, such as external person 3, is often difficult for other individuals to understand. Therefore, the guidance display 35 should display information that takes into account the play status of multiple users, so that multiple external individuals can understand whether or not it is a suitable time to move. Specifically, for example, "GO" may be displayed when the activity level of all users is low, and "STOP" may be displayed when the activity level is high. In addition to this information, the duration of the "GO" or "STOP" state may also be displayed. Furthermore, graphic symbols such as "GO" or "STOP" may be displayed as a result of a judgment regarding the play area (the judgment is made without dividing it into multiple areas).

[0239] On the other hand, each person's information processing device may display information that supplements the information displayed on the guidance display 35, or information that differs for each person (such as information about the directions in which each person can move from their current location). Alternatively, information may be notified by voice or other means, or by each person's information processing device. If the facility is equipped with a speaker or other voice output means, notifying via the speaker or other means allows common information to be delivered to multiple people in the facility simultaneously.

[0240] In this way, the guidance display 35 receives graphical information with a large amount of data, while the information processing devices of each individual receive information with a smaller amount of data. This reduces the overall amount of data transmitted and allows multiple individuals to receive information such as whether or not they are interfering with the user.

[0241] As described above, according to the HMD1 of the third embodiment, as with the first and second embodiments, movable information can be output to the information processing terminal 4 held by the external person 3 even without a request from the external person 3. This reduces the possibility of the external person 3 interfering with user 2 without interfering with user 2's VR content experience, as with the first embodiment. Therefore, the external person 3 can move around without interfering with multiple users.

[0242] Furthermore, with the HMD1 of the third embodiment, for example, even if there are multiple users, the external person 3 can know the possibility of interference with the users through the movement information. Therefore, the external person 3 can understand whether there is a possibility of interference with any of the users and select a route with a low probability of interference to reduce the possibility of interference with the users.

[0243] <Example of Functional Configuration> Figure 18 shows an example of a functional configuration. For example, HMD1, which is an example of a display device, includes a sensing unit 1F1, a storage unit 1F2, and a calculation unit 1F3, etc.

[0244] The sensing unit 1F1 performs a sensing procedure to acquire direction information 40 of an external person 3. For example, the sensing unit 1F1 can be implemented using a camera 10, a distance measuring sensor 11, a sensor group 15, or a combination thereof.

[0245] The memory unit 1F2 performs a memory procedure to record the user's movements. For example, the memory unit 1F2 is implemented using storage 23 or the like.

[0246] The calculation unit 1F3 performs setting procedures, determination procedures, output procedures, etc. For example, the calculation unit 1F3 is implemented by an arithmetic device 21 or the like.

[0247] The setup procedure involves setting the user's activity level, the range of possible user movement, or the period of user movement based on the user's movements. For example, the user's activity level and the period of user movement are set by analyzing the user's past movements, as shown in Figure 5. Similarly, the range of possible user movement is set by analyzing the user's past movements, as shown in range 42 in Figure 6.

[0248] The determination procedure generates a determination result based on the setting results based on the movement as shown in Figure 6, etc., and the direction information 40, determining whether the movements of the external person 3 and the user overlap.

[0249] The output procedure involves outputting movable information in the direction indicated by the directional information. For example, the output of movable information is performed as shown in Figure 7.

[0250] HMD1 senses whether or not there is an external person 3 around the user. If external person 3 is present, HMD1 analyzes how the user will move based on the content status or past movement history, and sets the range or period in which the user may move. At the same time, HMD1 identifies the direction in which external person 3 is moving using direction information 40. With these settings and direction information 40, HMD1 can determine, for example, whether or not there is a high possibility of overlap and interference between the movements of external person 3 and the user, as shown in Figure 6. Based on the determination result, HMD1 outputs movement-possible information to external person 3.

[0251] The movable information output in this way allows external person 3 to change direction to avoid interference with the user, or to move in a way that avoids periods when interference with the user is likely. Therefore, it is possible to avoid interference between the external person and the user without interfering with the user's VR content experience.

[0252] <Other Embodiments> The processes described above may be implemented by multiple devices. For example, the processes described above may be executed in a distributed, redundant, or parallel manner by a display system having multiple information processing devices.

[0253] Furthermore, the information processing equipment and other components that make up the display system may be located overseas. In other words, some of the information processing equipment that performs some of the processing performed by the display system may be located overseas.

[0254] Furthermore, the detection of people or objects may be implemented using, for example, AI (Artificial Intelligence). Specifically, to implement AI, a training model is first trained using training data that includes image data representing people or other objects to be detected, and correct answer data indicating whether or not an object is a target. After training, when unknown image data is input to the trained model, it performs image recognition based on the correlation between the trained image data and the correct answer. Using such AI, sensors and other devices can detect targets with high accuracy.

[0255] Detection may be implemented using a network of sensors that constitute the Internet of Things (IoT), etc.

[0256] The embodiments of the present invention described above are not limited to those described above, and some components may be replaced with those of other embodiments. Furthermore, the configuration of one embodiment may be combined with the configuration of another embodiment. All of these fall within the scope of the present invention. Moreover, the numbers or messages appearing in the text or figures are examples only, and using different ones will not impair the effects of the present invention.

[0257] Furthermore, some or all of the functions of the invention may be implemented in hardware, for example, by designing them using an integrated circuit. Alternatively, they may be implemented in software by a microprocessor unit or CPU interpreting and executing an operating program. Moreover, the scope of software implementation is not limited, and hardware and software may be used in combination.

[0258] The above display control method may be implemented by a program (including firmware). Furthermore, the program may be provided by recording it on a computer-readable, non-transient recording medium. Recording medium refers to, for example, auxiliary storage devices such as hard disks, SD cards (registered trademark), optical discs such as DVDs, or servers on the Internet. Therefore, the program may be distributed via telecommunication lines such as the Internet.

[0259] Furthermore, the program according to the present invention is not limited to being a single, identical program, but may be a collection of multiple programs. Also, the program according to the present invention is not limited to being executed on a single device, but may be executed by multiple information processing devices.

[0260] The responsibilities of the information processing device are not limited to the examples given above. In other words, some or all of the above processing may be performed by an information processing device other than the one described above.

[0261] The above embodiments include the following inventions.

[0262] (Note 1) The display device comprises a sensor that acquires direction information relating to the direction of movement of a person other than the user of the display device; a storage device that stores activity information relating to the user's movements; and a computing device that processes information based on the direction information and the activity information, wherein the computing device determines whether the user and the person interfere with each other based on the direction information and the activity information, and outputs movable information relating to whether the person can move.

[0263] (Note 2) The display device described in (Note 1), which outputs the movable information to an information terminal that outputs the movable information to the person.

[0264] (Note 3) The display device described in (Note 1) is a graphic symbol indicating that the person is movable, or a graphic symbol indicating that the external person is immovable.

[0265] (Note 4) The display device according to (Note 3), wherein the movable information further includes the time during which the person is movable or immovable.

[0266] (Note 5) The display device described in (Note 1), wherein the calculation device sets a plurality of regions, generates the judgment result for each of the regions, and outputs the movable information based on each of the judgment results.

[0267] (Note 6) The movable information is a display device as described in (Note 5), which displays, for each of the aforementioned areas, a graphic symbol indicating that the person is movable or a graphic symbol indicating that the person is immovable.

[0268] (Note 7) The display device described in (Note 6), wherein the movable information is displayed by setting multiple regions based on the person, and for each region, a graphic symbol indicating that the person is movable or a graphic symbol indicating that the person is immovable is displayed side by side.

[0269] (Note 8) The arithmetic unit is the display device described in (Note 5), wherein the area is shared by multiple display devices.

[0270] (Note 9) When multiple users and multiple areas are set, the display device according to (Note 5) integrates the determination results for each user and each area and outputs the movable information.

[0271] (Note 10) The display device described in (Note 1), wherein the setting result is set based on the range in which the user has moved in the past, the range in which the user is estimated to move, or a range set in advance for the user's position.

[0272] (Note 11) The display device described in (Note 1) is immersive.

[0273] (Note 12) The display device described in (Note 1), wherein the sensor photographs the area around the user, recognizes the person and obstacles, and uses the direction in which the person moves as direction information based on the change in the person's position over time.

[0274] (Note 13) The display device according to (Note 1), wherein the calculation device sets the amount of activity of the user, the range in which the user may move, or the period of time during which the user moves, based on the activity information.

[0275] (Note 14) The display device described in (Note 13), wherein the calculation device determines whether the user and the person interfere with each other based on the activity information and the direction information.

[0276] (Note 15) A display control method performed by a display device, wherein the display device performs the steps of: acquiring direction information indicating the direction in which a person other than the user of the display device is moving; storing the movements of the user; setting the amount of activity of the user, the range in which the user may move, or the period of time during which the user moves, based on the movements; generating a determination result that determines whether the person and the movements overlap, based on the setting result based on the movements and the direction information; and controlling the display of movable information indicating whether the person is able to move in the direction indicated by the direction information, based on the determination result.

[0277] (Note 16) A display system having a display device, comprising: a sensor that acquires direction information relating to the direction of movement of a person other than the user of the display device; a storage device that stores activity information relating to the user's movement; and a computing device that processes information based on the direction information and the activity information, wherein the computing device determines whether the user and the person interfere with each other based on the direction information and the activity information, and outputs movable information relating to whether the person can move.

[0278] 1: HMD, 1F1: Sensing unit, 1F2: Memory unit, 1F3: Processing unit, 1a: First HMD, 1b: Second HMD, 2: User, 2a: First user, 2b: Second user, 3: External person, 4: Information processing terminal, 5: Earphone, 6a: Access point, 6b: Network, 6c: VR service server, 8: Third HMD, 10: Camera, 11: Distance sensor, 12: Display, 13: Speaker, 14: Microphone, 15: Sensor group, 16: Communication device, 17: Bus, 18a: Wearing housing, 20: Communication device, 21: Processing unit, 22: Memory, 23: Storage, 24: Basic operation program, 25: VR experience program, 26: Movable information determination program, 30: Bookshelf, 3 1: TV, 32: Smart speaker, 33: Play area, 33a: First play area, 33b: Second play area, 34: Living room, 35: Information display, 36: Rest area, 40: Directional information, 40a: Threshold, 41: Trajectory information, 41a: First trajectory information, 41b: Second trajectory information, 42: Range, 42a: First range, 42b: Second range, 43: First region, 44: Second region, 45: Third region, 50: First graphic symbol, 51: Second graphic symbol, 53: First time display, 54: Second time display, 55: First message, 56: Second message, 57: Progress bar, 61: First region display, 62: Second region display, 63: Third region display, T0: Current time, T1: First point in time, T2: Second point in time

Claims

1. A display device comprising: a sensor that acquires direction information relating to the direction of movement of a person other than the user of the display device; a storage device that stores activity information relating to the user's movement; and a computing device that performs processing based on the direction information and information relating to the activity information, wherein the computing device determines whether the user and the person interfere with each other based on the direction information and the activity information, and outputs movable information relating to whether the person can move.

2. The display device according to claim 1, which outputs the movable information to an information terminal that outputs the movable information to the person.

3. The display device according to claim 1, wherein the movable information is a graphic symbol indicating that the person is movable, or a graphic symbol indicating that the person is immovable.

4. The display device according to claim 3, wherein the movable information further includes a movement time during which the person is movable or immovable.

5. The display device according to claim 1, wherein the arithmetic unit sets a plurality of regions, generates a determination result based on the determination for each of the regions, and outputs the movable information based on each of the determination results.

6. The display device according to claim 5, wherein the movable information includes, for each of the aforementioned regions, a graphic symbol indicating that the person is movable or a graphic symbol indicating that the person is immovable.

7. The display device according to claim 6, wherein the movable information is provided by setting multiple regions based on the person, and displaying, in each region, a graphic symbol indicating that the person is movable or a graphic symbol indicating that the person is immovable.

8. The display device according to claim 5, wherein the calculation device is shared by a plurality of display devices.

9. When multiple users and multiple regions are set, the display device according to claim 5, wherein the arithmetic unit integrates the determination results for each user and each region and outputs the movable information.

10. The display device according to claim 1, wherein the calculation device is set based on the range in which the user has moved in the past, the range in which the user is estimated to move, or a range set in advance with respect to the user's position.

11. The display device according to claim 1, which is immersive.

12. The display device according to claim 1, wherein the sensor photographs the area around the user, recognizes the person and obstacles, and uses the direction in which the person moves as direction information based on the change in the person's position over time.

13. The display device according to claim 1, wherein the computing device sets the amount of activity of the user, the range in which the user may move, or the period of time during which the user moves, based on the activity information.

14. The display device according to claim 13, wherein the computing device determines whether the user and the person interfere with each other based on the activity information and the direction information.

15. A display control method performed by a display device, the display device performing the following steps: acquiring direction information indicating the direction in which a person other than the user of the display device is moving; storing the user's movements; setting the user's activity level, the range in which the user may move, or the period of time the user moves based on the movements; generating a determination result that determines whether the person and the movements overlap based on the setting result based on the movements and the direction information; and controlling the display of movable information indicating whether the person can move in the direction indicated by the direction information based on the determination result.

16. A display system having a display device, comprising: a sensor that acquires direction information relating to the direction of movement of a person other than the user of the display device; a storage device that stores activity information relating to the user's movement; and a computing device that performs processing based on the direction information and information relating to the activity information, wherein the computing device determines whether the user and the person interfere with each other based on the direction information and the activity information, and outputs movable information relating to whether the person can move.

Images