Information processing device, information processing method, and program

Abstract

While this can help avoid collisions with other users, issuing warnings may reduce the user's immersion in mixed reality. Furthermore, it is unavoidable that the user's immersion in mixed reality will be reduced in situations where other users obstruct virtual content outside the user's attention zone. [Solution] An information processing apparatus comprising: a first position and orientation acquisition means for acquiring the position and orientation of a first user; a second position and orientation acquisition means for acquiring the position and orientation of a second user; a relative position and orientation calculation means for calculating the relative position and orientation of the first user and the second user based on the first position and orientation and the second position and orientation; a play area calculation means for calculating a play area in which the second user experiences virtual content; and a content generation means for generating content that allows the first user to identify the play area based on the relative position and orientation and the play area.

Description

[Technical Field]

[0001] This invention relates to the generation of virtual content when multiple users simultaneously experience mixed reality (MR). [Background technology]

[0002] Mixed Reality (MR) technology, which overlays virtual content (CG and text) onto video footage captured by a camera, is well-known. This technology utilizes the estimation of the device's position and orientation (such as an HMD or smartphone) based on video information to calculate the rendering position of the virtual content in accordance with the device's movement.

[0003] When multiple users experience MR simultaneously, virtual content may obstruct other users, potentially leading to collisions due to the approach of other users. Furthermore, even if users are not at a collision-prone distance, other users passing in front of the displayed virtual content can obstruct it, potentially diminishing the immersive experience of MR. Patent Document 1 addresses the former problem by proposing a method to avoid collisions by issuing warnings based on the distance between users when they enter the attention zone. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Patent No. 04642538 [Overview of the project] [Problems that the invention aims to solve]

[0005] While the method described in Patent Document 1 can avoid collisions with other users, it may reduce the user's immersion in MR by issuing warnings. Furthermore, since the attention zone is not set considering the type and location of virtual content, it is unavoidable that the user's immersion in MR will be reduced in situations where other users obstruct the virtual content outside the attention zone.

[0006] The present invention aims to minimize the reduction in immersion in MR caused by the proximity of other users, while avoiding collisions with other users in situations where multiple users experience MR simultaneously. [Means for solving the problem]

[0007] To achieve the above objectives, the present invention The system is characterized by comprising: a first position and orientation acquisition means for acquiring the position and orientation of a first user; a second position and orientation acquisition means for acquiring the position and orientation of a second user; a relative position and orientation calculation means for calculating the relative position and orientation of the first user and the second user based on the first and second position and orientations; a play area calculation means for calculating a play area in which the second user experiences virtual content; and a content generation means for generating content that allows the first user to identify the play area based on the relative position and orientation and the play area. [Effects of the Invention]

[0008] In situations where multiple users experience MR simultaneously, it is possible to avoid collisions with other users while minimizing the reduction in immersion caused by other users' proximity. [Brief explanation of the drawing]

[0009] [Figure 1] A hardware configuration diagram showing an example of an information processing device. [Figure 2] An illustrative diagram showing the application scenario in Example 1. [Figure 3] An illustrative diagram showing the area of ​​the play area in Example 1. [Figure 4] A block diagram showing the logical configuration of the information processing device in Example 1. [Figure 5] A flowchart illustrating the processing of the information processing device in Example 1. [Figure 6] A flowchart showing the processing of the play area calculation unit in Example 1. [Figure 7] Flowchart showing the processing of the information processing apparatus in Modifications 1-5. [Figure 8] Flowchart showing the processing of the warning intensity calculation unit in Modifications 1-5. [Figure 9] Flowchart showing the processing of the play area calculation unit in Example 2.

Mode for Carrying Out the Invention

[0010] Prior to describing each embodiment according to the present invention, the hardware configuration capable of realizing the information processing apparatus shown in each embodiment will be described with reference to FIG. 1.

[0011] FIG. 1 is a hardware configuration diagram of an information processing apparatus according to an embodiment. The CPU 10 controls each unit connected to the bus 60 via the bus 60. The input I / F 40 acquires an input signal in a format that can be processed by the information processing apparatus from an external device (such as an imaging device, a display device, or an operating device). Further, the output I / F 50 outputs an output signal in a format that can be processed by an external device (such as a display device) to the external device.

[0012] The program for realizing the functions of each embodiment is stored in a storage medium such as a read-only memory (ROM) 20. Further, the ROM 20 stores an operating system (OS) and device drivers. A memory such as a random access memory (RAM) 30 stores these programs temporarily. Then, by the CPU 10 executing the programs stored in the RAM 30, the processing according to each flowchart described later is executed to realize the functions of each embodiment. However, instead of software processing using the CPU 10, it is also possible to realize the functions of each embodiment using hardware having an arithmetic unit or a circuit corresponding to the processing of each functional unit.

[0013] [Example 1] In Example 1, it is assumed that the first user and the second user are wearing HMDs capable of experiencing MR equipped with the information processing apparatus of the present invention.

[0014] Figure 2 is an illustrative diagram showing a usage scenario for an HMD equipped with the information processing device of the present invention. The first user 1 is wearing HMD3, and the second user 2 is wearing HMD4. HMD3 and HMD4 display images in which virtual content is superimposed on images of the real world. The second user 2 sees virtual content 5 in the form of a virtual monitor through HMD4.

[0015] In this scenario, virtual content 5 is displayed on the second user's HMD4, but not on the first user's HMD3. As a result, the first user 1 passes between the second user 2 and virtual content 5 without realizing that the second user 2 is viewing it. This reduces the second user 2's immersion in the mixed reality experience.

[0016] Figure 3 shows how the present invention allows the first user 1 to view the play area 6 of the second user 2 through the HMD 3. The play area is the real-world range in which a user experiencing virtual content through the HMD can move around, such as an area where safe movement is guaranteed without colliding with surrounding objects, people, or walls. The play area also includes the area between the user and the virtual content when the user is experiencing the virtual content while remaining nearly stationary. For example, the area between the user and the virtual monitor when the user is viewing a virtual monitor. The play area 6 is represented by an area encompassing the virtual content 5 that the first user 1 is viewing and the first user 1. By viewing the play area 6 of the second user 2, the first user 1 can recognize that the second user 2 is experiencing MR, and this can be used to avoid collisions with the second user 2 or to avoid interfering with the second user 2's MR experience.

[0017] Figure 4 is a block diagram showing the device configuration including the information processing device 100 in this embodiment.

[0018] The information processing device 100 includes a first position and orientation acquisition unit 101, a second position and orientation acquisition unit 102, a user information acquisition unit 103, a relative position and orientation calculation unit 104, a play area calculation unit 105, and a content generation unit 106. The first position and orientation acquisition unit 101 is connected to the first position and orientation calculation unit 32, and the second position and orientation acquisition unit 102 is connected to the second position and orientation calculation unit 42. The user information acquisition unit 103 is connected to the transmission unit 43, and the content generation unit 106 is connected to the display unit 33.

[0019] The first position and attitude acquisition unit 101 acquires the position and attitude of the HMD3 calculated by the first position and attitude calculation unit 32.

[0020] The second position and attitude acquisition unit 102 acquires the position and attitude of the HMD4 calculated by the second position and attitude calculation unit 42.

[0021] The user information acquisition unit 103 acquires user information held by the user information holding unit 44 via the transmission unit 43.

[0022] The relative position and attitude calculation unit 104 calculates the relative position and attitude of HMD3 and HMD4 based on the two position and attitude acquisition units 101 and 102.

[0023] The play area calculation unit 105 calculates the play area for the second user 2 based on the position and orientation of the HMD4 acquired by the second position and orientation acquisition unit 102 and the user information acquired by the user information acquisition unit 103.

[0024] The content generation unit 106 generates content to be displayed on the HMD4 based on the relative position and orientation calculated by the relative position and orientation calculation unit 104 and the play area calculated by the play area calculation unit 105.

[0025] The first display device 3 (HMD3) consists of an input unit 31, a first position and orientation calculation unit 32, an information processing device 100, and a display unit 33.

[0026] The input unit 31 acquires images input by the camera mounted on the HMD.

[0027] The first position and attitude calculation unit 32 acquires the image output by the input unit 31, calculates the position and attitude of the HMD3 based on the input image, and outputs it to the first position and attitude acquisition unit 101.

[0028] The display unit 33 receives virtual content representing the play area of ​​the second user 2, output by the content generation unit 106, and displays an image in which the virtual content representing the play area is superimposed on the display image for the second user 2's MR experience.

[0029] The second display device (HMD4) consists of an input unit 41, a second position and orientation calculation unit 32, a transmission unit 43, and a user information holding unit 44. Although the second display device (HMD4) is assumed to have the same function of displaying images as the first display device, this is not directly related to the processing of the information processing device 100, so its explanation is omitted.

[0030] The input unit 41 acquires images input by the camera mounted on the HMD.

[0031] The second position and attitude calculation unit 42 acquires the image output by the input unit 41, calculates the position and attitude of the HMD4 based on the input image, and outputs it to the transmission unit 43.

[0032] The transmitting unit 43 outputs the position and orientation of the HMD4 output by the second position and orientation calculation unit 42, and the user information held by the user information holding unit 44, to the user information acquisition unit 103. Communication is performed using P2P (Peer-to-Peer) communication.

[0033] The user information storage unit 44 stores the user information of the second user 2. The contents of the user information will be described later.

[0034] In this embodiment, the user information includes attribute information of the virtual content. The attribute information of the virtual content includes information representing the relative position and orientation information of the HMD4 and the virtual content 5, and the three-dimensional model information of the virtual content 5. It may also include information representing the dynamic characteristics (static, dynamic) of the virtual content, indicating whether the virtual content 5 is moving or not. Furthermore, it may also include information representing the dynamic characteristics (static, dynamic) of the user experiencing the virtual content 5, indicating whether the user is moving or not.

[0035] The flowcharts illustrating the processing steps in the following explanation are not limited to examples; any combination of steps, grouping of multiple processes, or subdivision of processes is possible as long as the results of the present invention are satisfied. Furthermore, each process can be individually separated and function as a single functional element, and can be used in combination with processes other than those shown.

[0036] Figure 5 is a flowchart showing the processing procedure in this embodiment.

[0037] In step S1010, an initialization process is performed to make the information processing device 100 operational.

[0038] In step S1020, the position and orientation acquisition unit 101 of the first user 1 acquires the position and orientation of the HMD3 calculated by the first position and orientation calculation unit 32. The method used by the first position and orientation calculation unit 32 to calculate the position and orientation of the HMD3 is a method that estimates the position and orientation of the camera while creating a three-dimensional map of the real space using the method of Raul et al. (described later). The method of Raul et al. is Raul Mur-Artal et al., ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics.

[0039] In step S1030, the second user 2 position and attitude acquisition unit 102 acquires the position and attitude of the HMD4 calculated by the second position and attitude calculation unit 42. The second position and attitude calculation unit 42 uses the same method for calculating the position and attitude of the HMD4 as the first position and attitude calculation unit 32.

[0040] In step S1040, the user information acquisition unit 103 acquires user information held by the user information holding unit 44. In this embodiment, the user information includes attribute information of the MR virtual content being experienced by the second user 2 (including information indicating the relative position and orientation information of the HMD4 and the virtual content 5, and information indicating the three-dimensional model information of the virtual content 5). It also includes information indicating the dynamic characteristics (static, dynamic) of the virtual content, which shows whether the virtual content 5 is moving or not. Furthermore, it includes information indicating the dynamic characteristics (static, dynamic) of the user experiencing the virtual content, which shows whether the user is moving or not.

[0041] In step S1050, the relative position and attitude calculation unit 104 calculates the relative position and attitude of HMD3 and HMD4 based on the position and attitude of HMD3 acquired by the first position and attitude acquisition unit 101 and the position and attitude of HMD4 acquired by the second position and attitude acquisition unit 102. At this time, it is assumed that the position and attitude of the origin of the reference coordinate system for the position and attitude of HMD3 and HMD4 have been aligned in advance. In this embodiment, as in the method of Richard et al. (described later), a common area of ​​the three-dimensional map created by the first position and attitude calculation unit 32 and the three-dimensional map created by the second position and attitude calculation unit 42 is detected, and the position and attitude of the origin of the two coordinate systems are aligned based on the common area. The method of Richard et al. is Richard Elvira et al., ORBSLAM-Atlas: a robust and accurate multi-map system. As a result, the two position and attitude are calculated in the same coordinate system.

[0042] In step S1060, the play area calculation unit 105 calculates the play area for the second user 2 based on the user information obtained in step S1040. The play area for the second user 2 is calculated with the second user's position and orientation as the origin. Details of the process in step S1060 will be described later.

[0043] In step S1070, the content generation unit 106 generates virtual content that represents the area of ​​the play area of ​​the second user 2 to be displayed on the HMD3 of the first user 1. This process is performed based on the relative position and orientation calculated in step S1050 and the play area calculated in step S1060. Specifically, based on the vertices representing the play area generated in step S1070, a mesh composed of those vertices is calculated, and virtual content representing the area of ​​the play area is generated. In this embodiment, since the play area is represented by a bounding box, two triangular meshes are generated for each of the six faces of the rectangular prism, and a total of 12 triangular meshes constitute the faces of the virtual content representing the play area. The method of calculating the mesh is not limited to this; any method of representing faces over an area encompassing vertices is acceptable.

[0044] Next, based on the second position and orientation obtained in step S1030 and the relative position and orientation obtained in step S1050, the generated virtual content is converted to a position and orientation with the second user's position and orientation as the origin.

[0045] In step S1080, a decision is made as to whether or not to terminate the system. If a termination instruction is entered via the input means (not shown), the system is terminated. Otherwise, the process returns to step S1020, and steps S1020 through S1070 are repeated.

[0046] The details of the process in step S1060 are described below.

[0047] Figure 6 is a flowchart showing the process flow for calculating the play area. Here, we will explain an example of calculating the play area for a second user 2 who will be viewing virtual content 5.

[0048] In step S1061, the play area calculation unit 105 acquires the position and orientation of the second user 2 obtained in step S1030 and stores its history. The history is a record of the user's position and orientation since HMD3 and HMD4 started communicating.

[0049] In step S1062, the play area calculation unit 105 acquires attribute information of the virtual content included in the user information acquired in step S1040. For example, the attribute information of the virtual content includes relative position and orientation information of the HMD4 and the virtual content 5 and its history, and three-dimensional model information of the virtual content 5. The dynamic characteristics of the virtual content (static, dynamic), which indicates whether the virtual content 5 is moving, and the dynamic characteristics of the user (static, dynamic), which indicates whether the user experiencing the virtual content is moving, may also be acquired.

[0050] In step S1063, the play area calculation unit 105 determines, among the attribute information of the virtual content 5, whether the dynamic characteristics of the content are static and the dynamic characteristics of the user are static.

[0051] If the dynamic characteristics of the content are static and the dynamic characteristics of the user are static, proceed to step S1066; otherwise, proceed to step S1064.

[0052] In step S1064, the play area calculation unit 105 determines, from the attribute information of the virtual content 5, whether the content's dynamic characteristics are dynamic and the user's dynamic characteristics are static. If the content's dynamic characteristics are dynamic and the user's dynamic characteristics are static, the process proceeds to step S1067; otherwise, it proceeds to step S1065.

[0053] In step S1065, the play area calculation unit 105 determines, among the attribute information of the virtual content 5, whether the dynamic characteristics of the content are static and the dynamic characteristics of the user are dynamic. If the dynamic characteristics of the content are static and the dynamic characteristics of the user are dynamic, the process proceeds to step S1068; otherwise, it proceeds to step S1069.

[0054] In step S1066, the play area calculation unit 105 calculates a bounding box that encompasses all the three-dimensional point clouds included in the three-dimensional model of the content and the points representing the user's position. The data including the bounding box and its position and orientation is registered as the play area. The method of Gill et al. (described later) is used to calculate the bounding box. The method of Gill et al. is called Efficiently Approximating the Minimum-Volume Bounding Box of a Point Set in Three Dimensions. In this embodiment, the virtual content 5 is a virtual monitor, and the virtual content and the user are assumed to be stationary relative to each other. Therefore, the dynamic characteristics of the virtual monitor are set to static, and the dynamic characteristics of the user are set to static, and the processing in step S1066 is executed.

[0055] In step S1067, the play area calculation unit 105 calculates a bounding box that represents the area to which the content has moved. This process is performed based on the history of the position and orientation of the content's movement and the three-dimensional model, which are included in the attribute information of the virtual content 5 obtained in step S1063. The bounding box that represents the area to which the virtual content has moved is recalculated using the bounding box of the three-dimensional model of the virtual content as a reference, along with the point cloud of the three-dimensional model at the destination and the vertices of the reference bounding box. By repeating this process each time the content moves, the bounding box expands by the amount of the area moved from the start. Next, a bounding box is calculated that encompasses all the points representing the user's position and the vertices of the bounding box representing the virtual content's movement area, and the data including the bounding box and its position and orientation is registered as the play area.

[0056] In step S1068, the play area calculation unit 105 calculates a bounding box representing the area the user has moved to, based on the history of the position and orientation of the HMD4 as the user moves. The calculation of the bounding box representing the area the user has moved to involves calculating a bounding box that encompasses all the point cloud representing the history of the user's movement. Next, it calculates a bounding box that encompasses all the vertices of the bounding box representing the area the user has moved to and all the three-dimensional point cloud of the virtual content, and registers the data including the bounding box and its position and orientation as the play area.

[0057] In step S1069, the play area calculation unit 105 calculates a bounding box representing the area where the content has moved, based on the history of the content's position and orientation and the three-dimensional model. This bounding box is calculated in the same way as in step S1067. In addition, it also calculates a bounding box representing the area where the user has moved. This bounding box is calculated in the same way as in step S1068. Next, it calculates a bounding box that encompasses all the vertices that make up the bounding box showing the area where the content has moved and the bounding box showing the area where the user has moved. Finally, it registers the data, including the bounding boxes and their position and orientation, as a play area.

[0058] <Effects> As described in this embodiment, by superimposing the play area of ​​the second user 2 onto the video of the first user 1, the first user 1 can identify and recognize the MR play area of ​​the second user 2. Therefore, the first user 1 can avoid collisions with the second user 2 by passing around the play area, thereby preventing an unconscious decrease in the second user 2's immersion in the MR.

[0059] <Variation> <Variation 1-1> In Example 1, the user's movement area and the content's movement area were calculated based on the position and orientation history when calculating the play area, but this is not necessarily the only method. For example, the user's movement area and the content's movement area may be set in advance for each piece of content. The content creator may set the user's movement area and the content's movement area in advance, or a second user may set them in advance. This setting information may also be included in the user information.

[0060] <Variation 1-2> In Example 1, the user's movement area and the content's movement area were calculated based on the position and orientation history since the start of communication, but this is not necessarily the only method. For example, the movement area may be calculated by referring to the position and orientation history for a predetermined period of time from the current position and orientation, and encompassing the position and orientation history for that predetermined period of time. Specifically, if the time for referring to the past position and orientation history is set to 1 minute, the movement area will be calculated as the area encompassing the position data for the past 1 minute from the current time. In this case, at the start of communication, the position and orientation history for a predetermined period of time from the current time may be received and defined as the movement area at the start of communication.

[0061] <Variation 1-3> In Example 1, the play area was calculated based on the user's dynamic characteristics and the content's dynamic characteristics. However, it is not always necessary to consider the dynamic characteristics of both the user and the content; considering only one of them is acceptable. Alternatively, regardless of dynamic characteristics, the play area may be calculated as only the region surrounding the second user and the virtual content. In that case, the method for calculating the play area is the same as when the user's dynamic characteristics and the content's dynamic characteristics are static.

[0062] <Variation 1-4> In Example 1, the play area was represented by a bounding box that encompassed the user's movement area and the content's movement area, but this is not necessarily the only way. For example, the bounding box could be extended vertically to the area in contact with the ground. Furthermore, it does not necessarily have to be a bounding box; any method that represents an area encompassing the user's movement area and the content's movement area is acceptable.

[0063] <Variations 1-5> In Example 1, the play area of ​​the second user 2 was simply displayed on the HMD3 of the first user 1. However, a configuration that displays the play area in a way that warns the first user 1 is also possible. An example of such a configuration will be described below.

[0064] The differences in the device configuration compared to Example 1 include the addition of a warning intensity calculation unit 107 and changes to the input data of the content generation unit 106; therefore, only these parts will be explained.

[0065] The warning intensity calculation unit 107 calculates the warning intensity based on the play area calculated by the play area calculation unit 105 and the relative position and attitude calculated by the relative position and attitude calculation unit 104 or the position and attitude of the first user 1 acquired by the first position and attitude acquisition means.

[0066] The content generation unit 106 generates content to be displayed on the HMD4 based on the relative position and orientation calculated by the relative position and orientation calculation unit 104, the play area calculated by the play area calculation unit 105, and the warning intensity calculated by the warning intensity calculation unit 107.

[0067] Figure 7 is a flowchart showing the processing procedure in modified examples 1-5. The only differences from Example 1 are the addition of step S1160 and the processing in step S1070; therefore, explanations of other processes are omitted.

[0068] In step S1160, the warning intensity calculation unit 107 calculates a warning intensity between 0 and 1 based on the relative position and orientation obtained in step S1050 and the play area calculated in step S1060. Details of the process in step S1160 will be described later.

[0069] In step S1070, the content generation unit 106 generates virtual content that represents the area of ​​the play area of ​​the second user 2 to be displayed on the HMD3 of the first user 1. This process is performed based on the relative position and orientation calculated in step S1050 and the play area calculated in step S1060. Specifically, the higher the warning intensity, the greater the degree to which the virtual content representing the play area is provided as a visual stimulus to make it easier for the first user to perceive. For example, the transparency of the virtual content is determined based on the warning intensity. The transparency is calculated using the following formula.

[0070] Transparency = 1 - Warning Intensity As a result, the higher the warning intensity, the lower the transparency becomes, and the higher the contrast with the background.

[0071] Next, based on the second position and orientation obtained in step S1030 and the relative position and orientation obtained in step S1050, the generated virtual content is converted to a position and orientation with the second user's position and orientation as the origin.

[0072] The details of the process in step S1070 are described below.

[0073] Figure 8 is a flowchart showing the process for calculating warning intensity. Here, we will explain an example of calculating warning intensity based on the distance between the HMD3 and the play area.

[0074] In step S1071, the warning intensity calculation unit 107 obtains the relative position and orientation of HMD3 and HMD4 calculated in step S1050.

[0075] In step S1072, the warning intensity calculation unit 107 obtains the bounding box and its position and orientation of the area representing the play area calculated in step S1060.

[0076] In step S1073, the warning intensity calculation unit 107 calculates the distance between the area representing the play area and the HMD3. First, it calculates the relative position and orientation of the bounding box representing the play area and the HMD4. Next, using the relative position and orientation of the HMD3 and HMD4 obtained in step S1071, and the relative position and orientation of the play area and the HMD4, it calculates the position and orientation of the play area as seen from the HMD3, and converts the vertex coordinates of the play area to coordinates as seen from the HMD3. Next, for each face constituting the bounding box representing the play area, it calculates the distance from the HMD3 and finds the minimum distance d HMD-playArea This is calculated as the distance between the play area and the HMD3. Here, it does not necessarily have to be the distance to a surface; it may also be the distance to the nearest vertex that makes up the play area.

[0077] In step S1074, the warning intensity calculation unit 107 calculates the distance d calculated in step S1073. HMD-playArea Based on this, the warning intensity is calculated. Here, the warning intensity is expressed in the range of 0 to 1. The maximum distance at which a warning is required is D. max Therefore, the warning intensity can be calculated using the following formula.

number

[0078] As a result, the closer the distance between the first user 1 and the play area, the stronger the warning becomes, making it easier for the first user 1 to perceive the second user 2's play area.

[0079] <Effects> In addition to the effects described in Example 1, by changing the display mode of the play area based on the warning intensity, the first user 1 becomes more likely to perceive the second user 2's play area within the play area. This improves the possibility of avoiding collisions with the second user 2 and a decrease in the second user's immersion in MR.

[0080] In Modification 1-5, the warning intensity was calculated based on the distance between the play area and the HMD3, but this is not necessarily limited to that. For example, the movement speed of the first user may also be reflected in the calculation of the warning intensity. Specifically, the warning intensity based on movement speed may be A velocity , the minimum speed at which a warning is required v min The speed at which the warning intensity is maximized is v max When the current movement speed of the first user 1 is v, the warning intensity A depends on the movement speed. velocity It can be expressed by the following formula.

number

[0081] Furthermore, the warning intensity based on distance, which was used in the modified examples 1-5, is A distanceThen, the warning intensity A based on the distance and the moving speed can be expressed by the following formula. A = w1A distance + w2A velocity Note that w1 and w2 represent the weights of each element, and the sum is assumed to be 1.

[0082] In addition to the distance and the moving speed, the warning intensity may also be calculated based on the traveling direction of the first user and the position of the play area. When the traveling direction vector of the first user is p and the vector connecting the position of the play area and HMD3 is q, the angle θ formed by p and q can be obtained by the following formula.

Equation

[0083] Let θ be the θ when the warning intensity is maximum, and θ be the θ when the warning intensity is minimum. Let the warning intensity based on the traveling direction be A min , then A max can be calculated by the following formula. direction Then, A direction is calculated by the following formula.

Equation

[0084] In addition, when the warning intensity A is the integrated warning intensity of the warning intensity based on the distance, the warning intensity based on the moving speed, and the warning intensity based on the traveling direction, the warning intensity can be calculated by the following formula. A = w1A distance + w2A velocity + w3A direction Note that w1, w2, and w3 represent the weights of each element, and the sum is assumed to be 1.

[0085] Also, as long as the index can represent the warning intensity with an index other than those listed here, it is not limited to this.

[0086] In Modification 1-5, the content generation unit 106 modified the visual stimulus by changing the transparency based on the warning intensity, but it is not necessarily limited to this. For example, the content may vibrate or flash as the warning intensity increases. Specifically, the frequency of vibration or flashing when the warning intensity A is at its maximum may be set to f max The frequency at which the warning intensity A is minimized is f min The frequency f of the content's vibration or flashing can be calculated using the following formula: f=f min +A·(f max -f min ) Additionally, users may be allowed to customize the type of visual stimulus (transparency, vibration, flashing).

[0087] Furthermore, any method of modifying visual stimuli based on warning intensity is acceptable, but the method is not limited to this one, and the visual stimuli themselves are not limited to those described herein.

[0088] In Modification 1-5, the content generation unit 106 used transparency, vibration, and flashing as visual stimuli to express warning intensity, but it is not necessarily limited to these. For example, it could be a visual stimulus that generates warning content (such as warning signs or prohibition signs) in the area surrounding the play area. The larger the warning intensity, the larger the size of the warning content or the more of it it may be. It may also be possible to switch the type of content displayed based on the warning intensity. For example, when the warning intensity is low, warning content may be displayed, and when the warning intensity is high, prohibition content may be generated. Furthermore, it is not limited to the examples given here; any method that changes the display manner of the generated content according to the warning intensity is acceptable.

[0089] [Example 2] <Overview> In Example 1, the play area calculation unit 105 calculated the play area based on the attribute information of the virtual content 5. In this embodiment, a method for determining the play area based on user attribute information will be described. In this embodiment, the user attribute information is assumed to be the age of the second user 2. However, it is not limited to this, and any information that expresses the user's characteristics, such as the user's height, gender, relationship with the first user 1, and presence or absence of disabilities, may be used. The user attribute information is assumed to be held by the user information holding unit 44.

[0090] This embodiment is implemented with the same configuration as in Embodiment 1, using the information processing device 100 shown in Figure 4. Since the input and output data are the same as in Embodiment 1, their explanation is omitted here.

[0091] The processing flow of the information processing device 100 in this embodiment is described below. The flowchart is the same as in Embodiment 1 and is shown in Figure 5. In this embodiment, only the processing in steps S1040 and S1060 differs from that of Embodiment 1, so the processing other than steps S1040 and S1060 will not be explained.

[0092] In step S1040, the play area calculation unit 105 acquires user information for the second user 2. In this embodiment, the user information includes attribute information of the MR virtual content that the second user 2 is experiencing (relative position and orientation information of HMD4 and virtual content 5, and three-dimensional model information of virtual content 5), and user attribute information (age of the second user 2).

[0093] In step S1060, the play area calculation unit 105 calculates the play area of ​​the second user 2 based on the user attribute information included in the user information acquired in step S1040.

[0094] Figure 9 is a flowchart showing the detailed processing of the play area calculation unit 105 in Example 2.

[0095] In step S2061, the play area calculation unit 105 obtains the second position and orientation acquired in step S1020.

[0096] In step S2062, the play area calculation unit 105 obtains the relative position and orientation of the HMD4 and virtual content included in the user information acquired in step S1040.

[0097] In step S2063, the play area calculation unit 105 obtains a three-dimensional model of the virtual content included in the user information acquired in step S1040.

[0098] In step S2064, the play area calculation unit 105 obtains user attribute information included in the user information obtained in step S1040.

[0099] In step S2065, the play area calculation unit 105 calculates the play area for the second user 2. This process is performed based on the relative position and orientation of the HMD4 and the virtual content, the three-dimensional model of the virtual content, and the user attribute information obtained in steps S2062 to S2064.

[0100] First, the reference play area is calculated based on the relative position and orientation of HMD4 and virtual content 5, and the 3D model of virtual content 5. Specifically, the vertex coordinates of the 3D model of virtual content 5, with HMD4 as the origin, are calculated using the relative position and orientation of HMD4 and virtual content 5. Then, the play area is calculated by calculating a bounding box that encompasses the origin and all the vertices of the 3D model. The area represented by this bounding box is used as the reference play area.

[0101] Next, the size of the play area is calculated based on the age of the second user, User 2, which is user attribute information. Specifically, a coefficient α is calculated to multiply the size of the standard play area in order to scale it. For example, if the user is 17 years old or younger, the coefficient α is set to 1.3; if they are 18 to 64 years old, the coefficient α is set to 1.0; and if they are 65 years old or older, the coefficient α is set to 1.3. The coefficient α is set in advance according to the age group. Next, the virtual content representing the standard play area is scaled by the coefficient α. This allows for the reduction of risk for users in age groups that pose a higher risk from a safety perspective by expanding the play area.

[0102] <Effects> In addition to the effects described in Example 1, adjusting the size of the play area according to age can further reduce the risk of collisions.

[0103] <Variation> In Example 2, when calculating the play area, a scaling factor was set based on the user's age to adjust the size of the play area. However, it is not necessary to set the factor based on age. For example, the factor α could be determined based on the user's height. Specifically, if the reference height is x cm and the height of the second user 2 is y cm, the value of the factor α could be changed by ±0.1 for every ±10 change in the difference y-x cm from the reference height. This is not the only method that reflects the size of the play area.

[0104] Furthermore, a scaling factor may be set based on the relationship between the first user 1 and the second user 2. The stronger the relationship, the smaller the play area; and the weaker the relationship, the larger the play area. As a numerical value representing the relationship, for example, the first user's face may be recognized when they meet, and the number of times they have met may be kept, with the relationship increasing as the number of meetings increases. Alternatively, the relationship may increase as the number of messages exchanged on social media increases. Alternatively, a relationship may be set in advance for each individual user, and the scaling factor may be set based on that value. Moreover, the system is not limited to the examples given here; any indicator that represents the relationship between the first user 1 and the second user 2 is acceptable.

[0105] Alternatively, the scaling factor may be set based on the movement history of the second user 2. For example, the position and orientation history information of the second user 2 may be received as user attribute information, the position covariance of the second user 2 may be calculated, and the scaling factor may be set to increase as the value of the covariance increases. Furthermore, any indicator that shows the movement trends of the second user 2 that can be extracted from the position and orientation history of the second user 2 is not necessarily limited to this.

[0106] Examples 1 and 2 describe cases in which the information processing device 100 is provided on the HMD3. The information processing device does not necessarily have to be provided on the HMD and may be located externally. For example, it may be located on a mobile terminal such as a smartphone connected to the HMD3. Communication between the first user 1 and the second user 2 may be conducted between the mobile terminals to which their respective HMDs are connected, or communication may be conducted between the HMD and the mobile terminal.

[0107] Furthermore, the information processing device 100 does not necessarily have to be located in a device owned by the first user 1. For example, the information processing device 100 may be located in a server or computer that centrally manages information for each user. Alternatively, it may be implemented as a computer program and executed by a computer.

[0108] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. At least a part of each embodiment may be combined. [Explanation of Symbols]

[0109] 1. The first user 2. Second User 3 HMD 4 HMD 5. Virtual Content 6 Play Areas 10 CPU 20 ROM 30 RAM 31 Input section 32 First position and attitude calculation unit 33 Display section 40 Input Interfaces 41 Input section 42 Second position and attitude calculation unit 43 Transmitter 44 User Information Storage Unit 50 Output Interfaces 60 bus 100 Information Processing Devices 101 First position and attitude acquisition unit 102 Second position and attitude acquisition unit 103 User Information Acquisition Unit 104 Relative position and attitude calculation unit 105 Play Area Calculation Unit 106 Content Generation Department 107 Warning intensity calculation unit

Claims

1. A first position and orientation acquisition means for acquiring the position and orientation of a first user, A second position and orientation acquisition means for acquiring the position and orientation of a second user, A relative position and attitude calculation means that calculates the relative position and attitude between the first user and the second user based on the first position and attitude and the second position and attitude, The play area calculation means calculates a play area in which the second user experiences virtual content, Content generation means that generates content that allows the first user to identify the play area based on the relative position and orientation and the play area, An information processing device characterized by comprising:

2. The system further comprises a user information acquisition means for acquiring user information of the second user, The play area calculation means calculates the play area based on the user information. The information processing apparatus according to feature 1.

3. The system further includes a warning intensity calculation means for calculating a warning intensity that indicates the degree to which the first user should be warned about the play area. The content generation means generates the content such that the degree of visual stimulation increases as the warning intensity increases. The information processing apparatus according to feature 2.

4. The user information includes relative position and orientation information between the second user and the virtual content, and three-dimensional model information of the virtual content. The play area calculation means calculates a region encompassing the second user and the virtual content as a play area based on the relative position and orientation information and the three-dimensional model. The information processing apparatus according to feature 2.

5. The user information includes at least one of the dynamic characteristics of the virtual content and the dynamic characteristics of the second user. The play area calculation means calculates a play area as an area encompassing the movement area of ​​the virtual content and the movement area of ​​the second user, based on at least one of the dynamic characteristics of the virtual content and the dynamic characteristics of the second user. The information processing apparatus according to claim 4, characterized by the above.

6. The user information includes at least one of the following user attribute information of the second user: the second user's age, height, the relationship between the second user and the first user, and the second user's position and posture history. The play area calculation means calculates a scaling factor for the play area based on the user attribute information, and scales the play area based on the scaling factor. The information processing apparatus according to claim 4, characterized by the above.

7. The warning intensity calculation means calculates the distance between the first user and the play area based on the relative position and orientation of the first user and the second user calculated by the relative position and orientation calculation means and the play area calculated by the play area calculation means, and calculates the warning intensity to increase as the distance decreases. The information processing apparatus according to claim 3, characterized by the following:

8. The warning intensity calculation means calculates the movement speed of the first user based on the position and orientation of the first user acquired by the first position and orientation acquisition means, and calculates the warning intensity so that the greater the movement speed, the greater the warning intensity. The information processing apparatus according to claim 3, characterized by the following:

9. The warning intensity calculation means calculates the angle between a vector from the first user to the second user calculated based on the relative position and orientation of the first user and the second user calculated by the relative position and orientation calculation means, and a vector indicating the direction of travel of the first user based on the first position and orientation acquired by the first position and orientation acquisition means, and calculates the warning intensity so that the smaller the angle, the greater the warning intensity. The information processing apparatus according to claim 3, characterized by the following:

10. A first position and orientation acquisition step to acquire the position and orientation of the first user, A second position and orientation acquisition process to acquire the position and orientation of a second user, A relative position and attitude calculation step that calculates the relative position and attitude between the first user and the second user based on the first position and attitude and the second position and attitude, The play area calculation process involves calculating a play area in which the second user experiences virtual content, A content generation step that generates content that allows the first user to identify the play area based on the relative position and orientation and the play area, An information processing method performed by an information processing device, characterized by comprising the above.

11. A computer program for causing a computer to execute each step of the information processing method described in claim 10.

Images