Information processing device and computer readable storage medium

Abstract

The present invention provides an information processing device and a computer-readable storage medium, which can easily set a shared area compared to a case where a user specifies coordinates or real objects in order to set a shared area in a three-dimensional space. A processor sets an area where the fields of view of multiple users intersect in the three-dimensional space where the multiple users are located, i.e., an intersection area, as an area shared by the multiple users, i.e., a shared area.

Description

Technical Field

[0001] The present invention relates to an information processing device and a computer-readable storage medium. Background Art

[0002] There is known a technology for virtually setting a shared area, which is an area shared by a plurality of users, in a real space by using a technology such as AR (Augmented Reality) or Mixed Reality (MR).

[0003] Patent document 1 discloses a device that stores position data indicating the position of a virtual object (object) displayed superimposed on a real space on a screen of a terminal device, sets a virtual shared area in the real space, and allows or denies the display of each virtual object on the terminal device depending on whether the virtual object is located in the shared area.

[0004] Patent document 2 discloses an information communication terminal that obtains a captured image of a real space and a virtual object inherent to a user, detects an event related to information sharing based on the captured image, detects the event simultaneously with another terminal that captures the real space from a different direction, and when the event indicates the start of sharing at a position specified by collaboration with the other terminal, a shared area in which virtual objects can be shared with the other terminal is displayed overlappingly at the position of the captured image.

[0005] [Prior art literature]

[0006] [Patent Document]

[0007] Patent Document 1: Japanese Patent Application Publication No. 2012-168646

[0008] Patent Document 2: Japanese Patent Application Publication No. 2017-84100 Summary of the invention

[0009] [Problems to be solved by the invention]

[0010] In addition, as a method of virtually setting a shared area in a three-dimensional space, there is a method in which the user specifies the coordinates of the shared area or specifies a real object. In these methods, it takes time and effort for the user to specify the coordinates or the real object.

[0011] An object of the present invention is to provide a structure that can set a shared area more easily than a case where a user specifies coordinates or a real object in order to set a shared area in a three-dimensional space.

[0012] [Technical means to solve the problem]

[0013] The present invention is an information processing device having a processor that sets an intersection area, which is an area where fields of vision of multiple users intersect in a three-dimensional space where multiple users are located, as an area shared by the multiple users, which is a shared area.

[0014] The present invention is a computer-readable storage medium that enables a computer to set an area where the fields of vision of multiple users intersect in a three-dimensional space where multiple users are located, namely, an intersection area, as an area shared by the multiple users, namely, a shared area.

[0015] [Effects of the Invention]

[0016] According to the present invention, it is possible to provide a configuration that allows a shared area to be set more easily than in a case where a user specifies coordinates or a real object in order to set a shared area in a three-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a block diagram showing the structure of the information processing system according to the present embodiment.

[0018] Figure 2 This is a block diagram showing the hardware configuration of an information processing device.

[0019] Figure 3 This is a block diagram showing the functional structure of an information processing device and a server.

[0020] Figure 4 This is a diagram showing a shared area management table.

[0021] Figure 5 This is a diagram showing a history management table.

[0022] Figure 6 This is a flowchart showing the processing performed by the information processing system according to the present embodiment.

[0023] Figure 7 is a diagram schematically showing users located in a three-dimensional space.

[0024] Figure 8 is a diagram schematically showing users located in a three-dimensional space.

[0025] Fig. 9 is a diagram schematically showing users located in a three-dimensional space.

[0026] Fig.10 is a diagram schematically showing users located in a three-dimensional space.

[0027] Fig.11 is a diagram schematically showing users located in a three-dimensional space.

[0028] Fig.12 is a diagram schematically showing users located in a three-dimensional space.

[0029] Fig.13 is a diagram schematically showing users located in a three-dimensional space.

[0030] Fig.14 This is a diagram of the user viewed from the Z-axis direction.

[0031] Fig.15 This is a diagram where the user is viewed from the X-axis direction or the Y-axis direction.

[0032] Fig.16 is a diagram schematically showing users located in a three-dimensional space.

[0033] Fig.17 is a diagram schematically showing users located in a three-dimensional space.

[0034] Fig.18 is a diagram schematically showing users located in a three-dimensional space.

[0035] Fig.19 is a diagram schematically showing users located in a three-dimensional space.

[0036] Fig. 20 is a diagram schematically showing users located in a three-dimensional space.

[0037] Fig.21 A diagram representing an image of a user.

[0038] Fig. 22 is a diagram schematically showing users located in a three-dimensional space.

[0039] [Explanation of Symbols]

[0040] 10.12: AR glasses

[0041] 14: Information processing device

[0042] 16: Server DETAILED DESCRIPTION

[0043] Reference Figure 1 The information processing system according to this embodiment is described. Figure 1 An example of the hardware configuration of the information processing system of this embodiment is shown in FIG.

[0044] The information processing system of this embodiment is a system that virtually sets an area shared by multiple users, namely, a shared area, in a three-dimensional space by using technologies such as augmented reality (AR), mixed reality (MR), or virtual reality (VR).

[0045] The "three-dimensional space" here can be a real space or a virtual space. For example, a shared area can be virtually set in a real space by using AR technology or MR technology, or a shared area can be virtually set in a virtual space by using VR technology.

[0046] Here, as an example, an example of virtually setting a shared area in a real space by using AR technology or MR technology will be described. More specifically, it is assumed that AR technology is used.

[0047] For example, an AR glasses, which is a glasses-type terminal device that uses AR technology, is used. AR glasses are devices with display functions, photography functions and communication functions. Moreover, AR glasses have various sensors such as gyro sensors, acceleration sensors, geomagnetic sensors, and global positioning system (GPS) sensors. The tilt, acceleration and orientation of the AR glasses are measured by gyro sensors, acceleration sensors and geomagnetic sensors, and the direction and posture of the AR glasses are calculated based on the results of the measurements. The calculation can be performed by the AR glasses or by the information processing device 14. Moreover, the position of the AR glasses (such as latitude, longitude and altitude) is measured by a GPS sensor. Information indicating the measurement results of each sensor is sent to the information processing device 14.

[0048] Instead of AR glasses, a head mounted display (HMD) using AR technology, or MR glasses (i.e., glasses-type terminal devices) or HMD using MR technology can be used. In addition, instead of AR glasses, a mobile terminal such as a smartphone, mobile phone, or game console with a camera can be used. When VR technology is used, VR glasses, HMD, or mobile terminals using VR technology can be used instead of AR glasses.

[0049] In the following, objects in real space are sometimes referred to as "real objects". Virtual objects are concepts that are compared with real objects, such as images, strings, graphics, or other information. Images are still images, dynamic images, or a combination of them. For example, a shared area virtually set in three-dimensional space displays virtual objects.

[0050] In the case of using AR technology or MR technology, the shared area can be defined as a specific area set at a relative position with the position of the real object as a reference in association with the real object in the real space, or it can be defined as a specific area on the coordinates of the real space without being associated with the real object. In the case of using VR technology, the shared area can be defined as a specific area set at a relative position with the position of the virtual object as a reference in association with the virtual object in the virtual space, or it can be defined as a specific area on the coordinates of the virtual space without being associated with the virtual object.

[0051] Moreover, the shared area is an area shared by users participating in the information processing system of the present embodiment, and is an area not shared by users not participating in the information processing system. Users participating in the information processing system are, for example, users who have logged into the information processing system, or users who have logged into a service (such as a meeting) provided by the information processing system. Furthermore, one or more groups may be registered in the information processing system, and a shared area may be set for each group. In this case, the shared area set for the group is shared by users belonging to the group, and is not shared by users who do not belong to the group. Users who share the shared area are allowed to access the shared area, display the shared area, display virtual objects in the shared area, view virtual objects in the shared area, and edit virtual objects in the shared area, for example. Some of these functions may also be restricted based on the permissions that the user has.

[0052] Figure 1 In the example shown, the information processing system of the present embodiment includes AR glasses 10 , AR glasses 12 , an information processing device 14 , and a server 16 . Figure 1 , two AR glasses are shown, but the information processing system may include more than three AR glasses. The AR glasses 10, AR glasses 12, information processing device 14, and server 16 have the function of communicating with other devices via a communication path N such as a local area network (LAN) or the Internet. Each device can also communicate with other devices by using short-range wireless communication such as Bluetooth (registered trademark).

[0053] The AR glasses 10 include a glasses-type display device 10a and a camera 10b that captures the real space in the direction of sight of the user wearing the AR glasses 10. Similarly, the AR glasses 12 include a glasses-type display device 12a and a camera 12b. Hereinafter, the image generated by the camera shooting is referred to as a "captured image". The display device 10a and the display device 12a can be either a see-through display device or a non-see-through display device. Through AR technology, objects are virtually displayed on the display device 10a and the display device 12a relative to real objects. The so-called virtual display of objects relative to real objects refers to overlapping and displaying a virtual object as an example of an image representing the object on the screen of the display device 10a and the display device 12a in the real space. For example, considering the parallax of the user's two eyes, a screen for the left eye and a screen for the right eye are generated and displayed on the respective display devices on the left and right. When the user looks at the real space through the display device 10a or the display device 12a, the virtual object is displayed overlappingly as if it actually exists in the real space.

[0054] In this embodiment, it is assumed that the AR glasses 10 are worn by user A, the AR glasses 12 are worn by user B, and user A and user B log in to the service provided by the information processing system of this embodiment.

[0055] The information processing device 14 is, for example, a desktop or note-type personal computer (hereinafter referred to as "PC"), a tablet terminal, a smart phone or a workstation, and is a device that supports information sharing between multiple users. For example, the information processing device 14 virtually sets a shared area shared by multiple users in a three-dimensional space. When AR technology or MR technology is used, a shared area is virtually set in the real space, and when VR technology is used, a shared area is set in the virtual space. For example, data defining the shared area, i.e., shared area data, is stored in the information processing device 14 or the server 16, and the information processing device 14 sets the shared area based on the shared area data. In addition, the information processing device 14 can also display the information shared by multiple users in the real space by overlapping it on the screen of the AR glasses.

[0056] The server 16 is a device that stores and manages information related to the shared area, etc. The server 16 may also be included in the information processing device 14.

[0057] Below, refer to Figure 2 The hardware structure of the information processing device 14 will be described. Figure 2 An example of the hardware configuration of the information processing device 14 is shown in FIG.

[0058] The information processing device 14 includes, for example, a communication device 18 , a UI 20 , a memory 22 , and a processor 24 .

[0059] The communication device 18 is a communication interface (for example, a network interface) including a communication chip, etc., and has a function of transmitting data to another device and a function of receiving data transmitted from another device.

[0060] UI20 is a user interface, including at least one of a display device and an operating device. The display device is a liquid crystal display or an electroluminescence (EL) display, etc. The operating device is a keyboard, an input key, or an operating panel, etc. UI20 can also be a UI such as a touch panel that has both a display device and an operating device.

[0061] The memory 22 is a device that constitutes one or more storage areas for storing data. The memory 22 is, for example, a hard disk drive, various memories (such as random access memory (RAM), dynamic random access memory (DRAM) or read only memory (ROM), etc.), other storage devices (such as optical disks, etc.), or a combination thereof.

[0062] The processor 24 is configured to control the operation of each part of the information processing device 14. The processor 24 may also include a memory. In addition, the processor 24 virtually sets the area where the fields of view of the multiple users intersect in the three-dimensional space where the multiple users are located, that is, the intersection area, as a shared area shared by the multiple users. For example, the processor 24 can virtually set the shared area in the real space, and can also virtually set the shared area in the virtual space. The processing will be described in detail later.

[0063] Below, refer to Figure 3 The functions of the information processing device 14 and the server 16 will be described. Figure 3 , an example of the functional configuration of the information processing device 14 and the server 16 is shown.

[0064] The information processing device 14 includes a calculation unit 26, a display control unit 28, a selection unit 30, and a communication unit 32. The server 16 includes a shared area management unit 34 and a history management unit 36.

[0065] The calculation unit 26 is configured to calculate an intersection area where the fields of vision of multiple users intersect in a three-dimensional space (e.g., a real space or a virtual space). For example, the calculation unit 26 calculates the direction and posture of the AR glasses 10 in the real space based on the measurement results obtained by various sensors provided in the AR glasses 10. Since the AR glasses 10 are worn by the user A, the direction and posture correspond to the direction and posture of the user A in the real space. The calculation unit 26 estimates the direction as the direction of the line of sight of the user A, and estimates the area with a predetermined size including the line of sight as the field of vision of the user A in the real space. For example, the calculation unit 26 may also estimate the area where the line of sight of the user A is located at the center and becomes wider as it is farther away from the user A as the field of vision of the user A. As another example, a sensor (e.g., a known sensor) for measuring the line of sight of the user A may also be provided in the AR glasses 10. In this case, the calculation unit 26 may also estimate the area including the line of sight measured by the sensor as the field of vision of the user A. As another advantage, the calculation unit 26 can also estimate the line of sight of user A relative to the actual object based on the image recognition result of the captured image generated by the camera 10b of the AR glasses 10, and estimate the area including the line of sight as the field of view of user A. Similarly, the calculation unit 26 estimates the field of view of user B in the real space.

[0066] The calculation unit 26 calculates the intersection area where the field of view of user A and the field of view of user B intersect in a three-dimensional space (e.g., a real space or a virtual space), sets the calculated intersection area as a candidate for the shared area of ​​user A and user B, and generates shared area data defining the shared area. When using AR technology or MR technology, the calculation unit 26 calculates the intersection area where the fields of view of each user intersect in the real space based on the fields of view of each user in the real space inferred in this way. When using VR technology, the calculation unit 26 calculates the intersection area where the fields of view of each user intersect in the virtual space. The processing when using VR technology will be described in detail later. The shared area data includes, for example, information for identifying the shared area, i.e., a shared area ID, information indicating the number of vertices in the shared area, information on the coordinates of the vertices in the three-dimensional space, and information for identifying the user who shares the shared area, i.e., a user ID.

[0067] Furthermore, the calculation unit 26 may calculate the candidates of the shared area to be presented to the user when no intersection area is formed.

[0068] The display control unit 28 is configured to virtually overlap and display the intersection area calculated by the calculation unit 26 or the shared area used in the past as a candidate for the shared area in a three-dimensional space (e.g., a real space or a virtual space). The shared area data defining the shared area used in the past is managed by the history management unit 36 ​​described later. For example, the display control unit 28 virtually displays the candidate for the shared area on the display screen of the display device 10a of the AR glasses 10, overlapping the real space. Specifically, the display control unit 28 calculates the display position of the candidate for the shared area on the display screen of the display device 10a based on the position of the candidate for the shared area in the real space and the direction and posture of the AR glasses 10, and displays the candidate for the shared area at the calculated display position. The same is true for the AR glasses 12 worn by user B. When using VR technology, the shared area is displayed in a virtual space.

[0069] The display control unit 28 may display information such as an object for the user to recognize the candidates of the sharing area. For example, the display control unit 28 displays a line surrounding the candidates of the sharing area, or displays the candidates of the sharing area in a semi-transparent manner.

[0070] The selection unit 30 is configured to select a shared area actually used by the user A and the user B from the candidates of the shared area displayed by the display control unit 28. For example, the selection unit 30 may select a shared area based on the selection of the user A or the user B, or may select a shared area based on the selection of a user with selection authority, or may select the candidates of the shared area selected by both the user A and the user B as the shared area actually used. For example, the user selects the shared area by gesture, voice, or operation of the user's terminal device.

[0071] The communication unit 32 is configured to transmit information to the server 16 and receive information transmitted from the server 16. For example, the communication unit 32 transmits the shared area data generated by the calculation unit 26 to the server 16 and receives the shared area data defining the shared area used in the past from the server 16.

[0072] The shared area management unit 34 is configured to manage the shared area. For example, the candidates of the shared area or the shared area actually used in the past are managed by the shared area management unit 34.

[0073] The history management unit 36 ​​is configured to manage the shared area used in the past.

[0074] In addition, the shared area data itself may be stored in the server 16 or in the information processing device 14 .

[0075] The calculation unit 26, the display control unit 28, the selection unit 30, and the communication unit 32 are implemented by the processor 24 included in the information processing device 14. For their implementation, the memory 22 and the like can also be used. In addition, the shared area management unit 34 and the history management unit 36 ​​are implemented by the processor included in the server 16. For their implementation, the memory and the like can also be used. In addition, the AR glasses 10 and the AR glasses 12 can also have all or part of the functions of the information processing device 14, and can also have all or part of the functions of the server 16.

[0076] Below, refer to Figure 4 The information managed by the shared area management unit 34 will be described. Figure 4 An example of a shared area management table for managing the shared area is shown in FIG.

[0077] In the shared area management table, for example, the shared area ID, information indicating the number of vertices in the shared area, and information indicating the coordinates of the vertices in a three-dimensional space (for example, a real space or a virtual space) are associated. The coordinates of the vertices are, for example, coordinates on a three-dimensional orthogonal coordinate system (for example, an X coordinate, a Y coordinate, and a Z coordinate), and are coordinates of a relative position with respect to the coordinates of a predetermined position in the three-dimensional space.

[0078] Below, refer to Figure 5 The information managed by the history management unit 36 ​​will be described. Figure 5 An example of a history management table for managing a shared area used in the past is shown.

[0079] In the history management table, for example, information for identifying a shared area used in the past, namely a shared area ID, information indicating the number of vertices in the shared area, information indicating the coordinates of vertices in a three-dimensional space (such as a real space or a virtual space), and information for identifying a user who used the shared area, namely a user ID are associated.

[0080] Below, refer to Figure 6 The processing performed by the information processing system of this embodiment is described. Figure 6 A flowchart representing the processing is shown in FIG.

[0081] First, the user participates in the service provided by the information processing system (such as a meeting using a virtual shared area, etc.) (S01). When the user participates in the service, the service starts. Here, it is assumed that user A and user B log in to the information processing system and participate in the meeting provided by the information processing system. User A wears AR glasses 10, and user B wears AR glasses 12. For example, user A uses AR glasses 10 or a terminal device such as a smart phone to log in to the information processing system. The same is true for user B. A user ID is used when logging in, and the user ID is sent from the device used for login (such as AR glasses or a terminal device, etc.) to the information processing device 14. Here, the user IDs of user A and user B are sent to the information processing device 14. In addition, after starting a service such as a meeting, the user can also join the service midway.

[0082] Next, the display control unit 28 checks whether there is a shared area that was previously used by the same user who participated in the meeting by referring to the history management table managed by the history management unit 36 ​​(S02). Here, since user A and user B participated in the meeting, the display control unit 28 checks whether there is a shared area that was previously used by user A and user B. Specifically, the display control unit 28 checks whether a shared area associated with two user IDs of user A and user B is registered in the history management table.

[0083] If there is a shared area used by user A and user B in the past, that is, if a shared area associated with two user IDs of user A and user B is registered in the history management table (S02, yes), the display control unit 28 registers the shared area as a candidate for the shared area in the shared area candidate list (S03). Then, the processing is transferred to step S04. In addition, the display control unit 28 may also register a shared area that was used in the past by not all users who participated in the meeting, but was used in the past by a predetermined number of users or a predetermined proportion of users as a candidate for the shared area in the shared area candidate list. For example, in the case where three users participated in the meeting, if two of the three users used the same shared area in the past, the display control unit 28 may also register the shared area in the shared area candidate list. As another example, if more than half of the users who participated in the meeting used the same shared area in the past, the display control unit 28 may also register the shared area in the shared area candidate list. For example, if there is a shared area that user A of user A and user B has used in the past, the display control unit 28 may register the shared area in the shared area candidate list.

[0084] If there is no shared area used in the past by user A and user B, that is, if the shared area associated with the two user IDs of user A and user B is not registered in the history management table ( S02 , Yes), the process proceeds to step S04 .

[0085] Next, the calculation unit 26 estimates the direction and posture of user A in the real space based on the measurement results of various sensors provided in the AR glasses 10, and estimates the field of view of user A in the real space based on the estimated results. Similarly, the calculation unit 26 estimates the field of view of user B. Then, the calculation unit 26 calculates the intersection area where the field of view of user A and the field of view of user B intersect in the real space (S04). The same is true when using MR technology. When using VR technology, the calculation unit 26 calculates the intersection area where the field of view of user A and the field of view of user B intersect in the virtual space.

[0086] If there is an intersection region where the visual field of user A and the visual field of user B intersect ( S04 , Yes), the display control unit 28 registers the intersection region as a candidate for a shared region in the shared region candidate list ( S05 ). The process then proceeds to step S06 .

[0087] If there is no intersection area where the visual field of the user A and the visual field of the user B intersect ( S04 , Yes), the process proceeds to step S06 .

[0088] In addition, if there is no intersection area, the calculation unit 26 may also superimpose information such as the guide portion to which the user A and the user B should direct their sight lines in order to form the intersection area on the real space and display it on the display device of the AR glasses of the user A and the user B. The same is true when MR technology is used. When VR technology is used, information such as the guide portion is displayed in the virtual space.

[0089] Next, the calculation unit 26 calculates an area that is within a predetermined distance from each user and that may become a candidate for a shared area (S06). For example, the calculation unit 26 calculates an area that is within a predetermined distance from each position of user A and user B and that may become a candidate for a shared area. The same is true when using MR technology. For example, the position of user A is determined by a GPS sensor provided in the AR glasses 10, and the position of user B is determined by a GPS sensor provided in the AR glasses 12. In addition, the distance from the user may vary depending on the physical characteristics of the user. For example, information indicating the physical characteristics of each user (such as height or arm length, etc.) may be pre-stored in the memory 22 of the information processing device 14, and the calculation unit 26 uses the physical characteristics of each user to calculate an area that is within a predetermined distance from each user. When using VR technology, the calculation unit 26 calculates an area that is within a predetermined distance from each position of user A and user B in the virtual space and that may become a candidate for a shared area.

[0090] If there is an area that may be a candidate for a sharing area within a predetermined distance from each user ( S06 , Yes), the display control unit 28 registers the area as a candidate for a sharing area in the sharing area candidate list ( S07 ). The process then proceeds to step S08 .

[0091] If there is no area that can be a candidate for a shared area within a predetermined distance range from each user ( S06 , Yes), the process proceeds to step S08 .

[0092] The display control unit 28 virtually displays the candidates for the shared area registered in the shared area candidate list on the respective screens of the display device 10a of the AR glasses 10 and the display device 12a of the AR glasses 12, overlapping the real space (S08). The same is true when using MR technology. When using VR technology, the candidates for the shared area are displayed in the virtual space. The display control unit 28 can also display an overview of the candidates, and can also display each candidate in sequence. For example, the display control unit 28 can also display the shared area used in the past at the very beginning, and then display the area that has not been used in the past.

[0093] Each candidate for a shared area registered in the shared area candidate list is associated with a shared area ID and managed by the shared area management unit 34. For example, each candidate for a shared area is registered in Figure 4 The shared area management table shown in FIG.

[0094] Hereinafter, the processing performed by the information processing system of this embodiment will be described in more detail with reference to specific examples.

[0095] Figure 7 In the figure, user A and user B are shown in a three-dimensional space (such as a real space or a virtual space). AR technology is used here. Figure 7 , which shows user A and user B in the real space. The same is true when using MR technology. Below, the processing when the three-dimensional space is the real space and AR technology is used is explained, but the same processing is also performed when MR technology is used. Moreover, when VR technology is used, the three-dimensional space is a virtual space, and each processing described below is performed in the virtual space. Moreover, Figure 7 The X-axis, Y-axis, and Z-axis shown are axes that are orthogonal to each other and form a three-dimensional orthogonal coordinate system in real space.

[0096] User A wears AR glasses 10, and user B wears AR glasses 12. When user A and user B log in to the information processing system and participate in a service (such as a meeting) provided by the information processing system, the service starts. If there is a shared area used by user A and user B in the past, the shared area is virtually displayed in the real space as a candidate for the shared area. For example, area 38 is a shared area used by user A and user B in the past. At this time, the display control unit 28 virtually displays the area 38 on the screen of the display device 10a of the AR glasses 10 and the display device 12a of the AR glasses 12 by overlapping the real space. For example, the display control unit 28 can display a line surrounding the area 38, set the area 38 to be semi-transparent and display it, or mark the area 38 with a color to display it. The area 38 can be a three-dimensional area or a two-dimensional area. In addition, the area 38 can also be a shared area used by one of the users A and B in the past. At this time, the display control unit 28 can also display the names or user IDs of the said part of users. As another example, the area 38 may be a shared area set by a user who has hosted a service (eg, a meeting) in the past, among the user A and the user B. In this case, the display control unit 28 may display the name or user ID of the user.

[0097] When one of the multiple users participating in the service, a predetermined number of users or more of the multiple users, all of the multiple users, or a user with selection authority among the multiple users selects a shared area to be actually used from the candidates of the shared area, the selection unit 30 determines the selected candidate of the shared area as the shared area to be actually used. At this time, the display control unit 28 virtually displays the selected shared area on the screen of the display device of each user's AR glasses by overlapping the real space. For example, the display control unit 28 displays virtual objects shared by multiple users participating in the service in the shared area. Users participating in the service can browse the virtual objects in the shared area. Moreover, when the user specifies a virtual object to give an indication of display, the display control unit 28 displays the specified virtual object in the shared area. In addition, the user selects the shared area by gestures, voice, or operation of the user's terminal device.

[0098] Furthermore, when there are multiple candidates for the shared area and they overlap in the real space, the display control unit 28 may switch and display the overlapping candidates at a predetermined time interval. Figure 7 and Figure 8 to explain. Figure 8 , user A and user B are shown in a three-dimensional space (eg, a real space or a virtual space). , user A and user B are shown in a real space. Figure 8 The area 40 shown is Figure 7 The area different from the area 38 shown is a shared area that was used by user A and user B in the past. That is, both area 38 and area 40 were used by user A and user B in the past. At this time, the display control unit 28 switches area 38 and area 40 at a predetermined time interval on the screens of the display device 10a of the AR glasses 10 and the display device 12a of the AR glasses 12, respectively, and overlaps them in the real space and virtually displays them. For example, the display control unit 28 displays area 38, and when a predetermined time has passed, area 38 is not displayed and area 40 is displayed. Furthermore, when a predetermined time has passed since the display of area 40, the display control unit 28 does not display area 40 and displays area 38. Thereafter, the display control unit 28 repeats the above processing. Moreover, when area 38 and area 40 overlap in the real space (for example, when area 38 and area 40 partially overlap), the display control unit 28 may switch the display of area 38 and area 40 at a predetermined time interval. The same is true when MR technology is used. When VR technology is used, candidates for shared areas are displayed in the virtual space.

[0099] An example of calculating the intersection area will be described below.

[0100] Fig. 9, user A and user B in the real space are shown. For example, the display control unit 28 virtually displays a button 42 for indicating the calculation of the intersection area on the screen of each of the display device 10a of the AR glasses 10 and the display device 12a of the AR glasses 12 by overlapping it in the real space. When user A or user B presses the button 42 by gesture, voice, or operation of their own terminal device as indicated by symbol 44, the mode of the information processing device 14 is changed to the intersection area calculation mode, and the information processing device 14 performs the function of calculating the intersection area.

[0101] In the intersection area calculation mode, the calculation unit 26 estimates the direction and posture of the user A in the real space based on the measurement results of various sensors provided in the AR glasses 10, and estimates the field of view of the user A in the real space based on the estimated results. The display control unit 28 virtually displays the estimated field of view superimposed on the real space on the display screen of the display device 10a of the AR glasses 10. For example, the display control unit 28 may display a line surrounding the field of view or display the field of view as semi-transparent. Fig.10 4 shows the estimated visual field 46 of user A. Here, as an example, a line surrounding the visual field 46 of user A is shown. The visual field 46 has, for example, a quadrangular pyramid shape. The above shape is only an example, and may be Fig.11 The field of view 48 shown in FIG. 1 has a cone shape. Moreover, the shape and size of the field of view can be predetermined or changed by the user's operation. The calculation unit 26 also estimates the field of view of user B, and the display control unit 28 virtually displays the estimated field of view on the display screen of the display device 12a of the AR glasses 12 by overlapping the real space. The same is true when using MR technology. When using VR technology, the field of view is displayed in the virtual space.

[0102] The calculation unit 26 calculates the intersection area where the field of view of user A and the field of view of user B intersect in the real space. The display control unit 28 uses the calculated intersection area as a candidate for the shared area, and virtually displays it on the display screens of the display device 10a and the display device 12a, overlapping the real space. The same is true when using MR technology. When using VR technology, the intersection area is displayed in the virtual space.

[0103] The intersection area calculation mode may be automatically terminated when an intersection area is calculated, or when a shared area to be actually used is selected from the candidates of shared areas, or the intersection area calculation mode may be terminated after a predetermined time has passed since the time when the shared area is selected and the use of a service (e.g., a meeting) is started. The intersection area may be calculated before the time has passed, and the calculated intersection area may be displayed as a candidate of the shared area.

[0104] When the calculation unit 26 calculates an intersection region having a size (for example, volume or surface area) equal to or larger than a predetermined threshold, the display control unit 28 displays the intersection region as a candidate for the shared region.

[0105] The calculation unit 26 may calculate the area formed by the intersection of the field of view of user A and the field of view of user B for more than a predetermined time as the intersection area, but not calculate the area formed by the intersection of the field of view of user A and the field of view of user B for less than the said time as the intersection area.

[0106] The following describes the process when no intersection region is formed.

[0107] Fig.12 , user A and user B are shown in a three-dimensional space (eg, real space or virtual space). Here, user A and user B are shown in the real space. Field of view 50 is the field of view of user A, and field of view 52 is the field of view of user B. Fig.12 In the example shown, the fields of view 50 and 52 do not intersect and do not form an intersection area. At this time, the display control unit 28 virtually displays information such as a guide that users A and B should direct their sight lines in order to form an intersection area in the real space.

[0108] Fig.13 . The guide unit 54 is virtually displayed on the display screens of the display device 10a and the display device 12a, superimposed on the real space. For example, the display control unit 28 calculates the position where the movement of the field of view 50 and the field of view 52 is the smallest based on the position of the field of view 50 of the user A and the position of the field of view 52 of the user B, and virtually displays the guide unit 54 at the calculated position. The position where the movement of the field of view 50 and the field of view 52 is the smallest is, for example, the position where the sum of the movement amount of the field of view 50 and the movement amount of the field of view 52 is the smallest. The position where the movement of the field of view 50 and the field of view 52 is the smallest may also be a position between the field of view 50 and the field of view 52 (for example, a middle position). As another example, the guide unit 54 may also be displayed at a position between the position of the user A and the position of the user B (for example, a middle position). As another example, the guide unit 54 may also be displayed at a position between the position of the user A and the position of the user B (for example, a middle position) and at a position where the movement of the field of view 50 and the field of view 52 is the smallest. Furthermore, the display control unit 28 may virtually display a message “Please direct your gaze toward the guidance unit 54 ” on the display screens of the display device 10 a and the display device 12 a by superimposing the message on the real space.

[0109] Here, refer to Fig.14 and Fig.15 An example of the position of the display guide will be described in detail. Fig.14This is a diagram viewed from above user A and user B (i.e., in the direction of the Z axis). Fig.15 This is a diagram viewed from the X-axis or Y-axis direction of user A or user B.

[0110] Fig.14 56 and 58 are shown in FIG. 56. Trajectory 56 is a trajectory of the line of sight of user A that can be formed by user A changing the direction. Trajectory 58 is a trajectory of the line of sight of user B that can be formed by user B changing the direction. Trajectory 56 and trajectory 58 are formed in a plane including the X-axis and the Y-axis (i.e., in a plane orthogonal to the Z-axis). The calculation unit 26 calculates trajectory 56 based on the position of user A in the real space, and calculates trajectory 58 based on the position of user B in the real space. Moreover, the calculation unit 26 calculates the position where the distance from each user A and user B is equal based on the respective positions of user A and user B. Line 60 is a line indicating the position where the distance from each user A and user B is equal.

[0111] Fig.15 The line 62 shown is a line indicating the average height of the eye height of user A and the eye height of user B. For example, the calculation unit 26 estimates the height of the AR glasses 10 as the eye height of user A and the height of the AR glasses 12 as the eye height of user B, and calculates the average height of these eye heights. The height is indicated by the line 62.

[0112] For example, the calculation unit 26 calculates the position on the line 60 and the line 62 and the position where the movement of the field of view 50 of the user A and the field of view 52 of the user B is minimized. That is, the calculation unit 26 calculates the position where the distance from each of the user A and the user B is equal, the average height of the height of the eyes of the user A and the user B, and the movement of the field of view 50 and the field of view 52 is minimized. The display control unit 28 displays the guide unit 54 at the above position.

[0113] Alternatively, the calculation unit 26 may calculate a position where the size (eg, volume or surface area) of the intersection region is maximized, and the display control unit 28 may display the guide 54 at the position.

[0114] When the MR technology is used, the guide unit 54 is virtually displayed in the real space, similarly to the case of the AR technology. When the VR technology is used, the guide unit is displayed in the virtual space.

[0115] Fig.16Another candidate for a shared area is represented in . Area 64 is an area within a predetermined distance range (for example, within 1 m) from the respective positions of user A and user B in the real space. The calculation unit 26 calculates area 64 based on the respective positions of user A and user B in the real space. Area 64 is an area that may be a candidate for a shared area, and the display control unit 28 uses area 64 as a candidate for a shared area and virtually displays it on the display screens of the display device 10a and the display device 12a, overlapping the real space. The same is true when using MR technology. When using VR technology, area 64 is displayed in the virtual space. In addition, the display control unit 28 may also display information indicating the reason why area 64 is displayed as a candidate for a shared area. For example, the display control unit 28 displays a message indicating that area 64 is an area within a predetermined range from the respective positions of user A and user B.

[0116] As another example, the display control unit 28 may also display the intersection area as a candidate for a shared area when an intersection area is formed where the fields of view of the plurality of users intersect within a predetermined distance from the respective positions of the plurality of users. For example, when an intersection area is formed where the fields of view of user A and user B intersect within a predetermined distance from the respective positions of user A and user B in the real space, the display control unit 28 uses the intersection area as a candidate for a shared area and virtually displays it on the display screens of the display device 10a and the display device 12a, overlapping the real space. The same is true when using MR technology. When using VR technology, the intersection area is displayed in a virtual space.

[0117] Moreover, it is also possible to Fig.16 As shown, the display control unit 28 uses area 66 that was previously used as a shared area and area 68, the intersection area of ​​users A and B, together with area 64 as candidates for shared areas, and virtually displays them overlapping in the real space on the display screens of the display device 10a and the display device 12a.

[0118] The conditions for displaying the candidates of the shared area may be set by the user, or the candidates of the shared area may be displayed according to the user's instruction. Fig.17 and Fig.18 To illustrate these processes. Fig.17 and Fig.18 , user A and user B are shown in a three-dimensional space (eg, a real space or a virtual space). Here, user A and user B are shown in a real space.

[0119] like Fig.17As shown, the display control unit 28 virtually displays an image 70 on the display screens of the display device 10a and the display device 12a, for the user to indicate whether to set the reduction conditions for the shared area, by overlapping the real space. In the image 70, a "Yes" button and a "No" button are displayed. When either user A or user B presses the "No" button, the display control unit 28 virtually displays all candidates registered in the shared area candidate list on the display screens of the display device 10a and the display device 12a, by overlapping the real space. The same is true when using MR technology. When using VR technology, the image 70 is displayed in a virtual space.

[0120] When either user A or user B presses the "Yes" button as indicated by symbol 72, Fig.18 As shown, the display control unit 28 virtually displays a setting field 74 for the user to set the narrowing-down conditions on the display screens of the display device 10 a and the display device 12 a by superimposing the setting field 74 on the real space. Fig.18 In the example, as the narrowing condition, the condition "shared area used in the past" and the condition "specific distance range from the user: 1 m" are displayed in the setting column 74. When either user A or user B specifies the narrowing condition as indicated by symbol 76, the display control unit 28 displays the candidates of the shared area that meet the specified narrowing condition on the display screens of the display device 10a and the display device 12a in a virtually overlapping manner in the real space. The same is true when using MR technology. When using VR technology, the setting column 74 is displayed in the virtual space.

[0121] For example, when the condition "shared area used in the past" is specified, the display control unit 28 displays the shared area used in the past by user A and user B as a candidate for the shared area. Furthermore, when the condition "specific distance range from user: 1 m" is specified, the display control unit 28 displays candidates for the shared area within 1 m from the respective positions of user A and user B. Furthermore, when the narrowing-down condition displayed in the setting bar 74 is not specified, the display control unit 28 may display an intersection area as a candidate for the shared area. For example, when a predetermined time has passed since the time point when the setting bar 74 was displayed, the display control unit 28 displays the intersection area as a candidate for the shared area.

[0122] When the user selects a shared area to be actually used from the candidates of the shared area, the selection unit 30 sets the candidate of the shared area selected by the user as a shared area shared by each user participating in a service (eg, a meeting). Each user participating in the service can browse virtual objects in the shared area.

[0123] Reference Fig.19 and Fig. 20 , a specific example is given to illustrate the operation of selecting the shared area to be actually used. Fig.19 FIG. 4 shows user A, user B and candidates for a shared area in a three-dimensional space (eg, a real space or a virtual space). Fig. 20 , user A and user B in a three-dimensional space and a shared area are shown. Here, user A and user B in a real space are shown.

[0124] Fig.19 The areas 64, 66, and 68 shown are candidates for shared areas. When either user A or user B selects area 68 as a shared area to be actually used by gesture, voice, or operation of a terminal device, as indicated by symbol 78, the selection unit 30 sets the selected area 68 as a shared area shared by user A and user B. As a result, virtual objects in area 68 as a shared area are shared by user A and user B. That is, virtual objects in area 68 are virtually displayed by overlapping the real space on the display screens of the display device 10a and the display device 12a. User A and user B can view or edit the virtual objects by gesture, voice, or operation of a terminal device.

[0125] Furthermore, the shape of the shared area and the shapes of the candidates of the shared area can also be changed by user operation. For example, the display control unit 28 changes the shapes of the shared area or candidates to be changed by gestures, voice, or operation of the terminal device.

[0126] The display control unit 28 may change the size of the sharing area according to the virtual object displayed in the sharing area. Fig. 20 In the example shown, the display control unit 28 may also change the size of the area 68 according to the virtual object displayed in the area 68. For example, the more pages of the document as the virtual object, the larger the area 68. Also, the more images as the virtual object, the larger the area 68. Also, the display control unit 28 may change the size of the area 68 according to the display size of the virtual object. For example, the larger the display size of the virtual object, the larger the area 68. The display control unit 28 may also display the guide in a manner that forms a shared area having a size corresponding to the virtual object, and may also display a guide that allows the user to move to a position where the shared area as described above is formed.

[0127] After setting a shared area shared by multiple users, an area shared by a part of the multiple users, namely an individual shared area, can also be set. An individual shared area is an area shared only by the part of users, and is not shared by users other than the part of users. For example, the display control unit 28 displays virtually on the display screen of the display device of the AR glasses of each user an image representing each user participating in the service provided by the information processing system (such as a meeting) by overlapping the real space. The image representing the user is a photo of the user, or an image schematically representing the user (such as an icon or an avatar, etc.).

[0128] Fig.21 An image representing a user is shown in . Fig.21 It is a diagram showing an image that is virtually displayed superimposed on the real space. Here, it is assumed that user A, user B, and user C log in to the information processing system and participate in the services (such as a meeting) provided by the information processing system. Image 80 is an image schematically showing user A, image 82 is an image schematically showing user B, and image 84 is an image schematically showing user C. Image 80, image 82, and image 84 are virtually displayed superimposed on the real space on the display screen of the display device of the AR glasses of user A, user B, and user C respectively. The same is true when using MR technology. When using VR technology, the image of each user is displayed in a virtual space.

[0129] For example, when an image representing the user is specified by a user who has joined the service, the mode of the information processing system is changed to a mode for setting an individual sharing area, that is, an individual sharing area setting mode. In the individual sharing area setting mode, a function of setting an individual sharing area to be shared by the user who specified the image and the user represented by the specified image is executed. For example, by executing Figure 6 The display control unit 28 virtually displays the candidate of the individual shared area by overlapping it in the real space on the display screen of the display device of the AR glasses of the user who designated the image and on the display screen of the display device of the AR glasses of the designated user.

[0130] For example, assume that user B specifies image 80 of user A as indicated by symbol 86. Image 80 is specified by gesture, voice, or operation of user B's terminal device. At this time, information processing device 14 executes according to Figure 6The process of the flowchart shown in the figure determines the candidate of the individual shared area to be shared by the user B who designated the image 80 and the designated user A. The display control unit 28 virtually displays the candidate of the individual shared area by superimposing it on the real space on the display screens of the display devices 10a and 12a.

[0131] When either user A or user B selects an individual sharing area to be actually used from the candidates of the individual sharing area, the selected individual sharing area is shared by user A and user B. The individual sharing area is shared only by user A and user B, and is not shared by user C. Therefore, the virtual objects in the individual sharing area are only viewed by user A and user B, and user C cannot view the virtual objects in the individual sharing area.

[0132] Furthermore, the display control unit 28 may display the virtual object in the shared area in the individual shared area according to the operation of the user who shares the individual shared area. Fig. 22 The processing will be described below. Fig. 22 , user A, user B and a shared area in the real space are shown. Here, it is assumed that area 66 and area 68 are shared areas shared by user A, user B, and user C, and area 64 is an individual shared area shared by user A and user B. That is, user A and user B can browse virtual objects in area 64, which is an individual shared area, but user C cannot browse virtual objects in area 64. Furthermore, user A, user B, and user C can browse virtual objects in area 66 and area 68.

[0133] For example, when user A moves the virtual object α displayed in the area 68 as the shared area to the area 64 as the individual shared area by gesture, voice, or operation of the terminal device of user A as indicated by symbol 88, the display control unit 28 displays the virtual object α in the area 64. The display control unit 28 may not display the moved virtual object α in the area 68 as the moving source, but may display it in the area 68. Thus, in the area 64 as the moving destination, the virtual object α is shared only by user A and user B, and is not shared by user C. Furthermore, the virtual object displayed in the area 64 may be displayed in the area 68 as the shared area according to the operation of the user who shares the individual shared area. In addition, when the user wants to move the virtual object α displayed in the area 68 as the shared area, the mode of the information processing system is changed to the setting mode of the individual shared area, and the user who wants to share the virtual object α individually is selected to set the individual shared area.

[0134] In the above-mentioned embodiment, the shared area or candidate is virtually displayed in the real space by using AR technology or MR technology. As another example, the shared area or candidate can also be displayed in the virtual space by using VR technology. The virtual space is a three-dimensional virtual space compared with the real space. For example, the coordinate system in the real space corresponds to the coordinate system in the virtual space one by one.

[0135] For example, a glasses-type terminal device using VR technology, i.e., VR glasses, or an HMD using VR technology, etc. is used. The VR glasses or HMD are worn by the user, and a shared area or virtual objects in a virtual space are displayed on a display screen of the VR glasses or HMD.

[0136] When VR technology is used, candidates for the shared area are determined in the same manner as in the above-described embodiment.

[0137] Here, it is assumed that user A and user B log in to the information processing system and participate in a service (eg, a conference) provided by the information processing system.

[0138] First, the processor 24 sets the respective positions of user A and user B in the virtual space. For example, the initial position of user A in the virtual space is predetermined, and the processor 24 sets the initial position to the position of user A when user A logs into the information processing system. The same is true for user B. As another example, user A may also specify the position of user A in the virtual space by gestures, voice, or operation of user A's terminal device. In this case, the processor 24 sets the position specified by user A as the position of user A. The same is true for user B. As another advantage, the processor 24 may also set the position of user A in the virtual space based on the position of user A in the real space. For example, the coordinate system of the real space corresponds one-to-one to the coordinate system of the virtual space, and the processor 24 sets the position of user A in the real space to the position of user A in the virtual space. The same is true for user B.

[0139] When the positions of user A and user B in the virtual space are set, the display control unit 28 displays candidates for the shared area in the virtual space. For example, similarly to the above embodiment, the shared area used by user A and user B in the past is displayed as a candidate for the shared area.

[0140] Furthermore, the calculation unit 26 may also calculate the intersection area where the field of view of user A and the field of view of user B in the virtual space intersect, and the display control unit 28 may display the intersection area in the virtual space as a candidate for a shared area. The direction of the field of view of user A in the virtual space corresponds to the direction of the VR glasses or HMD worn by user A, and the calculation unit 26 calculates the direction of the field of view of user A in the virtual space based on the direction of the VR glasses or HMD. The same is true for the field of view of user B. Furthermore, similarly to the above embodiment, the display control unit 28 may display a guide in the virtual space to form an intersection area in the virtual space. At this time, the display control unit 28 may also display the guide at a position in the virtual space where the movement of the fields of view of user A and user B is minimized.

[0141] Moreover, similar to the case of using AR technology or MR technology, the display control unit 28 can also display the intersection area as a candidate for the shared area in the virtual space when an intersection area is formed within a predetermined distance from the respective positions of user A and user B in the virtual space.

[0142] Furthermore, similarly to the case of using AR technology or MR technology, an individual shared area shared by some of the plurality of users participating in the service may be set in the virtual space.

[0143] In addition, similarly to the case of using AR technology or MR technology, the shape of the shared area or the candidate may be changed in the virtual space, or the virtual object may be moved between shared areas.

[0144] In each of the above embodiments, the so-called processor refers to a processor in a broad sense, including a general-purpose processor (such as a central processing unit (CPU), etc.) or a special-purpose processor (such as a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, etc.). Moreover, the actions of the processor in each of the above embodiments may be performed not only by one processor, but also by a plurality of processors located in physically separate positions. Moreover, the order of the actions of the processor is not limited to the order described in each of the above embodiments, and may also be appropriately changed.

Claims

1. An information processing device comprising a processor, The processor sets an area where the fields of vision of the plurality of users intersect in the three-dimensional space where the plurality of users are located, namely, an intersection area, as an area shared by the plurality of users, namely, a shared area, Furthermore, when the intersection area is not formed, the processor displays a guide portion indicating the direction in which the plurality of users should direct their sight lines in order to form the intersection area, at a position where the movement of the fields of vision of the plurality of users is minimal.

2. The information processing device according to claim 1, characterized in that Furthermore, the processor displays the intersection area as a candidate of the shared area in the three-dimensional space when the intersection area is formed within a predetermined distance from the positions of each of the plurality of users.

3. The information processing device according to claim 1, characterized in that The processor further displays, in the three-dimensional space, an area that was previously shared by the plurality of users as a candidate for the shared area.

4. The information processing device according to claim 1, characterized in that Furthermore, when there are a plurality of candidates for the shared region and the candidates overlap in the three-dimensional space, the processor switches and displays the overlapping candidates at a predetermined time interval.

5. The information processing device according to any one of claims 1 to 4, characterized in that: The processor further sets an individual sharing area, which is an area shared by a part of the plurality of users in the three-dimensional space.

6. The information processing device according to claim 5, characterized in that: The processor further displays a virtual object displayed in the sharing area in a three-dimensional space in the individual sharing area according to an operation of a user who shares the individual sharing area.

7. The information processing device according to any one of claims 1 to 4, characterized in that: The processor further changes the size of the sharing area according to the virtual object displayed in the sharing area.

8. A computer-readable storage medium that enables a computer to set an area where the fields of vision of multiple users intersect, i.e., an intersection area, in a three-dimensional space where multiple users are located as an area shared by the multiple users, i.e., a shared area, and further, when the intersection area is not formed, a guide portion that the multiple users should direct their lines of sight in order to form the intersection area is displayed at a position where the fields of vision of the multiple users move the least.

Images