Virtual image sharing method and virtual image sharing system

Abstract

A virtual image sharing system 100 displays a first virtual image 30, which is positioned in a space and is operable by a first user U1 using one wearable terminal device (first display device) 20 and a second user U2 using another wearable terminal device (second display device) 20, on each of the one wearable terminal device 20 and the other wearable terminal device 20. Further, the virtual image sharing system 100 displays, on the one wearable terminal device 20, a second virtual image 40 that is positioned in the space, is operable by the first user U1, and is not displayed on the other wearable terminal device 20.

Description

[Technical Field]

[0001] The present disclosure relates to a virtual image sharing method and system. [Background technology]

[0002] Conventionally, virtual reality (VR), mixed reality (MR), and augmented reality (AR) are known technologies that allow a user to experience virtual images and / or virtual spaces using a wearable terminal device worn on the user's head. A wearable terminal device has a display unit that covers the user's field of vision when worn by the user. By displaying virtual images and / or virtual spaces on this display unit according to the user's position and orientation, a visual effect is achieved that makes it seem as if the virtual images and / or virtual spaces are actually present (for example, U.S. Patent Application Publication Nos. 2019 / 0087021 and 2019 / 0340822).

[0003] MR is a technology that allows a user to experience mixed reality, in which real space and virtual images are fused together, by displaying a virtual image that appears to exist at a predetermined position in real space while allowing the user to view the real space. For example, Patent Document 1 discloses a technology that allows multiple users wearing see-through head-mounted displays to share a virtual object (e.g., a virtual object of a building) displayed on the see-through head-mounted display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2013 / 0293468 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology disclosed in Patent Document 1, when an operation (for example, a rotation operation or a zoom operation) is performed on a virtual object, the way the virtual object is viewed by each user changes, and if the operation is performed carelessly, the virtual object becomes difficult to see. Furthermore, with the technology described above, it is assumed that some users may want to operate a virtual object without being seen by other users.

[0006] The virtual image sharing method and virtual image sharing system of the present disclosure have been made in consideration of the above-mentioned problems, and aim to improve the usability of virtual images shared among multiple users. [Means for solving the problem]

[0007] In order to solve the above problem, a virtual image sharing method according to the present disclosure includes: At least a first display device and a second display device A virtual image sharing method for sharing a virtual image between devices, comprising: virtuality displaying, on each of the first display device and the second display device, a first virtual image located in a space and operable by a first user of the first display device and a second user of the second display device; displaying a first avatar in the virtual space, the first avatar being displayable on each of the first display device and the second display device and whose display changes in response to an action of the first user; The aforementioned virtuality Located in space, When the first user performs an operation on a second virtual image that is not displayed on the second display device, the action corresponding to the operation is not reflected in the display of the first avatar. .

[0008] In order to solve the above problem, another virtual image sharing method according to the present disclosure includes: A virtual image sharing method for sharing a virtual image among a plurality of display devices, comprising: displaying, on each of the plurality of display devices, a virtual image located within the space and operable by a user of each of the plurality of display devices; a first mode in which, when a user of one of the plurality of display devices performs an operation to change a display mode of the virtual image, the display mode of the virtual image changed based on the operation is reflected in the display of the virtual image by the one display device and is also reflected in the display of the virtual image by the other display devices of the plurality of display devices; a second mode in which, when a user of one of the plurality of display devices performs an operation to change a display mode of the virtual image, the display mode of the virtual image changed based on the operation is reflected in the display of the virtual image by the one display device, but is not reflected in the display of the virtual image by the other display devices of the plurality of display devices; Equipped with.

[0009] In order to solve the above problem, a virtual image sharing system according to the present disclosure includes: At least a first display device and a second display device A virtual image sharing system for sharing virtual images between devices, comprising: virtuality displaying, on each of the first display device and the second display device, a first virtual image located in a space and operable by a first user of the first display device and a second user of the second display device; displaying a first avatar in the virtual space, the first avatar being displayable on each of the first display device and the second display device and whose display changes in response to an action of the first user; The aforementioned virtuality Located in space, When the first user performs an operation on a second virtual image that is not displayed on the second display device, the action corresponding to the operation is not reflected in the display of the first avatar. .

[0010] In order to solve the above problem, another virtual image sharing system according to the present disclosure includes: A virtual image sharing system for sharing a virtual image among a plurality of display devices, displaying, on each of the plurality of display devices, a virtual image located within the space and operable by a user of each of the plurality of display devices; a first mode in which, when a user of one of the plurality of display devices performs an operation to change a display mode of the virtual image, the display mode of the virtual image changed based on the operation is reflected in the display of the virtual image by the one display device and is also reflected in the display of the virtual image by the other display devices of the plurality of display devices; a second mode in which, when a user of one of the plurality of display devices performs an operation to change a display mode of the virtual image, the display mode of the virtual image changed based on the operation is reflected in the display of the virtual image by the one display device, but is not reflected in the display of the virtual image by the other display devices of the plurality of display devices; Equipped with. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to improve the usability of virtual images shared among multiple users. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of a virtual image sharing system. [Figure 2] FIG. 2 is a block diagram showing a functional configuration of the information processing device. [Figure 3] FIG. 1 is a schematic perspective view showing the external configuration of a wearable terminal device. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of the wearable terminal device. [Figure 5] 10 is a flowchart showing a control procedure of a virtual image display control process. [Figure 6] 10 is a flowchart showing a control procedure for a first reflected display process. [Figure 7] 10 is a flowchart showing a control procedure for a second reflected display process. [Figure 8] 10A and 10B are diagrams showing an example of a visual recognition area and a first virtual image visually recognized by a first user wearing a wearable terminal device. [Figure 9]10A and 10B are diagrams showing examples of a visual recognition area visually recognized by a first user wearing a wearable terminal device, as well as a first virtual image and a second virtual image. [Figure 10] 10A and 10B are diagrams illustrating modified examples of the display mode of the first virtual image. [Figure 11] 10A and 10B are diagrams illustrating modified examples of the display mode of the first virtual image. [Figure 12] 10A and 10B are diagrams showing examples of a visual recognition area visually recognized by a first user wearing a wearable terminal device, as well as a first virtual image and a second virtual image. [Figure 13] 10A and 10B are diagrams showing examples of a visual recognition area visually recognized by a first user wearing a wearable terminal device, as well as a first virtual image and a second virtual image. [Figure 14] 10A and 10B are diagrams illustrating modified examples of the display mode of the second virtual image. [Figure 15] 10A and 10B are diagrams illustrating modified examples of the display mode of the second virtual image. [Figure 16] 10A and 10B are diagrams showing examples of a visual recognition area visually recognized by a first user wearing a wearable terminal device, as well as a first virtual image and a second virtual image. [Figure 17] 10A and 10B are diagrams showing examples of a visual recognition area visually recognized by a first user wearing a wearable terminal device, as well as a first virtual image and a second virtual image. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described with reference to the drawings. However, for the sake of convenience, the drawings referred to below show only the main components necessary for explaining the embodiments in a simplified form.

[0014] <Configuration of virtual image sharing system 100> First, the virtual image sharing system 100 will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic configuration of the virtual image sharing system 100.

[0015] As shown in FIG. 1, the virtual image sharing system 100 includes an information processing device 10 and a plurality of (e.g., two) wearable terminal devices (display devices) 20 communicatively connected to the information processing device 10.

[0016] The information processing device 10 is a server device that performs display control of virtual images displayed on each wearable terminal device 20, etc.

[0017] The wearable terminal device 20 is a head mounted display (HMD) worn on the head of a user. Specifically, the wearable terminal device 20 is so-called MR / AR goggles that provide MR or AR to the user.

[0018] <Configuration of information processing device 10> Next, the configuration of the information processing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the information processing device 10.

[0019] 2, the information processing device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a storage unit 13, a communication unit 14, and a bus 15. The components of the information processing device 10 are connected to each other via the bus 15.

[0020] The CPU 11 is a processor that performs various arithmetic processing and controls the overall operation of each unit of the information processing device 10. The CPU 11 performs various control operations by reading and executing a program 131 stored in the storage unit 13. Although a single CPU 11 is illustrated in FIG. 2, this is not limiting. Two or more processors such as CPUs may be provided, and the processing performed by the CPU 11 of this embodiment may be shared and executed by these two or more processors.

[0021] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0022] The storage unit 13 is a non-transitory recording medium readable by the CPU 11. The storage unit 13 stores a program 131 executed by the CPU 11, various setting data, and the like. The program 131 is stored in the storage unit 13 in the form of computer-readable program code. The storage unit 13 may be, for example, a non-volatile storage device such as an SSD (Solid State Drive) equipped with a flash memory or an HDD (Hard Disk Drive).

[0023] The data stored in the storage unit 13 includes virtual image data relating to a virtual image, etc. The virtual image data includes data relating to the display content of the virtual image, data on the display position, data on the orientation, etc.

[0024] The communication unit 14 communicates with and transmits data to and from each wearable terminal device 20. For example, the communication unit 14 receives data including some or all of the detection results by the sensor unit 25 of each wearable terminal device 20, and information related to user operations detected by each wearable terminal device 20. The communication unit 14 may also be capable of communicating with devices other than the wearable terminal device 20.

[0025] <Configuration of the wearable terminal device 20> Next, the external configuration of the wearable terminal device 20 will be described with reference to Fig. 3. Fig. 3 is a schematic perspective view showing the external configuration of the wearable terminal device 20.

[0026] As shown in FIG. 3, the wearable terminal device 20 includes a main body 20a, a visor 241 (display member) attached to the main body 20a, and the like.

[0027] The main body 20a is an annular member whose circumference is adjustable. Various devices such as a depth sensor 253 and a camera 254 are built into the main body 20a. When the main body 20a is worn on the head, the user's field of vision is covered by a visor 241.

[0028] The visor 241 is optically transparent. The user can view the real space through the visor 241. An image such as a virtual image is projected and displayed on a display surface of the visor 241 facing the user's eyes from a laser scanner 242 (see FIG. 4) built into the main body 20a. The user views the virtual image through light reflected from the display surface. At this time, the user also views the real space through the visor 241, providing a visual effect as if the virtual image were present in the real space.

[0029] Next, the functional configuration of the wearable terminal device 20 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configuration of the wearable terminal device 20.

[0030] As shown in Fig. 4, the wearable terminal device 20 includes a CPU 21, a RAM 22, a storage unit 23, a display unit 24, a sensor unit 25, and a communication unit 26, and these units are connected via a bus 27. Of the components shown in Fig. 4, all units except for the visor 241 of the display unit 24 are built into the main body unit 20a and operate using power supplied from a battery also built into the main body unit 20a.

[0031] The CPU 21 is a processor that performs various calculation processes and controls the overall operation of each part of the wearable terminal device 20. The CPU 21 performs various control operations by reading and executing a program 231 stored in the storage unit 23. The CPU 21 executes, for example, a visual recognition area detection process by executing the program 231. The visual recognition area detection process is a process that detects the user's visual recognition area within a space.

[0032] 4 shows a single CPU 21, but is not limited to this. Two or more processors such as CPUs may be provided, and the processing executed by the CPU 21 of the present embodiment may be shared and executed by these two or more processors.

[0033] The RAM 22 provides a working memory space for the CPU 21 and stores temporary data.

[0034] The storage unit 23 is a non-transitory recording medium readable by the CPU 21 as a computer. The storage unit 23 stores a program 231 executed by the CPU 21, various setting data, and the like. The program 231 is stored in the storage unit 23 in the form of computer-readable program code. The storage unit 23 may be, for example, a non-volatile storage device such as an SSD equipped with a flash memory.

[0035] The display unit 24 includes a visor 241, a laser scanner 242, and an optical system that guides light output from the laser scanner 242 onto the display surface of the visor 241. The laser scanner 242 irradiates the optical system with pulsed laser light, which is on / off controlled for each pixel, while scanning in a predetermined direction in accordance with a control signal from the CPU 21. The laser light incident on the optical system forms a display screen consisting of a two-dimensional pixel matrix on the display surface of the visor 241. The type of the laser scanner 242 is not particularly limited, but for example, a type in which a mirror is operated by a MEMS (microelectromechanical system) to scan the laser light can be used. The laser scanner 242 includes three light-emitting elements that emit, for example, RGB laser light. The display unit 24 can display in color by projecting light from these light-emitting elements onto the visor 241.

[0036] The sensor unit 25 includes an acceleration sensor 251, an angular velocity sensor 252, a depth sensor 253, a camera 254, and an eye tracker 255. The sensor unit 25 may further include sensors not shown in FIG.

[0037] The acceleration sensor 251 detects acceleration and outputs the detection result to the CPU 21. From the detection result by the acceleration sensor 251, translational movement of the wearable terminal device 20 in three orthogonal axial directions can be detected.

[0038] The angular velocity sensor 252 (gyro sensor) detects the angular velocity and outputs the detection result to the CPU 21. From the detection result by the angular velocity sensor 252, the rotational movement of the wearable terminal device 20 can be detected.

[0039] The depth sensor 253 is an infrared camera that detects the distance to a subject using a ToF (Time of Flight) method, and outputs the distance detection result to the CPU 21. The depth sensor 253 is provided on the front surface of the main body 20a so that it can capture an image of the visible area. By repeatedly performing measurements by the depth sensor 253 every time the position and orientation of the user change in space and combining the results, it is possible to perform three-dimensional mapping of the entire space (i.e., to obtain the three-dimensional structure).

[0040] The camera 254 captures an image of the space using a group of RGB imaging elements, acquires color image data as the captured image, and outputs the acquired color image data to the CPU 21. The camera 254 is provided on the front surface of the main body 20a so as to capture an image of the visible area. The image output from the camera 254 is used to detect the position and orientation of the wearable terminal device 20, and is also transmitted from the communication unit 26 to an external device and used to display the visible area of ​​the user of the wearable terminal device 20 on the external device.

[0041] Eye tracker 255 detects the user's line of sight and outputs the detection result to CPU 21. The method for detecting the line of sight is not particularly limited, but for example, a method can be used in which an eye tracking camera captures an image of the reflection point of near-infrared light on the user's eye, and the image captured by camera 254 is analyzed to identify the object the user is looking at. A part of the configuration of eye tracker 255 may be provided on the periphery of visor 241, etc.

[0042] The communication unit 26 is a communication module that includes an antenna, a modulation / demodulation circuit, a signal processing circuit, etc. The communication unit 26 transmits and receives data to and from external devices via wireless communication in accordance with a predetermined communication protocol.

[0043] In the wearable terminal device 20 configured as above, the CPU 21 performs the following control operations.

[0044] The CPU 21 performs three-dimensional mapping of the space based on distance data to the subject input from the depth sensor 253. The CPU 21 repeatedly performs this three-dimensional mapping every time the user's position and orientation change, updating the results each time. The CPU 21 also performs three-dimensional mapping for each continuous space. Therefore, when the user moves between multiple rooms separated by walls or the like, the CPU 21 recognizes each room as a single space and performs three-dimensional mapping for each room separately.

[0045] The CPU 21 detects the user's visual recognition area in space. Specifically, the CPU 21 identifies the position and orientation of the user (wearable terminal device 20) in space based on the detection results of the acceleration sensor 251, angular velocity sensor 252, depth sensor 253, camera 254, and eye tracker 255, and the accumulated results of 3D mapping. Then, the CPU 21 detects (identifies) the visual recognition area based on the identified position and orientation and the predetermined shape of the visual recognition area. The CPU 21 also continuously detects the user's position and orientation in real time, and updates the visual recognition area in conjunction with changes in the user's position and orientation. Note that the detection of the visual recognition area may be performed using the detection results of some of the acceleration sensor 251, angular velocity sensor 252, depth sensor 253, camera 254, and eye tracker 255.

[0046] <Operation of the virtual image sharing system 100> Next, a control procedure of virtual image display control processing executed as part of the operation of the virtual image sharing system 100 will be described with reference to the flowchart of Fig. 5. The virtual image display control processing is executed by the CPU 11 of the information processing device 10.

[0047] Here, when executing the virtual image display control process, it is assumed that a first virtual image 30 (for example, a cylindrical first virtual image 30; see FIG. 8 ) is generated in advance with its display position and orientation determined in a certain real space (for example, a conference room). This first virtual image 30 is an image that can be operated by a user wearing each wearable terminal device 20.

[0048] FIG. 5 is a flowchart showing a control procedure of the virtual image display control process. 5, when the virtual image display control process is started, first, the CPU 11 of the information processing device 10 acquires information related to the visual recognition area of ​​each user wearing each wearable terminal device 20 from each wearable terminal device 20 via the communication unit 14 (step S101). Here, of the two wearable terminal devices 20 constituting the virtual image sharing system 100, a user wearing one of the wearable terminal devices (first display device) 20 is referred to as a first user U1, and a user wearing the other of the wearable terminal devices (second display device) 20 is referred to as a second user U2.

[0049] Next, based on the information relating to the visual recognition area of ​​each user acquired in step S101, CPU 11 determines whether or not there is a user whose first virtual image 30 is present within the visual recognition area (step S102).

[0050] In step S102, if it is determined that there is a user for whom the first virtual image 30 is present within the visible area (step S102; YES), the CPU 11 displays the first virtual image 30 on the wearable terminal device 20 (visor 241) of the user (step S103).

[0051] FIG. 8 is a diagram showing an example of a visual recognition area and a first virtual image 30 visually recognized by a first user U1 wearing the wearable terminal device 20. As shown in FIG. As shown in FIG. 8, when the first virtual image 30 is displayed, the first user U1 views the first virtual image 30 facing a predetermined direction at a predetermined position on a table T in a space (for example, a conference room). The first user U1 also views a second user U2 wearing a wearable terminal device 20 on the opposite side (opposite side) of the table T. This space is the real space viewed by the first user U1 through the visor 241. Because the first virtual image 30 is projected onto the optically transparent visor 241, it is viewed as a semi-transparent image overlapping the real space. In FIG. 8, the viewing area viewed by the first user U1 is indicated by a dashed line.

[0052] Returning to the explanation of the control procedure of the virtual image display control processing in Figure 5, if it is determined in step S102 that there is no user with the first virtual image 30 present within the visible area (step S102; NO), the CPU 11 skips step S103 and proceeds to step S104.

[0053] Next, the CPU 11 determines whether information instructing the display of the second virtual image 40 has been acquired from the wearable terminal device 20 via the communication unit 14 (step S104). Here, the second virtual image 40 is a virtual image obtained by duplicating (including a reduced copy or an enlarged copy) the first virtual image 30 displayed in the space. The second virtual image 40 is displayed only on the wearable terminal device 20 (visor 241) worn by the user who performed the operation instructing the display of the second virtual image 40. In other words, the second virtual image 40 displayed on the wearable terminal device 20 can be operated only by the user wearing the wearable terminal device 20. Examples of methods for operating the second virtual image 40 include a gesture operation performed by the wearable terminal device 20 detecting the movement of the user's hand, and an operation using a controller (not shown) attached to the wearable terminal device (the same applies to the first virtual image 30).

[0054] If it is determined in step S104 that information instructing the display of the second virtual image 40 has been acquired from the wearable terminal device 20 (step S104; YES), the CPU 11 displays the second virtual image 40 on the wearable terminal device 20 (visor 241) worn by the user (instructing user) who instructed the display of the second virtual image 40 (step S105). For example, if the user who instructed the display of the second virtual image 40 is a first user U1, the second virtual image 40 is displayed on the wearable terminal device 20 (visor 241) worn by the first user U1, but the second virtual image 40 is not displayed on the wearable terminal device 20 (visor 241) worn by a second user U2. On the other hand, when the second user U2 gives an instruction to display the second virtual image 40, the second virtual image 40 is displayed on the wearable terminal device 20 (visor 241) worn by the second user U2, but the second virtual image 40 is not displayed on the wearable terminal device 20 (visor 241) worn by the first user U1. In other words, when the first user U1 and the second user U2 each give an instruction to display the second virtual image 40, a dedicated second virtual image 40 is displayed on the wearable terminal device 20 (visor 241) worn by each user. As described above, according to the virtual image sharing system 100 of the present embodiment, the second virtual image 40, which is a copy of the first virtual image 30, can be displayed on each wearable terminal device 20, and the users wearing the wearable terminal devices 20 can simulate the operation of the first virtual image 30 using the second virtual image 40 displayed on the wearable terminal device 20 they are wearing. As a result, it is possible to prevent inadvertent operations on the first virtual image 30, thereby eliminating the problem of such operations making it difficult to see the first virtual image 30. Furthermore, each user can freely simulate operations on the first virtual image 30 without being seen by or disturbed by other users, thereby improving the usability of the first virtual image 30 shared among users.

[0055] FIG. 9 is a diagram showing an example of a visual recognition area visually recognized by a first user U1 wearing the wearable terminal device 20, as well as a first virtual image 30 and a second virtual image 40. In FIG. As shown in FIG. 9 , when the first user U1 issues an instruction to display the second virtual image 40, the first user U1 views the first virtual image 30 displayed on the table T, and also views the second virtual image 40, which is located at a predetermined position in front of the first virtual image 30 and faces the same direction as the first virtual image 30. Like the first virtual image 30, the second virtual image 40 is projected onto the optically transparent visor 241 and is therefore viewed as a translucent image superimposed on real space. Note that in the example of FIG. 9 , when the second virtual image 40 is displayed, the size of the first virtual image 30 is reduced by a predetermined magnification and displayed; however, the second virtual image 40 may be displayed at the same magnification as the first virtual image 30 or enlarged by a predetermined magnification.

[0056] Returning to the explanation of the control procedure of the virtual image display control processing in Figure 5, if it is determined in step S104 that information instructing the display of the second virtual image 40 has not been acquired from the wearable terminal device 20 (step S104; NO), the CPU 11 skips step S105 and proceeds to step S106.

[0057] Next, the CPU 11 determines whether or not information instructing a change in the display mode of the first virtual image 30 has been acquired from the wearable terminal device 20 via the communication unit 14 (step S106).

[0058] If it is determined in step S106 that information instructing a change in the display mode of the first virtual image 30 has been acquired from the wearable terminal device 20 (step S106; YES), the CPU 11 changes the display mode of the first virtual image 30 based on the information (step S107). Specifically, if the information instructing a change in the display mode of the first virtual image 30 acquired from the wearable terminal device 20 is, for example, information instructing a change in the shape of the first virtual image 30 to a rectangular parallelepiped, the CPU 11 changes the shape of the first virtual image 30 to a rectangular parallelepiped based on the information, as shown in FIG. 10. Furthermore, if the information acquired from the wearable terminal device 20 is, for example, information instructing a change in the shape of the first virtual image 30 to a rectangular parallelepiped and then to a triangular prism, the CPU 11 changes the shape of the first virtual image 30 to a rectangular parallelepiped and then to a triangular prism based on the information, as shown in FIG. 11.

[0059] FIG. 12 is a diagram showing an example of a visual recognition area visually recognized by a first user U1 wearing the wearable terminal device 20, as well as a first virtual image 30 and a second virtual image 40. In FIG. 12, for example, in a state where the shape of the first virtual image 30 is changed to a rectangular parallelepiped shape in response to an operation by the second user U2, the first user U1 views the first virtual image 30 changed to a rectangular parallelepiped shape, and also views a second virtual image 40 (cylindrical second virtual image 40) at a predetermined position in front of the first virtual image 30. Although not shown in the drawings, at this time, the second user U2 wearing the wearable terminal device 20 also views the first virtual image 30 changed to a rectangular parallelepiped shape.

[0060] Returning to the description of the control procedure of the virtual image display control process in FIG. 5, after executing the process of step S107, the CPU 11 executes the first reflected display process (step S108).

[0061] FIG. 6 is a flowchart showing the control procedure of the first reflection display process. As shown in Figure 6, when the first reflection display process is started, first, the CPU 11 of the information processing device 10 determines whether or not instruction information for reflecting the display mode of the first virtual image 30 in the second virtual image 40 has been acquired from the wearable terminal device 20 via the communication unit 14 (step S121).

[0062] In step S121, if it is determined that instruction information for reflecting the display mode of the first virtual image 30 in the second virtual image 40 has not been acquired from the wearable terminal device 20 (step S121; NO), the CPU 11 returns the processing to the virtual image display control processing (see Figure 5) and performs processing from step S109 onwards.

[0063] Also, in step S121, if it is determined that instruction information for reflecting the display mode of the first virtual image 30 in the second virtual image 40 has been acquired from the wearable terminal device 20 (step S121; YES), the CPU 11 determines whether there were multiple change patterns for the display mode of the first virtual image 30 changed in step S107 of the virtual image display control process (see Figure 5) (step S122).

[0064] If it is determined in step S122 that there are multiple change patterns for the display mode of the first virtual image 30 (step S122; YES), the CPU 11 determines whether or not selection information for a change pattern has been acquired from the wearable terminal device 20 worn by the instructing user (the user who has instructed to reflect the display mode of the first virtual image 30 in the second virtual image 40) via the communication unit 14 (step S123). Note that in a case where a desired change pattern is selected from multiple change patterns, if the change patterns include a change pattern in which a first additional image (not shown) is added to the first virtual image 30 in a certain display mode and a change pattern in which a second additional image (not shown) is added to the first virtual image 30 in the certain display mode, a change pattern in which both the first additional image and the second additional image are added to the first virtual image 30 in the certain display mode may be further added.

[0065] In step S123, if it is determined that the selection information of the change pattern has not been acquired (step S123; NO), the CPU 11 repeats the determination process of step S123 until the selection information is acquired.

[0066] Furthermore, when it is determined in step S123 that selection information for a change pattern has been acquired (step S123; YES), the CPU 11 reflects the selected change pattern based on the acquired selection information in the second virtual image 40 displayed on the wearable terminal device 20 worn by the instructing user (step S124). For example, when the shape of the first virtual image 30 is changed to a rectangular parallelepiped shape (first change pattern) as described above and then changed to a triangular prism shape (second change pattern) (see FIG. 11), and selection information selecting the rectangular parallelepiped shape (first change pattern) is acquired as selection information for the change pattern from the wearable terminal device 20 worn by the instructing user, the CPU 11 reflects the rectangular parallelepiped shape, which is the selected change pattern (first change pattern), in the second virtual image 40 displayed on the wearable terminal device 20 of the instructing user.

[0067] FIG. 13 is a diagram showing an example of a visual recognition area visually recognized by a first user U1 wearing the wearable terminal device 20, as well as a first virtual image 30 and a second virtual image 40. In FIG. As shown in Figure 13, when the display mode (rectangular parallelepiped shape) of the first virtual image 30 is reflected in the second virtual image 40 in response to the operation of the first user U1, the first user U1 views the first virtual image 30 in a rectangular parallelepiped shape and also views the second virtual image 40 in which the rectangular parallelepiped shape is reflected at a predetermined position in front of the first virtual image 30.

[0068] Returning to the explanation of the control procedure of the first reflected display process in FIG. 6, after executing the process of step S124, the CPU 11 determines whether or not information instructing the user to decide on the reflected display has been acquired from the wearable terminal device 20 worn by the instructing user (step S125).

[0069] In step S125, if it is determined that information instructing determination of reflected display has been acquired (step S125; YES), the CPU 11 returns the process to the virtual image display control process (see FIG. 5), and performs the processes from step S109 onwards.

[0070] Furthermore, in step S125, if it is determined that information instructing to determine the reflected display has not been acquired (step S125; NO), the CPU 11 returns the process to step S123 and repeats the subsequent processes.

[0071] Furthermore, if it is determined in step S122 that there are not multiple change patterns for the display mode of the first virtual image 30 (step S122; NO), the CPU 11 reflects the changed display mode of the first virtual image 30 in the second virtual image 40 displayed on the wearable terminal device 20 of the instructing user (the user who issued the instruction to reflect the display mode of the first virtual image 30 in the second virtual image 40) (step S126). For example, if the shape of the first virtual image 30 is changed to a rectangular parallelepiped shape as described above (see FIG. 10), the CPU 11 reflects the changed rectangular parallelepiped shape in the second virtual image 40 displayed on the wearable terminal device 20 of the instructing user (see FIG. 13). Then, the CPU 11 returns the process to the virtual image display control process (see FIG. 5) and performs the processes from step S109 onwards.

[0072] Returning to the explanation of the control procedure of the virtual image display control processing in Figure 5, the CPU 11 determines whether or not information instructing a change in the display mode of the second virtual image 40 has been acquired from the wearable terminal device 20 via the communication unit 14 (step S109).

[0073] If it is determined in step S109 that information instructing a change in the display mode of the second virtual image 40 has been acquired from the wearable terminal device 20 (step S109; YES), the CPU 11 changes the display mode of the second virtual image 40 displayed on the wearable terminal device 20 based on the information (step S110). Specifically, if the information instructing a change in the display mode of the second virtual image 40 acquired from the wearable terminal device 20 instructs, for example, to change the display color of the second virtual image 40 to red, the CPU 11 changes the display color of the second virtual image 40 to red (in the drawing, red is represented by a diagonal line slanting upward to the right) based on the information, as shown in FIG. Furthermore, if the above information acquired from the wearable terminal device 20 is, for example, information instructing to change the display color of the second virtual image 40 to red and then to yellow, the CPU 11 changes the display color of the second virtual image 40 to red and then to yellow (in the figure, yellow is represented by a diagonal line slanting downward to the right) based on the information, as shown in FIG. 15.

[0074] FIG. 16 is a diagram showing an example of a visual recognition area visually recognized by a first user U1 wearing the wearable terminal device 20, as well as a first virtual image 30 and a second virtual image 40. In FIG. As shown in Figure 16, when the display color of the second virtual image 40 is changed to red in response to the operation of the first user U1, the first user U1 views the first virtual image 30 (cylindrical first virtual image 30) displayed on the table T, as well as the second virtual image 40 that has been changed to red.

[0075] Returning to the explanation of the control procedure of the virtual image display control process in FIG. 5, after executing the process of step S110, the CPU 11 executes the second reflection display process (step S111).

[0076] FIG. 7 is a flowchart showing the control procedure of the second reflection display process. As shown in Figure 7, when the second reflection display process is started, first, the CPU 11 of the information processing device 10 determines whether or not instruction information for reflecting the display mode of the second virtual image 40 in the first virtual image 30 has been acquired from the wearable terminal device 20 via the communication unit 14 (step S141).

[0077] In step S141, if it is determined that instruction information for reflecting the display mode of the second virtual image 40 in the first virtual image 30 has not been acquired from the wearable terminal device 20 (step S141; NO), the CPU 11 returns the processing to the virtual image display control processing (see Figure 5) and performs processing from step S112 onwards.

[0078] Also, in step S141, if it is determined that instruction information for reflecting the display mode of the second virtual image 40 in the first virtual image 30 has been acquired from the wearable terminal device 20 (step S141; YES), the CPU 11 determines whether there were multiple change patterns for the display mode of the second virtual image 40 changed in step S110 of the virtual image display control process (see Figure 5) (step S142).

[0079] If it is determined in step S142 that there are multiple change patterns for the display mode of the second virtual image 40 (step S142; YES), the CPU 11 determines whether or not change pattern selection information has been acquired from the wearable terminal device 20 worn by the instructing user (the user who has instructed to reflect the display mode of the second virtual image 40 in the first virtual image 30) via the communication unit 14 (step S143). Note that in a case where a desired change pattern is selected from multiple change patterns, if the change patterns include a change pattern in which a third additional image (not shown) is added to the second virtual image 40 in a certain display mode and a change pattern in which a fourth additional image (not shown) is added to the second virtual image 40 in the certain display mode, a change pattern in which both the third additional image and the fourth additional image are added to the second virtual image 40 in the certain display mode may be further added.

[0080] In step S143, if it is determined that the selection information of the change pattern has not been acquired (step S143; NO), the CPU 11 repeats the determination process of step S143 until the selection information is acquired.

[0081] Furthermore, when it is determined in step S143 that selection information of a change pattern has been acquired (step S143; YES), the CPU 11 reflects the selected change pattern based on the acquired selection information in the first virtual image 30 displayed on each wearable terminal device 20 (step S144). For example, when the display color of the second virtual image 40 is changed to red (first change pattern) and then changed to yellow (second change pattern) as described above (see FIG. 15), and selection information selecting red (first change pattern) is acquired as selection information of the change pattern from the wearable terminal device 20 worn by the instructing user, the CPU 11 reflects red, which is the selected change pattern (first change pattern), in the first virtual image 30 displayed on each wearable terminal device 20.

[0082] FIG. 17 is a diagram showing an example of a visual recognition area visually recognized by a first user U1 wearing the wearable terminal device 20, as well as a first virtual image 30 and a second virtual image 40. In FIG. As shown in Figure 17, when the display color (red) of the second virtual image 40, which has been changed in response to the operation of the first user U1, is reflected in the first virtual image 40, the first user U1 views the second virtual image 40 that has been changed to red, and also views the first virtual image 30 that reflects the red color.

[0083] Returning to the explanation of the control procedure of the second reflected display process in FIG. 7, after executing the process of step S144, the CPU 11 determines whether or not information instructing the user to decide on the reflected display has been acquired from the wearable terminal device 20 worn by the instructing user (step S145).

[0084] In step S145, if it is determined that information instructing determination of reflected display has been acquired (step S145; YES), the CPU 11 returns the process to the virtual image display control process (see FIG. 5), and performs the processes from step S112 onwards.

[0085] Furthermore, in step S145, if it is determined that information instructing to determine the reflected display has not been acquired (step S145; NO), the CPU 11 returns the process to step S143 and repeats the subsequent processes.

[0086] Furthermore, if it is determined in step S142 that there are not multiple change patterns for the display mode of the second virtual image 40 (step S142; NO), the CPU 11 reflects the changed display mode of the second virtual image 40 in the first virtual image 30 displayed on each wearable terminal device 20 (step S146). For example, if the display color of the second virtual image 40 is changed to red as described above (see FIG. 14), the CPU 11 reflects the changed red in the first virtual image 30 displayed on each wearable terminal device 20 (see FIG. 17). Then, the CPU 11 returns the process to the virtual image display control process (see FIG. 5) and performs the processes from step S112 onwards.

[0087] Returning to the explanation of the control procedure of the virtual image display control process in FIG. 5, the CPU 11 determines whether or not information instructing the end of the virtual image display control process has been acquired from the wearable terminal device 20 via the communication unit 14 (step S112).

[0088] In step S112, if it is determined that information instructing the end of the virtual image display control process has not been acquired from the wearable terminal device 20 (step S112; NO), the CPU 11 returns the process to step S101 and repeats the subsequent processes.

[0089] Furthermore, in step S112, if it is determined that information instructing the end of the virtual image display control process has been acquired from the wearable terminal device 20 (step S112; YES), the CPU 11 ends the virtual image display control process.

[0090] <Other> The above embodiment is an example, and various modifications are possible. For example, in the virtual image sharing system 100 of the above embodiment, a case has been described in which the first virtual image 30, displayed as if it were present in real space, is shared among users. However, the first virtual image 30 may be displayed in a virtual space and shared among avatars whose display changes in response to the actions of each user. In such a case, a VR-type wearable terminal device is used. Furthermore, in such a case, when each user performs an operation on the second virtual image 40, the operation corresponding to the operation is not reflected in the display of the avatar. This allows the second virtual image 40 to be operated without the knowledge of other users. Furthermore, in such a case, when one user performs an operation on the second virtual image 40, operations on the first virtual image 30 are disabled. This allows each user to refer to the first virtual image 30 when performing an operation on the second virtual image 40, thereby enabling the operation to be performed smoothly.

[0091] In addition, in the above embodiment, in step S107 of the virtual image display control process (see Figure 5), the display mode of the first virtual image 30 is changed based on information obtained from the wearable terminal device 20 that instructs a change in the display mode of the first virtual image 30, but it may also be possible to set in the information processing device 10 which users are allowed to make the change and which users are not allowed to make the change.

[0092] Furthermore, in the above embodiment, when it is determined that information instructing the display of the second virtual image 40 has been acquired from the wearable terminal device 20, the second virtual image 40 is displayed on the wearable terminal device 20 worn by the user who issued the instruction to display the second virtual image 40. However, for example, the second virtual image 40 may be displayed on the wearable terminal device 20 on the condition that the first virtual image 30 is displayed on the wearable terminal device 20. Furthermore, even if the first virtual image 30 is not displayed on the wearable terminal device 20, the second virtual image 40 may be displayed on the wearable terminal device 20 on the condition that the second virtual image 40 is located near the display position of the first virtual image 30.

[0093] In addition, in the above embodiment, in step S111 of the virtual image display control process (see Figure 5), a second reflection display process is executed, and the display mode of the second virtual image 40 is reflected in the first virtual image 30 based on instruction information for reflecting the display mode of the second virtual image 40 in the first virtual image 30.However, it may also be possible to set in the information processing device 10 the users who are allowed to perform the reflection display and the users who are not allowed to perform the reflection display.

[0094] Furthermore, the display modes of the first virtual image 30 and the second virtual image 40 described in the above embodiment are merely examples.

[0095] Furthermore, in the above embodiment, the first virtual image 30 and the second virtual image 40 are used in combination to improve the usability of the first virtual image 30. However, for example, when a first user U1 wearing one wearable terminal device 20 performs an operation to change the display mode of the first virtual image 30, the display mode of the first virtual image 30 changed based on the operation is reflected in the display of the first virtual image 30 by the one wearable terminal device 20, and is also reflected in the display of the first virtual image 30 by other wearable terminal devices 20. The wearable terminal device 20 may have a first mode in which the display mode of the first virtual image 30 is reflected in the display of the first virtual image 30 by the first user U1 wearing the wearable terminal device 20, and a second mode in which, when a first user U1 wearing the wearable terminal device 20 performs an operation to change the display mode of the first virtual image 30, the display mode of the first virtual image 30 changed based on the operation is reflected in the display of the first virtual image 30 by the one wearable terminal device 20, but is not reflected in the display of the first virtual image 30 by other wearable terminal devices 20. By switching between these modes, the usability of the first virtual image 30 may be improved.

[0096] In the second mode, a method for reflecting the display mode of the first virtual image 30 changed based on the operation of the first user U1 only in the display of the first virtual image 30 by the wearable terminal device 20 of the first user U1 is, for example, to transmit information instructing the first user U1 to change the display mode of the first virtual image 30 from the wearable terminal device 20 worn by the first user U1 to the information processing device 10. Then, the information processing device 10 generates information related to the changed display mode of the first virtual image 30 based on the information and transmits the information only to the wearable terminal device 20 worn by the first user U1. Another method is, for example, to transmit virtual image data 132 related to the first virtual image 30 in advance from the information processing device 10 to the wearable terminal device 20 worn by the first user U1. One method is for the wearable terminal device 20 that has acquired the virtual image data 132 to change the display mode of the first virtual image 30 by performing a display control process independently (standalone) in response to the operation of the first user U1.

[0097] In addition, the specific details of the configurations and controls shown in the above embodiments can be appropriately changed without departing from the spirit of the present disclosure. Furthermore, the configurations and controls shown in the above embodiments can be appropriately combined without departing from the spirit of the present disclosure. [Industrial Applicability]

[0098] The present disclosure can be used for a virtual image sharing method and a virtual image sharing system. [Explanation of symbols]

[0099] 100 Virtual Image Sharing System 10. Information processing equipment 11 CPU 12 RAM 13 Storage section 131 Programs 132 Virtual Image Data 14 Communications Department 15 Bus 20 Wearable terminal device 21 CPU 22 RAM 23 Memory section 231 Programs 24 Display section 241 Visor 242 Laser Scanner 25 Sensor section 251 Accelerometer 252 Angular velocity sensor 253 Depth Sensor 254 Camera 255 Eye Tracker 26 Communications Department 27 Bus

Claims

1. A virtual image sharing method for sharing a virtual image between at least a first display device and a second display device, comprising: displaying, on each of the first display device and the second display device, a first virtual image located within a virtual space and operable by a first user of the first display device and a second user of the second display device; displaying a first avatar in the virtual space, the first avatar being displayable on each of the first display device and the second display device and whose display changes in response to an action of the first user; A virtual image sharing method, wherein when the first user performs an operation on a second virtual image that is located in the virtual space and not displayed on the second display device, the action corresponding to the operation is not reflected in the display of the first avatar.

2. The virtual image sharing method according to claim 1 , wherein the second virtual image is a virtual image based on the first virtual image.

3. The virtual image sharing method according to claim 2 , wherein the second virtual image is a virtual image obtained by duplicating the first virtual image.

4. A virtual image sharing method as described in claim 1, wherein when at least one of the first user and the second user performs an operation to change the display mode of the first virtual image, the change is reflected in the first virtual image displayed on the first display device and the second display device, respectively.

5. The virtual image sharing method according to claim 1 , wherein when the first user is performing an operation on the second virtual image, an operation on the first virtual image is disabled.

6. A virtual image sharing method as described in claim 3, wherein when the second user performs an operation to change the display mode of the first virtual image, the changed display mode of the first virtual image is reflected in the second virtual image displayed on the first display device in accordance with a predetermined operation by the first user.

7. A virtual image sharing method as described in claim 6, wherein when the second user performs an operation to change the display mode of the first virtual image to a first display mode and an operation to change it to a second display mode, either the first display mode or the second display mode is selectively reflected in the second virtual image displayed on the first display device in accordance with a predetermined operation by the first user.

8. 8. The virtual image sharing method of claim 7, wherein, when the first display mode is a display mode in which a first additional image is added to the first virtual image and the second display mode is a display mode in which a second additional image is added to the first virtual image, the second display mode, or a third display mode in which the first additional image and the second additional image are added to the first virtual image is selectively reflected in the second virtual image displayed on the first display device in accordance with a predetermined operation by the first user.

9. A virtual image sharing method as described in claim 3, wherein when the first user performs an operation to change the display mode of the second virtual image, the changed display mode of the second virtual image is reflected in the first virtual image in accordance with a predetermined operation by the first user.

10. A virtual image sharing method as described in claim 9, wherein when the first user performs an operation to change the display mode of the second virtual image to a fourth display mode and an operation to change it to a fifth display mode, either the fourth display mode or the fifth display mode is selectively reflected in the first virtual image in accordance with a predetermined operation by the first user.

11. The virtual image sharing method of claim 10, wherein, when the fourth display mode is a display mode in which a third additional image is added to the second virtual image and the fifth display mode is a display mode in which a fourth additional image is added to the second virtual image, the first virtual image is selectively reflected in one of the fourth display mode, the fifth display mode, and a sixth display mode in which the third additional image and the fourth additional image are added to the second virtual image in accordance with a predetermined operation by the first user.

12. A virtual image sharing method for sharing a virtual image among a plurality of display devices, comprising: displaying, on each of the plurality of display devices, a virtual image located within the space and operable by a user of each of the plurality of display devices; When an operation to change the display mode of the virtual image is performed by a user of one of the plurality of display devices, the display mode of the virtual image changed based on the operation is a first mode in which the virtual image is reflected in the display of the virtual image by the one display device and is also reflected in the display of the virtual image by the other display devices of the plurality of display devices; When an operation to change the display mode of the virtual image is performed by a user of one of the plurality of display devices, the display mode of the virtual image changed based on the operation is a second mode in which the virtual image is reflected in the display of the one display device, but is not reflected in the display of the virtual image of the other display devices of the plurality of display devices; A virtual image sharing method comprising:

13. A virtual image sharing system for sharing a virtual image between at least a first display device and a second display device, displaying, on each of the first display device and the second display device, a first virtual image located within a virtual space and operable by a first user of the first display device and a second user of the second display device; displaying a first avatar in the virtual space, the first avatar being displayable on each of the first display device and the second display device and whose display changes in response to an action of the first user; A virtual image sharing system in which, when the first user performs an operation on a second virtual image located in the virtual space and not displayed on the second display device, the action corresponding to the operation is not reflected in the display of the first avatar.

14. A virtual image sharing system for sharing a virtual image among a plurality of display devices, displaying, on each of the plurality of display devices, a virtual image located within the space and operable by a user of each of the plurality of display devices; When an operation to change the display mode of the virtual image is performed by a user of one of the plurality of display devices, the display mode of the virtual image changed based on the operation is a first mode in which the virtual image is reflected in the display of the virtual image by the one display device and is also reflected in the display of the virtual image by the other display devices of the plurality of display devices; When an operation to change the display mode of the virtual image is performed by a user of one of the plurality of display devices, the display mode of the virtual image changed based on the operation is a second mode in which the virtual image is reflected in the display of the one display device, but is not reflected in the display of the virtual image of the other display devices of the plurality of display devices; A virtual image sharing system comprising:

Images