Communication support system

Abstract

To provide a communication support system which can enhance a value of use of a composite real space by making it possible to confirm a state of a target point.SOLUTION: A communication support system according to one embodiment of the present invention has a plurality of user terminals, a first server for providing user terminals with an object for a composite real space, and a second server for providing the user terminal with 360 degree video captured by a 360 degree camera disposed at a target position. The plurality of user terminals includes a video transmission terminal for transmitting the 360 degree video to the second server, and a video integration terminal for integrating the object provided by the first server with the 360 degree video provided with the second server to display, on a display unit, a composite real space including the object with the 360 degree video.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a communication support system, and more particularly to a communication support system that displays a mixed reality space on a display unit of a user terminal. [Background technology]

[0002] A technology for displaying a mixed reality space in which an image of a real space is combined with an object such as a virtual object may be used, for example, to support communication between users. In this case, the user operates a client terminal (hereinafter also referred to as a user terminal) to use (display) the mixed reality space. The user terminal communicates with a server to obtain information required to display the mixed reality space, thereby displaying the mixed reality space on a display unit of the user terminal (for example, see Patent Document 1).

[0003] According to the mixed reality space sharing system described in Patent Document 1, a user terminal operated by a user downloads from a server 3D information obtained by sensing a real space in which another user exists, and anchor information for fixing the 3D information and virtual information. Then, a mixed reality space created by the downloaded 3D information and anchor information is displayed on a display unit of the user terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7207549 Summary of the Invention [Problem to be solved by the invention]

[0005] The types of objects included in the mixed reality space can be set according to the user's requests, and the system specifications and performance. If it were possible to display an object that represents the situation of a location (hereinafter also referred to as a target location) that is of great interest to users, such as a venue where multiple users gather, it is expected that the utility value of the mixed reality space can be further increased. Furthermore, it would be even more preferable if such an object provided detailed information about the target location.

[0006] Therefore, the present invention has been made in consideration of the above-mentioned problems, and its object is to provide a communication support system that can increase the utility of mixed reality space by making it possible to check the situation at a target location. [Means for solving the problem]

[0007] The above-mentioned problem is solved by the communication support system of the present invention, which has a plurality of user terminals, a first server that provides the user terminals with objects for a mixed reality space that is overlaid on the real space in which the user of the user terminal is located and can be displayed through a display unit of the user terminal, and a second server that provides the user terminals with 360-degree video captured by a 360-degree camera placed at a target location, wherein the plurality of user terminals include a video transmission terminal that captures 360-degree video with the 360-degree camera and transmits the 360-degree video to the second server, and a video integration terminal that integrates the 360-degree video provided by the second server with the objects provided by the first server, and displays a mixed reality space including the objects with the 360-degree video on the display unit.

[0008] According to the communication support system of the present invention configured as described above, a 360-degree image of a target location is displayed as one of the objects in the mixed reality space. This allows the user to check the situation of the target location, thereby further increasing the utility value of the mixed reality space displayed on the user terminal.

[0009] In addition, in the communication support system of the present invention, the video integration terminal may have a function of transmitting information on the voice collected by the video integration terminal to the first server. Also, the user terminal of the user at the target location may have a function of transmitting information on the voice collected by the user terminal to the second server. In this configuration, when an operation for starting a call is performed on the video integration terminal displaying the mixed reality space on the display unit, the second server may transmit information on the voice collected by the user terminal of the user at the target location to the video integration terminal on which the operation for starting the call was performed. Also, the first server may transmit information on the voice collected by the video integration terminal to the user terminal of the user at the target location. According to the above configuration, a user who operates the video integrated terminal can make a call with a user who is at a target location by performing an operation for starting a call.

[0010] In addition, in the communication support system of the present invention, the mode in which the second server provides the 360-degree video to the user terminal may be switchable between a first providing mode in which the second server provides the current 360-degree video most recently received from the video transmitting terminal, and a second providing mode in which the second server provides the past 360-degree video stored in the storage device. In this case, the above mode may be determined according to a selection operation performed on the video integration terminal from the first providing mode or the second providing mode. According to the above configuration, it is possible for a user of the image integration terminal to display, as a 360-degree image of a target location, either a real-time image or a past image, whichever is desired, in a mixed reality space.

[0011] In the communication support system of the present invention, the plurality of user terminals may include two or more video integration terminals, and the first server may provide objects including avatars corresponding to the users of the respective video integration terminals. In this case, it is more preferable that each video integration terminal displays the mixed reality space in which the avatar generated for each video integration terminal is arranged when displaying the mixed reality space including the object with the 360-degree video on the display unit. According to the above configuration, by visually recognizing an avatar displayed in a mixed reality space, it becomes possible to recognize the presence of a user who is viewing (using) the same mixed reality space, and to share information, i.e., communicate, with that user.

[0012] Furthermore, in the communication support system of the present invention, when a user operation is performed on a 360-degree image in a mixed reality space at any of the image integration terminals, it is even more preferable for each image integration terminal to enlarge the display area of ​​the 360-degree image on the display unit and place the avatar generated for each image integration terminal within the enlarged display area. According to the above configuration, the display area of ​​the 360-degree video on the display unit is enlarged and each user's avatar is displayed within the enlarged display area, thereby giving the user using the display unit a sense of immersion in the 360-degree video.

[0013] In the communication support system of the present invention, the display unit of the video integration terminal may be configured with a head mounted display. In this case, the first server may provide an object including an avatar that does not have a portion corresponding to the head mounted display as an avatar corresponding to the user wearing the head mounted display. According to the above configuration, even if a user of the video integrated terminal is wearing a head mounted display, the user can easily recognize an avatar corresponding to the user in the mixed reality space.

[0014] In the communication support system of the present invention, the first server may provide the video integration terminal with an object including a pen-shaped object. Also, while a mixed reality space including an object with a 360-degree image is displayed on a display unit of the video integration terminal, an operation of moving the pen-shaped object in the mixed reality space may be performed on the video integration terminal. In this case, the first server may provide the video integration terminal with an object further including a drawing object based on a movement trajectory of the pen-shaped object. According to the above configuration, by displaying an object (drawn object) drawn by a user by moving a pen-type object in a mixed reality space, the object can be used as a tool for communication between users.

[0015] In the communication support system of the present invention, the video integration terminal may have a function of transmitting information on the voice collected by the video integration terminal. In this case, it is preferable that the first server provides an object including a text object indicating a text based on the voice collected by the video integration terminal. Furthermore, it is more preferable if the above text object has a frame, the text is displayed inside the frame, and the video integration terminal displays on the display unit a mixed reality space in which the text object is placed at a display position set in the mixed reality space. According to the above configuration, the contents of speech by a user of the video integrated terminal can be converted into text and displayed in the mixed reality space, and an object showing the text can be used as a tool for communication between users.

[0016] The communication support system of the present invention may further include a storage device that stores elements of an object included in the mixed reality space displayed on the display unit of the video integration terminal together with display positions of the elements in the mixed reality space. The first server may provide an object including the elements stored in the storage device. In this case, the video integration terminal may display on the display unit the mixed reality space in which the elements are arranged at the display positions stored in the storage device. According to the above configuration, elements in a mixed reality space can be stored (saved). Furthermore, when the mixed reality space is to be newly displayed at a later point in time, the elements of the stored objects can be read out and displayed in the new mixed reality space at the same positions as the previous display positions.

[0017] In the communication support system of the present invention, the video integration terminal may have a function of acquiring a capture image of the mixed reality space displayed on the display unit. In this case, the first server may receive the capture image from the video integration terminal that acquired the capture image, store the capture image in a storage device, and transmit the capture image stored in the storage device to the user terminal. According to the above configuration, a captured image (still image) of the mixed reality space is acquired, and the user can view the captured image through the user terminal. This allows the content of the mixed reality space at the time the captured image was acquired to be confirmed at a later time.

[0018] In the communication support system of the present invention, the first server may provide a three-dimensional object, and the object may include a spherical object. In this case, the video integration terminal may display a 360-degree video in an area corresponding to an outer surface of the spherical object in the mixed reality space. According to the above configuration, the 360-degree video can be displayed in the mixed reality space so that the 360-degree video can be easily viewed by a user viewing the mixed reality space on the display unit of the video integration terminal. Effect of the Invention

[0019] According to the present invention, a communication support system is realized that can increase the value of using mixed reality space by making it possible to check the situation at a target location. [Brief description of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating an example of the configuration of a communication support system according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a diagram showing an example of a mixed reality space displayed on a display unit of a user terminal. [Diagram 3] FIG. 13 is a diagram showing a mixed reality space in which an instruction object is displayed. [Figure 4]FIG. 13 is a diagram showing a mixed reality space in which a pen-shaped object is displayed. [Diagram 5] FIG. 13 is a diagram showing a mixed reality space in which a drawing object is displayed. [Figure 6] FIG. 13 is a diagram showing a mixed reality space in which a text object is displayed. [Figure 7] FIG. 13 illustrates a mixed reality space in which elements of a stored object are redisplayed. [Figure 8] FIG. 13 is a diagram showing a mixed reality space in which a captured image is displayed. [Figure 9] FIG. 13 is a diagram showing a mixed reality space in which selection objects are displayed. [Figure 10] FIG. 1 is a diagram showing a flow of data processing by a communication support system according to an embodiment of the present invention (part 1). [Figure 11] FIG. 2 is a diagram showing a flow of data processing by the communication support system according to an embodiment of the present invention (part 2). [Figure 12] FIG. 13 is a diagram showing an example of a screen when a 360-degree video is enlarged and displayed. [Figure 13] FIG. 13 is a diagram showing the flow of a voice call with a user at a target location. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] <<About one embodiment of the present invention>> Hereinafter, a communication support system according to one embodiment of the present invention (hereinafter, this embodiment) will be described with reference to the accompanying drawings. In the following, for convenience of explanation, the present invention may be described from the viewpoint of a GUI (Graphic User Interface). Also, among the basic technologies necessary for implementing the present invention, specifically, data processing technologies such as communication / transmission technology, data acquisition technology, data recording technology, data processing / analysis technology, image processing technology, machine learning technology, voice analysis technology (including natural language processing), and visualization technology, explanations of well-known technologies will be omitted.

[0022] [Outline of the communication support system according to the present embodiment] The communication support system according to this embodiment (hereinafter, CS system 10) is an information and communication system constructed for the purpose of providing a mixed reality space MR (Mixed Reality) and supporting communication between users through the mixed reality space MR. A user is a user of the CS system 10 and an operator of a user terminal 12 shown in FIG. 1. According to the CS system 10, a plurality of users can view a mixed reality space MR as shown in FIG. 2 through a display unit 14 of each user terminal 12, in other words, the mixed reality space MR can be displayed through the display unit 14. There is no particular limitation on the use of the mixed reality space MR, but examples include conversations, meetings, gatherings, conferences, meetings, lessons, lectures, seminars, events, and other situations involving communication between users.

[0023] In this embodiment, the display unit 14 is configured by a head mounted display (hereinafter, HMD) such as MR goggles as shown in Fig. 1, and the user can view the mixed reality space MR by wearing the HMD. The HMD is connected to the main body of the user terminal 12 in a wired or wireless manner. However, the display unit 14 is not limited to the HMD, and may be a display provided in the user terminal 12 itself, or a stationary display monitor connected to the main body of the user terminal 12.

[0024] As shown in Fig. 2 etc., the mixed reality space MR is created by overlaying an object J composed of a virtual object, etc., on the real space where the user is located, and more specifically, is a composite (fusion) of an image of the real space and an object J generated by computer graphics and image processing technology. The mixed reality space MR is a three-dimensional space, and correspondingly, the object J included in the mixed reality space MR is also a three-dimensional object. However, it is not limited to this, and the object J may be a two-dimensional object.

[0025] In the following description, when the term "object J" is used, it means the entire object J included in the mixed reality space MR. In the present embodiment, the object J is composed of one or more elements. The elements are individual virtual objects and images displayed in the mixed reality space MR, and correspond to an avatar Ja, a drawing object Jb, a text object Jc, a pen-shaped object Jd, a spherical object Je, a captured image Jf, an instruction object Jg, and a selection object Jh, which will be described later.

[0026] Moreover, the image of the real space constituting the mixed reality space MR is acquired by a camera or a sensor attached to the HMD at a viewing angle or detection range set to match the line of sight of the user who is wearing the HMD. Also, when the direction of the head of the user wearing the HMD, i.e., the direction of the line of sight, changes, the image of the real space displayed on the HMD changes according to the changed direction, and accordingly, the display position of the object J also changes.

[0027] In addition, in this embodiment, users viewing the mixed reality space MR can make voice calls with each other. Specifically, the voice of one user is collected by the user terminal 12 (strictly speaking, the microphone 16 shown in FIG. 1), and the collected voice information is transmitted to the other user. On the other user's side, the user terminal 12 receives the voice information, reproduces the voice information, and emits the reproduced sound from the speaker 18. This allows the users to send their own voices to each other and hear the other user's voice.

[0028] Furthermore, in this embodiment, as shown in FIG. 2, an image of a predetermined location (hereinafter, a target location) is displayed in the mixed reality space MR. The user can check the situation of the target location by viewing the image of the target location displayed in the mixed reality space MR. The target location is a location away from the user's location, that is, a location in a remote location. The target location can be set arbitrarily in the real space, and for example, a location of high interest to the user, specifically, a location such as a venue for a meeting that the user is to attend, may be selected as the target location.

[0029] Moreover, the image of the target point displayed in the mixed reality space MR is a 360-degree image P captured by the 360-degree camera 20 shown in Fig. 1 (see Fig. 2). That is, in this embodiment, the 360-degree image P is integrated with an object J, and the object J with the 360-degree image is displayed in the mixed reality space MR. More specifically, the object J constituting the mixed reality space MR includes a spherical object Je shown in Fig. 2. Then, as shown in Fig. 2, the 360-degree image P is displayed in an area (display area) corresponding to the outer surface of the spherical object Je in the mixed reality space MR.

[0030] As described above, in this embodiment, the 360-degree image P is displayed as an image of the target location in the mixed reality space MR. This allows the user to view the omnidirectional image captured from the target location (specifically, the installation point of the 360-degree camera 20) through the mixed reality space MR and learn details about the situation at the target location. As described above, the 360-degree image P is displayed in an area corresponding to the outer surface of the spherical object Je in the mixed reality space MR, making it easier for the user viewing the mixed reality space MR through the HMD to view.

[0031] In this embodiment, the 360-degree video P displayed in the mixed reality space MR is displayed (played) in a streaming manner. The 360-degree video P displayed in the mixed reality space MR can be switched between a real-time video captured just before by the 360-degree camera 20 and an archive video captured in the past by the 360-degree camera 20 and stored (saved) in a predetermined storage device. In other words, the 360-degree video P selected by the user from the real-time video and the archive video is displayed in the mixed reality space MR (strictly speaking, the display area formed by the outer surface of the spherical object Je).

[0032] The real-time video may be displayed on the viewer's user terminal 12 with a slight delay from the time the video was acquired, and at least a delay sufficient to establish communication (more specifically, voice communication) between users through the mixed reality space MR is acceptable. As a delivery method for the real-time video, known live delivery methods such as HTTP Live Streaming, Common Media Application Format, Web Real-Time Communications, Real-Time Messasing Protocol, and MPEG DASH can be used.

[0033] Furthermore, in this embodiment, when a user viewing the mixed reality space MR through the display unit 14 performs an operation to start a call on the 360-degree image P, this triggers a voice call with a user who is located at the target location where the 360-degree camera 20 is installed.

[0034] More specifically, the voice of a user viewing the mixed reality space MR (hereinafter, MR-using user) is collected by a microphone 16 of a user terminal 12 on the MR-using user side. The collected voice information of the MR-using user is transmitted to a user at a target point (hereinafter, target point user). The user terminal 12 on the target point user side receives the voice information of the MR-using user, plays the voice, and outputs the played sound from a speaker 18. Similarly, the voice of the target point user is collected by the microphone 16 of the user terminal 12 on the target point user side, and the collected voice information of the target point user is transmitted to the MR using user. The user terminal 12 on the MR using user side receives the voice information of the target point user, plays the voice, and outputs the played sound from the speaker 18.

[0035] As described above, in the CS system 10, the MR user can have a two-way voice call with a user (target point user) who is at a target point shown in the 360-degree video P, and can hear each other's voice. With such a function of the CS system 10, the target point user can have a voice call with the MR user, for example, without using a headset. During this voice call, the user terminal 12 of the MR using user suspends the playback and output of voices of users other than the target location user. In other words, while the MR using user is talking to the target location user, the MR using user does not accept calls from users other than the target location user, and can concentrate on the call with the target location user. However, this is not limited to this, and the MR using user may continue to talk to other users (more specifically, other MR using users) while talking to the target location user. In other words, one MR using user may make calls with multiple users including the target location user at the same time.

[0036] [Example of the configuration of the communication support system according to this embodiment] Next, a configuration example of the CS system 10 will be described with reference to FIG. As shown in Fig. 1, the CS system 10 has a plurality of user terminals 12, a first server 22, a second server 24, and a storage device 26. Although four user terminals 12 are shown in Fig. 1, the number of user terminals 12 constituting the CS system 10 may be two or more, and in reality, the number of user terminals 12 corresponds to the number of users who use the CS system 10.

[0037] The user terminal 12 is a client terminal operated by a user, and is configured by, for example, a PC (Personal Computer), a smartphone, a mobile phone, a tablet terminal, other information processing terminals, etc. The user terminal 12 has a processor such as a CPU (Central Processing Unit), a semiconductor memory, a communication interface, a storage as an auxiliary storage device, an input device, and an output device.

[0038] The first server 22 provides an object J for the mixed reality space MR to the user terminal 12. The second server 24 provides the user terminal 12 with a 360-degree video P captured by a 360-degree camera 20 placed at a target point.

[0039] The first server 22 and the second server 24 are each communicatively connected to the user terminal 12 via a communication network N such as the Internet and a mobile communication network. The hardware configuration of each of the first server 22 and the second server 24 is common to that of a general server computer, and includes a processor such as a CPU (Central Processing Unit), a semiconductor memory, a communication interface, and a storage as an auxiliary storage device. In this embodiment, the first server 22 and the second server 24 are each configured by one computer, but the present invention is not limited to this, and each server may be configured by a plurality of computers (server computers).

[0040] The storage device 26 is a device that stores information about the object J, information about the 360-degree image P, and other information about the mixed reality space MR. The first server 22 and the second server 24 can each store (save) information in the storage device 26, and can also read out the information stored in the storage device 26. The storage device 26 may be configured by a storage included in the first server 22 or the second server 24, or may be configured by a database server or a server for cloud services. The storage device 26 may also be configured by a NAS (Network Attached Storage), online storage, or the like.

[0041] As shown in Fig. 1, the multiple user terminals 12 constituting the CS system 10 are divided into a video transmission terminal 12A, a video integration terminal 12B, and a target point terminal 12C. To be more specific, the user terminal 12 equipped with the 360-degree camera 20 arranged at the target point corresponds to the video transmission terminal 12A, and an application program for enabling the user terminal 12 to function as the video transmission terminal 12A is installed in the user terminal 12. The user terminal 12 on the MR user side described above corresponds to the video integration terminal 12B, and an application program for enabling the user terminal 12 to function as the video integration terminal 12B is installed in the user terminal 12. The user terminal 12 on the target point user side described above corresponds to the target point terminal 12C, and an application program for enabling the user terminal 12 to function as the target point terminal 12C is installed in the user terminal 12.

[0042] In this embodiment, there are two or more MR-using users. Therefore, the multiple user terminals 12 constituting the CS system 10 include two or more video integration terminals 12B. However, this is not limited to the above, and there may be only one MR-using user. In this case, among the multiple user terminals 12 constituting the CS system 10, there may be only one user terminal 12 corresponding to the video integration terminal 12B. In this embodiment, one target point is set, and the multiple user terminals 12 constituting the CS system 10 include one video transmission terminal 12A and one target point terminal 12C. However, without being limited to this, two or more target points may be set, and a 360-degree camera 20, a video transmission terminal 12A, and a target point terminal 12C may be used for each target point. In that case, the multiple user terminals 12 constituting the CS system 10 include two or more video transmission terminals 12A, and two or more target point terminals 12C. Then, when the 360-degree camera 20 and the video transmission terminal 12A are used at each of the two or more target points, the 360-degree video P for each target point may be displayed in the mixed reality space MR.

[0043] 1, the video transmission terminal 12A includes a 360-degree camera 20, captures 360-degree video P with the 360-degree camera 20, and transmits the 360-degree video P to the second server 24. The 360-degree camera 20 and the video transmission terminal 12 may be separate devices, or may be a single device housed in the same housing.

[0044] The image integration terminal 12B receives the object J provided from the first server 22 and the 360-degree image P provided from the second server 24. Then, the image integration terminal 12B integrates the 360-degree image P with the object J, and displays a mixed reality space MR including the object J with the 360-degree image on the display unit 14 of the image integration terminal 12B (specifically, an HMD connected to the terminal body). To explain in more detail, the image integration terminal 12B displays the 360-degree image P in an area corresponding to the outer surface of the spherical object Je in the mixed reality space MR.

[0045] The video integration terminal 12B also includes a microphone 16 as an input device, and has a function of transmitting voice information of the MR user collected by the microphone 16 to the first server 22. Furthermore, when audio information of a target point user is sent from the second server 24, the video integration terminal 12B has a function of receiving the audio information and emitting a playback sound based on the received audio information from a speaker 18 included in the video integration terminal 12B. The speaker 18 of the video integration terminal 12B may be a headphone worn by the MR user or may be mounted on the HMD, which is the display unit 14. Furthermore, the image integration terminal 12B has a function of acquiring a capture image (still image) of the mixed reality space MR displayed on the display unit 14, and a function of transmitting the acquired capture image to the first server 22. A known acquisition method can be used to acquire a capture image of the mixed reality space MR, and for example, an image (camera image) of the real space that constitutes the mixed reality space MR and an object J may be stored, and a capture image may be constructed from the stored information.

[0046] The target point terminal 12C has a function of transmitting to the second server 24 information on the voice of the target point user collected by the microphone 16 of the target point terminal 12C. Furthermore, the target point terminal 12C has a function of receiving the voice information of the MR user when the voice information is sent from the first server 22, and emitting a playback sound based on the received voice information from the speaker 18 of the target point terminal 12C. The speaker 18 of the target point terminal 12C may be a headphone worn by the target point user.

[0047] [Functions of the first server] In this embodiment, the first server 22 has a function of providing an object J to the video integration terminal 12B, a function of transmitting audio information to the user terminal 12, a function of storing elements of the object J in the storage device 26, and a function of reproducing the elements stored in the storage device 26. These functions are realized by cooperation between the hardware devices (specifically, a processor, a memory, a communication interface, and a storage) included in the first server 22 and a program installed in the first server 22. Hereinafter, each function of the first server 22 will be described.

[0048] (Functionality that provides objects) The first server 22 generates an object J for the mixed reality space MR, which is made up of one or more elements, and provides it to the video integration terminal 12 B. In this embodiment, the first server 22 provides a three-dimensional object J, as described above.

[0049] The object J provided by the first server 22 may include, as a first element, an avatar Ja corresponding to the user of the image integration terminal 12B (i.e., the MR-using user). The avatar Ja corresponding to the MR-using user (hereinafter, the MR-using user's avatar Ja) is displayed in the mixed reality space MR while the MR-using user is using the mixed reality space MR, and moves according to the operation of the MR-using user. The first server 22 generates the avatar Ja of the MR-using user by a generation procedure similar to that of a general avatar, and specifically, obtains a video (self-shot video) taken by the MR-using user himself / herself, and generates the avatar Ja by applying a 3D skeletal model to the video. Then, the first server 22 provides the object J including the generated avatar Ja.

[0050] In this embodiment, as shown in FIG. 1, an avatar Ja that reproduces the upper body of the MR user is provided. However, the present invention is not limited to this, and the avatar Ja may reproduce the entire body of the MR user. Furthermore, the avatar Ja is not limited to being generated based on a video of the MR user. For example, the avatar Ja may be generated by presenting options to the MR user for each part that constitutes the avatar Ja, and combining the parts selected by the MR user.

[0051] In addition, in this embodiment, an MR user wears an HMD as the display unit 14 when using a mixed reality space MR, and an avatar that does not have a part corresponding to the HMD is generated as an avatar Ja corresponding to the MR user. That is, even if the MR user wears an HMD, the first server 22 creates an avatar Ja that imitates the MR user in a state where the MR user does not wear an HMD, and provides an object J including the avatar Ja. In this way, when there are multiple users using the same mixed reality space MR, each user can experience communication in the same space (mixed reality space MR) without being aware that the other users are wearing HMDs.

[0052] In this embodiment, two or more MR-using users may simultaneously view (use) a common mixed reality space MR. In other words, the multiple user terminals 12 may include two or more video integration terminals 12B. In this case, the first server 22 provides the object J including the avatar Ja of each MR-using user to each video integration terminal 12B. As described above, each video integration terminal 12B integrates the 360-degree video P into the object J, and displays the mixed reality space MR including the object J with the 360-degree video on the display unit 14. At this time, the display unit 14 of each video integration terminal 12B displays the mixed reality space MR in which the avatar Ja generated for each MR-using user (in other words, for each video integration terminal 12B) is arranged, as shown in FIG. 1. When using the mixed reality space MR using a user terminal 12 having a tracking function, it is preferable that the position, orientation (angle), posture, etc. of the user terminal 12 are reflected in the avatar Ja in real time. When using the mixed reality space MR using a user terminal 12 configured with a PC or the like, it is preferable that the position, orientation, etc. of the avatar Ja can be changed in real time by operating through input devices such as a mouse, keyboard, and virtual controller.

[0053] The object J provided from the first server 22 may include, as a second element, a drawing object Jb drawn by the MR-using user in the mixed reality space MR. Specifically, when the MR-using user performs a predetermined operation while using the mixed reality space MR, the first server 22 provides an instruction object Jg shown in Fig. 3 to the user terminal 12 (video integration terminal 12B) of the MR-using user. As a result, the mixed reality space MR is in a state in which the instruction object Jg is arranged, as shown in Fig. 3. After the mixed reality space MR has transitioned to such a state, the MR-using user touches the "pen display" button among the buttons provided on the instruction object Jg. In addition, operations performed by the MR user on object J in the mixed reality space MR, such as touching a button on the instruction object Jg, can be detected by a known device or sensor, and the detection results are transmitted from the video integration terminal 12B to the first server 22.

[0054] When the MR user touches the "pen display" button, the first server 22 provides an object J including a pen-shaped object Jd shown in FIG. 4 to the user terminal 12 (video integration terminal 12B) of the MR user. The MR user can perform an operation to move the pen-shaped object Jd in the mixed reality space MR (specifically, the mixed reality space MR including the object J with a 360-degree video) displayed on the display unit 14. When an operation to move the pen-shaped object Jd is performed, information on the movement trajectory of the pen-shaped object Jd is transmitted from the video integration terminal 12B to the first server 22.

[0055] When the first server 22 receives the above information, it creates a drawing object Jb based on the movement trajectory of the pen-type object Jd, and provides an object J further including the created drawing object Jb to the user terminal 12 (image integration terminal 12B) of the MR user. As a result, the drawing object Jb shown in FIG. 5 is placed in the mixed reality space MR viewed by the MR user. Here, the drawing object Jb may be, for example, a line along the movement trajectory of the pen-type object Jd as shown in FIG. 5. By placing such a drawing object Jb in the mixed reality space MR, the MR user can communicate with other users by writing using the drawing object Jb, or point out to other users a location and object (element) of interest in the mixed reality space MR.

[0056] The object J provided by the first server 22 may include a sticky note-type text object Jc as a third element. The text object Jc is a virtual card or sheet having a frame (outer edge), and text is displayed inside the frame, and can be pasted at any position in the mixed reality space MR as shown in FIG. 6. When displaying the text object Jc, the MR user touches the "text display" button among the buttons provided on the instruction object Jg described above, and then speaks the content to be displayed in the text object Jc. The user terminal 12 (video integration terminal 12B) of the MR user collects the spoken voice by the microphone 16 and transmits the collected voice information to the first server 22.

[0057] Upon receiving the above information, the first server 22 analyzes the information of the voice collected by the video integration terminal 12B, and more specifically, converts the voice into text by applying voice recognition technology and natural language processing to the voice represented by the information. Then, the first server 22 provides the object J including the text object Jc representing the text (i.e., the text based on the voice collected by the video integration terminal 12B) to the video integration terminal 12B.

[0058] The image integration terminal 12B that has received the object J displays the mixed reality space MR in a state in which the text object Jc is arranged at a display position set in the mixed reality space MR on the display unit 14. This allows the MR user to communicate with other users by writing using the text object Jc, or to convert the contents of conversations or meetings with other users into text and display them, thereby enabling smoother communication. In addition, in this embodiment, text information obtained by converting voice into text can be displayed in a manner other than the display method using the text object Jc. Specifically, in the mixed reality space MR, the text information can be displayed in a balloon format in front of the avatar Ja of the MR user who is the speaker.

[0059] The object J provided by the first server 22 may include a spherical object Je as a fourth element. As described above, the spherical object Je has a 360-degree image P projected (displayed) onto its outer surface (see FIG. 1). In other words, when displaying the mixed reality space MR on the display unit 14, the image integration terminal 12B displays the 360-degree image P in an area in the mixed reality space MR that corresponds to the outer surface of the spherical object.

[0060] In addition, in this embodiment, the size of the spherical object Je is variable, and when an MR user viewing the mixed reality space MR performs a user operation for changing the size of the spherical object Je (in other words, the 360-degree image P in the mixed reality space MR), the size is changed according to the operation. Accordingly, the size of the display area of ​​the 360-degree image P on the display unit 14 of the image integration terminal 12B changes. Therefore, for example, when a user operation for enlarging the size is performed, the display area of ​​the 360-degree image P is enlarged, and in particular, in this embodiment, the 360-degree image P is displayed in a full-screen display format. At this time, the avatar Ja that was generated for each image integration terminal 12B and displayed in the mixed reality space MR is placed in the display area of ​​the enlarged 360-degree image P (see FIG. 12). As a result, when the 360-degree image P is enlarged while the MR user is viewing the mixed reality space MR with another user, the MR user can recognize the avatar Ja of the other user in the enlarged (i.e., full-screen displayed) 360-degree image P. As a result, the MR user can be given a visual effect that makes him / her feel as if he / she is inside the 360-degree video P, that is, a sense of immersion.

[0061] The object J provided from the first server 22 may include elements other than the above elements. In this embodiment, the first server 22 can provide the object J including the captured image Jf and the selection object Jh described later to the user terminal 12, particularly the user terminal 12 (video integration terminal 12B) of the MR user. Also, as shown in FIG. 2 etc., the object J may include a still image (captured image Ji) captured at another location in the real space.

[0062] (A function that transmits voice information) When the first server 22 receives information on the voice of the MR user collected by the video integration terminal 12B, the first server 22 can transmit the voice information to the user terminal 12 (the target point terminal 12C) of the user at the target point. This function is used to realize the voice call between the MR user and the target point user described above.

[0063] (Ability to save / restore objects) The first server 22 has a function of storing in the storage device 26 elements of the object J included in the mixed reality space MR displayed on the display unit 14 of the video integration terminal 12B. The MR user may specify which elements of the object J included in the mixed reality space MR are to be stored using a known means. In addition, in the storage device 26, the element specified as the storage target is associated with the display position of the element in the mixed reality space MR and stored together with the display position. The display position of the element is specified as a coordinate position in a three-dimensional coordinate space with a reference point set in the mixed reality space MR as the origin.

[0064] The first server 22 also has a function of providing an object J including elements stored in the storage device 26. That is, the first server 22 stores elements of an object J included in a previously created mixed reality space MR (hereinafter, stored elements) in the storage device 26. Thereafter, when the first server 22 creates a new mixed reality space MR and displays it on the display unit 14, it reads out the stored elements from the storage device 26 and provides an object J including the stored elements. At this time, the first server 22 identifies the display positions of the stored elements in the previously displayed mixed reality space MR, and provides information on the display positions of the identified stored elements together with the object J.

[0065] The image integration terminal 12B that has received the object J displays on the display unit 14 the mixed reality space MR in a state in which the storage element is arranged at the display position specified by the first server 22 (i.e., the display position stored in the storage device 26 in association with the storage element). This makes it possible to display the storage element at the same position as the previous display position in the new mixed reality space MR, as shown in Fig. 7. In other words, the element (storage element) displayed in the past mixed reality space MR can be displayed (redisplayed) by reproducing the display position of the storage element in the new mixed reality space MR.

[0066] Moreover, the elements (storage elements) stored in the storage device 26 may include a capture image that can be acquired by the video integration terminal 12B. That is, in this embodiment, when the video integration terminal 12B acquires a capture image as a still image obtained by photographing the mixed reality space MR, the first server 22 can receive the capture image from the video integration terminal 12B and store it in the storage device 26. In this case, the first server 22 can read the capture image from the storage device 26 and transmit the capture image, and more specifically, provide the MR user with an object J including the read capture image. As a result, the display unit 14 of the user terminal 12 (video integration terminal 12B) of the MR user displays the mixed reality space MR in which the capture image Jf is arranged at a predetermined position, as shown in FIG. 8. As a result, the MR user can check the contents of the mixed reality space MR at the time of acquisition of the capture image Jf (i.e., the past mixed reality space MR) at a later time point.

[0067] [Second server functions] In this embodiment, the second server 24 has a function of providing the 360-degree video P to the video integration terminal 12B and a function of transmitting audio information to the user terminal 12. These functions are realized by cooperation between the hardware devices (specifically, a processor, a memory, a communication interface, and a storage) included in the second server 24 and a program installed in the second server 24. Each function of the second server 24 will be described below.

[0068] (360-degree video function) The second server 24 receives the 360-degree video P from the video transmission terminal 12A, and transmits and provides the received 360-degree video P to the user terminal 12, more specifically, the video integration terminal 12B. In this embodiment, the modes in which the second server 24 provides the 360-degree video P to the video integration terminal 12B include a first providing mode and a second providing mode, and the second server 24 can switch between these two modes. The first providing mode is a mode in which the second server 24 provides the 360-degree video P most recently received from the video transmission terminal 12A, i.e., the current real-time video. The second providing mode is a mode in which the second server 24 provides the past 360-degree video P received from the video transmission terminal 12A in the past and stored in the storage device 26, i.e., the archive video.

[0069] The second server 24 can store (save) the 360-degree video P received from the video transmission terminal 12A in the storage device 26, and can also read out the stored 360-degree video P, i.e., archived video, from the storage device 26.

[0070] The user (MR user) of the video integration terminal 12B to which the 360-degree video P is provided can select which of the above two modes the second server 24 will provide the 360-degree video P in. Specifically, the MR user can perform an operation to select either the first provision mode or the second provision mode (hereinafter, a selection operation) through the video integration terminal 12B. The mode in which the second server 24 provides the 360-degree video P is determined according to the selection operation performed on the video integration terminal 12B.

[0071] The procedure of the selection operation is not particularly limited, but for example, when an MR user using the mixed reality space MR performs a predetermined operation, a selection object Jh shown in FIG. 9 is displayed in the mixed reality space MR. The selection object Jh displays a first and a second provision mode in a selectable state, and the MR user touches one of the selected modes on the selection object Jh. Such a touch operation can be detected by a known device or sensor connected to the video integration terminal 12B, and the detection result is transmitted from the video integration terminal 12B to the second server 24. The second server 24 identifies the content of the selection operation by the MR user from the information of the detection result transmitted from the video integration terminal 12B, and adopts the mode corresponding to the identified selection operation from the first and second provision modes.

[0072] (A function that transmits voice information) When the second server 24 receives information on the voice of the target point user collected by the target point terminal 12C, the second server 24 can transmit the voice information to the user terminal 12 (video integration terminal 12B) of the MR user. This function is used to realize the voice call between the MR user and the target point user described above.

[0073] [Data processing by communication support systems] Next, a process flow when using mixed reality space MR (hereinafter, MR use flow) will be described as data processing by the above-mentioned CS system 10. Note that the process flow described below is merely an example, and unnecessary steps may be deleted, new steps may be added, or the order of steps may be changed without departing from the spirit of the present invention.

[0074] In the following, a flow of displaying a mixed reality space MR created by fusing an object J including a 360-degree image P with an image in real space on the display unit 14 of the user terminal 12 (strictly speaking, the image integration terminal 12B) and using the mixed reality space MR will be mainly described. In addition, in the following, a case where there are multiple MR-using users and the multiple MR-using users simultaneously use (share) the same mixed reality space MR will be described as an example. Note that the processing on the MR-using user side, i.e., the processing on the image integration terminal 12B, is approximately common to multiple MR-using users, so in the following, a single MR-using user will be described.

[0075] An MR user who wishes to use the mixed reality space MR performs a user operation to start using the mixed reality space MR on the video integration terminal 12B. This triggers the start of an MR usage flow, which proceeds along the flow shown in FIGS. More specifically, first, the first server 22 generates an object J including an avatar Ja and the like, and provides it to the video integration terminal 12B, which is the user terminal 12 of the MR user (S001).

[0076] Also, the video transmission terminal 12A acquires the 360-degree video P captured by the 360-degree camera 20 installed at the target location (S002), and the second server 24 receives the 360-degree video P from the video transmission terminal 12A (S003). Then, the second server 24 provides the acquired 360-degree video P to the video integration terminal 12B (S004). At this time, the second server 24 provides the 360-degree video P according to the mode selected by the MR user from the first provision mode and the second provision mode. Specifically, when the first provision mode is selected, the second server 24 transmits the 360-degree video P acquired immediately before from the 360-degree camera 20, i.e., the real-time video, to the video integration terminal 12B in a live distribution manner. On the other hand, when the second provision mode is selected, the second server 24 reads out the past 360-degree video P stored in the storage device 26, i.e., the archive video, and transmits it to the video integration terminal 12B.

[0077] The image integration terminal 12B of each MR user receives the object J from the first server 22 and receives the 360-degree image P from the second server 24 (S005). Then, the image integration terminal 12B of each MR user integrates the 360-degree image P with the object J and displays the mixed reality space MR including the object with the 360-degree image on the display unit 14 (S006). This allows each MR user to grasp the situation of the target location through the 360-degree image P displayed in the mixed reality space MR, and in particular, if the 360-degree image P is a real-time image, the current situation of the target location can be known in real time.

[0078] Also, in the above step S006, the image integration terminal 12B of each MR-using user displays the mixed reality space MR in a state in which the avatar Ja generated for each image integration terminal 12B (i.e., the avatar Ja of each MR-using user) is arranged, as shown in Fig. 1. This allows each MR-using user to recognize the presence of other MR-using users who are viewing the same mixed reality space MR, and for example, to share information with the users, i.e., to communicate with them.

[0079] In this embodiment, the first server 22 creates an avatar Ja that does not have a part corresponding to the HMD as an avatar Ja corresponding to an MR user wearing an HMD, and provides an object J including the avatar Ja to each of the image integration terminals 12B. As a result, each MR user sees an avatar not wearing an HMD as an avatar Ja corresponding to another user in the mixed reality space MR. As a result, each MR user can communicate with another user in the same mixed reality space MR without being aware that the other user is wearing an HMD, and regardless of what device the other user is using as a display unit.

[0080] Also, assume that a user operation is performed on the 360-degree image P in the mixed reality space MR at one of the image integration terminals 12B among the user terminals 12 of the multiple MR users, specifically, a size enlargement operation is performed on the spherical object Je (S007). In this case, each image integration terminal 12B enlarges the display area of ​​the 360-degree image P on the display unit 14, specifically, displays the 360-degree image P in full screen (S008). At this time, each image integration terminal 12B displays the avatar Ja generated for each image integration terminal 12B within the display area of ​​the enlarged 360-degree image P.

[0081] By performing the above step S008, as shown in Fig. 12, each avatar Ja of a plurality of MR-using users is arranged in the 360-degree video P displayed in full screen. This allows each MR-using user to see the avatar Ja of the other users in the 360-degree video P. As a result, each MR-using user can be given a sense of being at the target location displayed in the 360-degree video P, that is, a sense of immersion. With this effect, each MR-using user can, for example, view an object present at the target location together with other users in the video space that reproduces the target location, or point to the object and explain it to other users.

[0082] In addition, in a situation where the mixed reality space MR is displayed on the display unit 14 in each image integration terminal 12B, when an operation for starting a call is performed on one or more image integration terminals 12B (S009), a voice call is performed between the MR user who performed the operation and the target location user (S010). The specific procedure for the operation for starting a call is not particularly limited, but for example, the operation for starting communication may be performed by touching a "start call" button of the instruction object Jg displayed in the mixed reality space MR. In this case, the operation can be detected by a known device or sensor, and the detection result is transmitted from the image integration terminal 12B to the first server 22, the second server 24, and the target location terminal 12C.

[0083] A voice call between an MR user and a target point user proceeds according to the flow shown in FIG. 13. Specifically, when a voice call is started, the video integration terminal 12B, which has performed an operation for starting a call, collects the voice of the MR user who performed the operation, and transmits the voice information to the first server 22 (S021). The first server 22 receives the voice information sent from the video integration terminal 12B, and transmits the received voice information to the target point terminal 12C (S022). The target point terminal 12C, which has received the voice information from the first server 22, reproduces the voice based on the voice information, and outputs the reproduced sound from the speaker 18 of the target point terminal 12C (S023). This allows the target point user to hear the voice of the MR user (more specifically, the user who performed the operation for starting a call) through the target point terminal 12C.

[0084] Similarly, the target point terminal 12C collects the voice of the target point user and transmits the voice information to the second server 24 (S024). The second server 24 transmits the voice information sent from the target point terminal 12C to the video integration terminal 12B where the operation for starting the call was performed (S025). The video integration terminal 12B that receives the voice information from the second server 24 reproduces the voice based on the voice information and outputs the reproduced sound from the speaker 18 of the video integration terminal 12B (S026). As a result, the MR user who performed the operation for starting the call can hear the voice of the target point user through the video integration terminal 12B.

[0085] In the above manner, the MR-using user and the target location user can make a voice call by initiating a call operation. Note that, on the MR-using user side, while a voice call with the target location user is being made, the output of the voice of the other user is interrupted. However, this is not limited to this, and a call between the MR-using user and another user may be made while the MR-using user is making a call with the target location user. The series of steps S021 to S026 for the voice call described above are repeated until either the MR user or the target point user performs an operation to end the voice call (S027, S028).

[0086] Returning to the explanation of the MR usage flow, the series of steps S001 to S010 explained above is repeated while the MR usage users use the mixed reality space MR. Then, when all the MR usage users have finished using the mixed reality space MR (S011), the MR usage flow ends at that point.

[0087] As described above, in this embodiment, the mixed reality space MR including the object J with a 360-degree image can be displayed on the display unit 14 of the user terminal 12 (more specifically, the image integration terminal 12B). Then, the MR user can talk with other users in the displayed mixed reality space MR, and can grasp the situation of the target point with the 360-degree image. This can further increase the utility value of the mixed reality space MR.

[0088] <<Other embodiments>> Although one embodiment of the communication support system and its usage method of the present invention has been described above, the above embodiment is merely an example for facilitating understanding of the present invention and does not limit the present invention. In other words, the present invention may be modified or improved without departing from the spirit of the present invention. Furthermore, the present invention naturally includes its equivalents.

[0089] Regarding another embodiment of the present invention, for example, a third server (not shown) may be interposed between the first server 22 and the second server 24 and the user terminal 12. In other words, the object J provided by the first server 22 and the 360-degree video P provided by the second server 24 may be temporarily stored in the third server and then transmitted to the user terminal 12 via the third server.

[0090] In the above embodiment, the user terminal 12 (video transmission terminal 12A) that transmits the 360-degree video P captured by the 360-degree camera 20 installed at the target point and the user terminal (target point terminal 12C) that transmits the voice information of the user at the target point are separate terminals. However, this is not limited to the above, and the user terminal 12 corresponding to the video transmission terminal 12A may have a function to transmit the voice information of the user at the target point. Alternatively, the user terminal 12 corresponding to the target point terminal 12C may be equipped with a 360-degree camera 20 and have a function to acquire and transmit the 360-degree video P.

[0091] In the above embodiment, the 360-degree video P is displayed in a display area corresponding to the outer surface of the spherical object Je in the mixed reality space MR. However, the display form of the 360-degree video P in the mixed reality space MR is not particularly limited, and for example, the 360-degree video P may be divided into a plurality of fragmentary images (e.g., image pieces for different orientations), and each fragmentary image may be displayed in a rectangular display area. [Explanation of symbols]

[0092] 10 CS System (Communication Support System) 12 User terminal 12A Video transmission terminal 12B Video integrated terminal 12C Target location terminal (user terminal of a user at a target location) 14 Display section 16. Mike 18 Speaker 20 360 degree camera 22 First Server 24 Second Server 26 Storage device J Object Avatar Jb drawing object Jc Text Object Jd pen object Je spherical object Jf Capture Image Jg Instruction Object Jh Selection object Ji Photo MR Mixed Reality Space N Communication Network P 360 degree video

Claims

1. A plurality of user terminals; a first server that provides the user terminal with an object for a mixed reality space that is superimposed on a real space in which a user of the user terminal is located and that can be displayed through a display unit of the user terminal; A second server that provides the user terminal with a 360-degree image captured by a 360-degree camera disposed at a target point, The plurality of user terminals include a video transmission terminal that captures the 360-degree video using the 360-degree camera and transmits the 360-degree video to the second server; a video integration terminal that integrates the object provided by the first server with the 360-degree video provided by the second server and displays the mixed reality space including the object with the 360-degree video on the display unit.

2. the video integration terminal has a function of transmitting information of audio collected by the video integration terminal to the first server; The user terminal of the user at the target location has a function of transmitting information of voice collected by the user terminal to the second server, When an operation for starting a call is performed on the video integrated terminal displaying the mixed reality space on the display unit, The second server transmits information on the voice collected by the user terminal of the user at the target location to the video integration terminal on which the operation for starting the call has been performed, 2. The communication support system according to claim 1, wherein the first server transmits information on the voice collected by the video integration terminal to the user terminal of the user located at the target location.

3. The mode in which the second server provides the 360-degree video to the user terminal is a first providing mode in which the second server provides the current 360-degree video most recently received from the video transmitting terminal; A second providing mode for providing the 360-degree images of the past stored in a storage device; 2. The communication support system according to claim 1, wherein the mode is determined according to a selection operation performed on the video integration terminal from among the first provision mode and the second provision mode.

4. The plurality of user terminals include two or more of the video integration terminals; the first server provides the object including an avatar corresponding to each user of the video integrated terminal; 2. The communication support system according to claim 1, wherein each of the video integration terminals, when displaying the mixed reality space including the object with the 360-degree image on the display unit, displays the mixed reality space in which the avatar generated for each of the video integration terminals is positioned.

5. 5. The communication support system of claim 4, wherein when a user operation is performed on the 360-degree image in the mixed reality space at any of the image integration terminals, each of the image integration terminals expands a display area of ​​the 360-degree image on the display unit, and positions the avatar generated for each of the image integration terminals within the enlarged display area.

6. the display unit of the video integration terminal is configured by a head mounted display, The communication support system according to claim 4 , wherein the first server provides the object including an avatar that does not have a part corresponding to the head mounted display as the avatar corresponding to the user wearing the head mounted display.

7. The first server Providing the objects including a pen-type object to the video integrated terminal; 2. The communication support system of claim 1, wherein when an operation of moving the pen-shaped object in the mixed reality space is performed on the video integration terminal while the mixed reality space including the object with the 360-degree image is being displayed on the display unit of the video integration terminal, the object further including a drawing object based on a movement trajectory of the pen-shaped object is provided to the video integration terminal.

8. the video integration terminal has a function of transmitting information on audio collected by the video integration terminal; The communication support system according to claim 1 , wherein the first server provides the objects including a text object indicating text based on a voice collected by the video integration terminal.

9. the text object has a frame and causes the text to be displayed inside the frame; The communication support system according to claim 8 , wherein the video integration terminal causes the display unit to display the mixed reality space in a state in which the text object is arranged at a display position set in the mixed reality space.

10. a storage device that stores elements of the object included in the mixed reality space displayed on the display unit of the video integration terminal together with display positions of the elements in the mixed reality space; 2. The communication support system according to claim 1, wherein when the first server provides the object including the element stored in the storage device, the video integration terminal causes the display unit to display the mixed reality space with the element arranged at the display position stored in the storage device.

11. the video integration terminal has a function of acquiring a capture image of the mixed reality space displayed on the display unit, 2. The communication support system according to claim 1, wherein the first server receives the captured image from the video integration terminal that acquired the captured image, stores the captured image in a storage device, and transmits the captured image stored in the storage device to the user terminal.

12. the first server providing the object in three dimensions; the object includes a spherical object; The communication support system according to claim 1 , wherein the image integration terminal displays the 360-degree image in an area corresponding to an outer surface of the spherical object in the mixed reality space.

Images