Communication device, communication system, display method, and program

Abstract

To solve the problem in that since communication in a physical space is carried out by three or more people, or users sit in different positions depending on the purpose of communication such as meetings or lectures, it is difficult for the users of a plurality of bases to feel like they are in the same room corresponding to the number of bases and the purpose of communication with conventional techniques.SOLUTION: A communication device 1a displays an image based on each piece of image data transmitted by other communication devices 1b, 1c, and 1d in a predetermined area on a display 4 (see Figs. 35(b), 36(b), 38( b), and 39(b)), based on a positional relationship determined according to the number of bases that perform video calls, that is, a position of a person (user D1) in a virtual space (world coordinate system) on its own device (communication device 1d) side and changeable positional relationships (Figs. 12 to 14) between a plurality of communication devices in the virtual space (world coordinate system).SELECTED DRAWING: Figure 35

Description

Technical Field

[0001] The present disclosure relates to a communication device, a communication system, a display method, and a program.

Background Art

[0002] In recent years, telework that makes effective use of time and place by using ICT (Information and Communication Technology) has been spreading. By doing so, it is possible to achieve work-life balance, improve business efficiency, and improve customer satisfaction. In addition, along with the spread of telework, the performance of systems such as video (TV) conferencing via the Internet has been improving (see FIG. 39).

[0003] In communication, the sense of distance between people is very important because it affects ease of conversation and tension (Edward Hall, Hidden Dimension, 2000, Misuzu Shobo). However, in video conferencing, there is a problem that it is difficult to obtain a feeling of being in the same room as the conversation partner (hereinafter referred to as "same-room feeling") like in communication in the real space (or physical space).

[0004] Therefore, in Patent Document 1, a technique is disclosed that realizes motion parallax that changes the video of the conversation partner at another base according to the viewpoint (eye position) of a person in the physical space of one base, and displays the conversation partner on a display on the side of one base with a person size corresponding to the distance in the virtual space, thereby giving each user a sense of being in the same room as the conversation partner.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, physical communication often involves three or more people, and the location of users varies depending on the number of participants in meetings, lectures, etc. Therefore, the conventional technology disclosed in Patent Document 1 presents the problem that it is difficult for users at multiple locations to obtain a sense of being in the same room in proportion to the number of locations (number of participants). [Means for solving the problem]

[0007] The invention according to claim 1 is a desk-type communication device equipped with display means for making video calls with other communication devices at one or more other locations, comprising: one or more shooting means for acquiring brightness image and depth image data of the local location by photographing a subject at the local location; a transmitting means for transmitting the brightness image and depth image data of the local location to the other communication devices at one or more other locations; a receiving means for receiving the brightness image and depth image data of one or more other locations obtained by the other communication devices at one or more other locations photographing a subject at the local location from the other communication devices at one or more other locations; and a virtual space based on the brightness image and depth image data of the local location. The communication device is characterized by comprising: a generation means that generates a three-dimensional object of a subject at its own location and generates a three-dimensional object of a subject at the one or more other locations in the virtual space based on the brightness image and depth image data of the one or more other locations; and a display control means that positions the three-dimensional object of a subject at its own location and the three-dimensional object of a subject at the one or more other locations in the virtual space at locations corresponding to the number of locations conducting the video call, and displays an image of the three-dimensional object of a subject at the other locations on the display means according to the positional relationship between the three-dimensional object of a subject at its own location and the three-dimensional object of a subject at the one or more other locations in the virtual space. [Effects of the Invention]

[0008] As explained above, the present invention has the effect of resolving, as much as possible, the problem that it is difficult for users of multiple locations to obtain a sense of being in the same room in proportion to the number of locations. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of a communication system according to an embodiment of the present invention. [Figure 2] This is an electrical hardware configuration diagram of a communication device. [Figure 3] This is an electrical hardware configuration diagram of the communication management server. [Figure 4] This is an electrical hardware configuration diagram of a 360-degree imaging device. [Figure 5] This is an electrical hardware configuration diagram of a smartphone. [Figure 6] This is a functional block diagram of a communication system. [Figure 7] This is a functional block diagram of a communication system. [Figure 8] This is a functional block diagram of a communication system. [Figure 9] This is a conceptual diagram of the image type management table. [Figure 10] This is a conceptual diagram of the placement management table. [Figure 11] This is a conceptual diagram showing the placement of objects in the world coordinate system, illustrating the virtual space when two people are having a video call. [Figure 12] This is a conceptual diagram showing the placement of three people in a world coordinate system, illustrating the virtual space used when three people are having a video call. [Figure 13] This is a conceptual diagram showing the placement of people in a world coordinate system, illustrating the virtual space when four people are having a video call. [Figure 14] (a) is a conceptual diagram of the communication management table, (b) is a conceptual diagram of the session management table, and (c) is a conceptual diagram of the image type management table. [Figure 15] This is a conceptual diagram of the display management table. [Figure 16] This is a conceptual diagram that visually represents the data managed by the display management table as points on the display. [Figure 17] This is a conceptual diagram of a virtual space information management table. [Figure 18] This is a conceptual diagram of the imaging equipment management table. [Figure 19] This is a sequence diagram showing the participation process in a specific virtual space. [Figure 20] (a) is an example display of a login screen on a communication device, and (b) is an example display of a login screen on a smartphone. [Figure 21] This is a diagram showing a participant number selection screen for selecting the number of people attending a video call. [Figure 22] This is a role selection screen. [Figure 23] This is a sequence diagram showing the management process of image type information for a communication device. [Figure 24] This is a sequence diagram showing the management process of image type information for a smartphone. [Figure 25] This is a sequence diagram showing the process until the luminance image data and audio data transmitted from the omnidirectional imaging device reach each communication device. [Figure 26] This is a sequence diagram showing the process until the luminance image data, depth image data, and audio data transmitted from a predetermined communication device reach other communication devices. [Figure 27] This is a conceptual diagram of the transmitted luminance image data and depth image data. [Figure 28] This is a sequence diagram showing the process of receiving image data and audio data by communication device 1d. [Figure 29] This is a flowchart showing the process of image display. [Figure 30] This is a flowchart showing the calculation process of the position of an object in the world coordinate system. [Figure 31] This is a conceptual diagram showing the three-dimensionalization by conversion from the physical space (camera coordinate system) to the first virtual space (modeling coordinate system), and the conversion from the first virtual space (modeling coordinate system) to the second virtual space (world coordinate system). [Figure 32] This is a conceptual diagram showing the arrangement of the image of user D1 and the virtual display in the modeling coordinate system. [Figure 33] This is an image diagram showing the conversion from the modeling coordinate system to the world coordinate system. [Figure 34] (a) is a diagram showing the line of sight of user D1 in the world coordinate system when four people are making a video call, and (b) is an example of the image that communication device 1d shows to user D1 in the case of (a). [Figure 35] (a) is a diagram showing the line of sight of user D1 after movement in the world coordinate system when four people are making a video call, and (b) is an example of the image that communication device 1d shows to user D1 in the case of (a). [Figure 36] (a) is a diagram showing the viewpoint of an observer in the world coordinate system when four people are having a video call, and (b) is an example of the image displayed on a smartphone in the case of (a). [Figure 37] (a) is a diagram showing the line of sight of user D1 in the world coordinate system when two people are making a video call, and (b) is an example of the image that the communication device 1d shows to user D1 in the case of (a). [Figure 38] (a) is a diagram showing the line of sight of user D1 in the world coordinate system when three people are making a video call, and (b) is an example of the image that communication device 1d shows to user D1 in the case of (a). [Figure 39] This is an example of how a video conference would look on a display. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings.

[0011] [Outline of the communication system] The communication system of this embodiment is a system for making video calls in a virtual space (also called a "virtual area").

[0012] First, the general configuration of the communication system of this embodiment will be explained using Figure 1. Figure 1 is a schematic diagram of the configuration of the communication system according to an embodiment of the present invention.

[0013] As shown in Figure 1, the communication system of this embodiment is constructed using a plurality of communication devices 1a, 1b, 1c, 1d, a communication management server 6, a 360-degree imaging device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection method of the communication network 100 may be either wireless or wired.

[0014] Communication devices 1a, 1b, 1c, and 1d are desk-type communication devices located at bases A, B, C, and D, respectively. Hereafter, any of the communication devices 1a, 1b, 1c, and 1d will be referred to as "communication device 1". Communication device 1 can take images and collect sound at its own base and output images obtained from images taken at other bases and sounds obtained from sound collected at other bases. Communication device 1 will be described in detail later.

[0015] Figure 1 also shows the case where communication devices 1a, 1b, 1c, and 1d are used by users A1, B1, C1, and D1, respectively. For example, communication device 1a takes pictures and collects sound of its own location A, and transmits image data and sound data to communication devices 1b, 1c, and 1d at other locations B, C, D, and G, as well as to a smartphone 9, via the communication network 100 and the communication management server 6. Similarly, communication devices b1, 1c, and 1d take pictures and collect sound of their own locations, and transmit image data and sound data to communication devices at other locations via the communication network 100 and the communication management server 6. The smartphone 9 will be described later.

[0016] The communication management server 6 is a computer that manages and controls communication between the communication device 1 and the smartphone 9, and manages the types of image data transmitted and received (general images and special images). Therefore, the communication management server is also a communication control server. The communication management server 6 is composed of one or more computers. If it is composed of multiple computers, the communication management server is also a communication management system (or communication control system).

[0017] The 360-degree camera 8 and smartphone 9 are being used by user G at base G. User G1 takes the 360-degree camera 8 at base G to photograph and record sound of base G, and transmits the image data and sound data to each communication device 1a, 1b, 1c, and 1d via smartphone 9, communication network 100, and communication management server 6.

[0018] The 360-degree camera 8 is a special digital camera used to capture subjects, landscapes, etc., and obtain two hemispherical images that form the basis of a 360-degree panoramic image. In this case, the 360-degree camera 8 cannot connect to the communication network 100 on its own, so it first transmits image data and sound data to the smartphone 9 using wireless communication technology such as Wi-Fi (Wireless Fidelity) or Bluetooth (registered trademark). The smartphone 9 then transmits the image data and sound data to other devices and servers such as the communication management server 6 via the communication network 100. Note that the 360-degree camera 8 may also be a device that can connect to the communication network 100 on its own.

[0019] As described above, the smartphone 9 acquires image data and sound data from the 360-degree camera 8 and transmits the image data and sound data to the communication management server 6 and the communication device 1 via the communication network 100. Furthermore, the smartphone 9 plays back the image data and sound data acquired from the communication device 1 at another location.

[0020] Furthermore, OpenGL ES is installed on the communication device 1 and the smartphone 9, enabling them to create predetermined region information that indicates a portion of a 360-degree image, or to create predetermined region images that are a portion of a 360-degree image from a 360-degree image sent from another communication device.

[0021] Furthermore, communication device 1 also includes digital televisions, smartphones (a separate device from smartphone 9), smartwatches, car navigation systems, medical devices, etc.

[0022] [Hardware configuration] Next, the hardware configurations of the communication device 1, communication management server 6, 360-degree imaging device 8, and smartphone 9 of this embodiment will be described in detail with reference to Figures 2 to 5.

[0023] <Hardware configuration of communication equipment> First, we will explain the electrical hardware configuration of communication device 1 using Figure 2. Figure 2 is an electrical hardware configuration diagram of the communication device.

[0024] The communication device 1 consists of a PC 3, a display 4, a brightness depth imaging device 5, a speaker 11, a keyboard 12, and a mouse 13.

[0025] (PC hardware configuration) As a computer, PC3 is equipped with a CPU 301, ROM 302, RAM 303, HD 304, HDD (Hard Disk Drive) controller 305, external device connection I / F (Interface) 308, network I / F 309, bus line 310, pointing device 312, and media I / F 314, as shown in Figure 2.

[0026] Of these, the CPU 301 controls the operation of the entire PC3. The ROM 302 stores programs used to drive the CPU 301, such as the IPL. The RAM 303 is used as the work area for the CPU 301. The HD 304 stores various data such as programs. The HDD controller 305 controls the reading or writing of various data to the HD 304 according to the control of the CPU 301. The external device connection I / F 308 is an interface for connecting various external devices. In this case, the external devices are the display 4, brightness depth imaging device 5, speaker 11, keyboard 12, and mouse 13. Furthermore, other external devices include, for example, USB (Universal Serial Bus) memory and printers.

[0027] Network I / F 309 is an interface for data communication using the communication network 100. Bus line 310 is an address bus, data bus, etc., for electrically connecting the various components such as the CPU 301 shown in Figure 2.

[0028] The pointing device 312 is a type of input means used for selecting and executing various instructions, selecting processing targets, and moving the cursor. Note that the function of the pointing device 312 may be turned OFF when the user uses the keyboard 12. The media I / F 314 controls the reading or writing (storage) of data to or from the recording medium 313, such as flash memory. The recording medium 313 includes DVDs and Blu-ray Discs.

[0029] (Hardware configuration of brightness depth imaging device) The brightness depth imaging device 5 comprises a brightness camera 501, a depth camera 502, a microphone 505, an operation button 506, an external device connection I / F 508, and a bus line 510. Of these, the brightness camera 501 is a device that uses an image sensor to convert received light into an electrical signal to obtain image data. For example, an RGB color camera is included.

[0030] The depth camera 502 acquires depth image data by capturing images of subjects such as people and objects, and measuring the distance from the subject to the depth camera 502 using the Time of Flight (TOF) method. The depth camera 502 is a camera that can capture the shape of a subject in three dimensions and detect various movements such as a person's skeleton, and reflect this in real time in subsequent processing. For this reason, the depth camera 502 is composed of an infrared projector 503 and an infrared sensor 504. The infrared projector 503 is a projector capable of emitting infrared light. The infrared sensor 504 is a device that receives the infrared light reflected by the infrared projector 503 when it is reflected by the subject, converts it into an electrical signal, and extracts the necessary information. Although the above describes a method of measuring depth using the TOF method, depth may also be measured using a pattern illumination method. Furthermore, depth may be measured using a stereo measurement method with a standard RGB color camera, and the method of measuring depth is not limited to those described above.

[0031] The microphone 505 collects ambient sound and converts it into sound (electrical signal) data. The operation button 506 is used to operate the brightness depth imaging device 5 and turn its power ON / OFF. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is the PC3. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component, such as the brightness camera 501 shown in Figure 3.

[0032] Note that the display 4, speaker 11, keyboard 12, and mouse 13 are general-purpose devices, so their descriptions will be omitted.

[0033] <Hardware configuration of the communication management server> Next, we will explain the electrical hardware configuration diagram of the communication management server using Figure 3. Figure 3 is the electrical hardware configuration diagram of the communication management server.

[0034] As shown in Figure 3, the communication management server 6, as a computer, is equipped with a CPU 601, ROM 602, RAM 603, HD 604, HDD (Hard Disk Drive) controller 605, display 606, external device connection I / F (Interface) 608, network I / F 609, bus line 610, pointing device 612, and media I / F 614.

[0035] Of these, the CPU 601 controls the operation of the entire communication management server 6. The ROM 602 stores programs used to drive the CPU 601, such as the IPL. The RAM 603 is used as the work area for the CPU 601. The HD 604 stores various data such as programs. The HDD controller 605 controls the reading or writing of various data to the HD 604 according to the control of the CPU 601. The display 606 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 608 is an interface for connecting various external devices. In this case, external devices include, for example, USB memory or printers. The network I / F 609 is an interface for data communication using the communication network 100. The bus line 610 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 601 shown in Figure 3.

[0036] Furthermore, the pointing device 612 is a type of input means used for selecting and executing various instructions, selecting processing targets, and moving the cursor. The media interface 614 controls the reading or writing (storage) of data to or from the recording medium 613, such as flash memory. The recording medium 613 includes DVDs and Blu-ray Discs.

[0037] <Hardware configuration of the 360-degree imaging system> Next, we will explain the electrical hardware configuration diagram of the 360-degree imaging device using Figure 4. Figure 4 is an electrical hardware configuration diagram of the 360-degree imaging device.

[0038] The 360-degree imaging device 8 is a 360-degree (omnidirectional) 360-degree imaging device using two image sensors, but it can use two or more image sensors. Furthermore, it does not necessarily have to be a device dedicated to omnidirectional imaging; a standard digital camera or smartphone can be fitted with an aftermarket omnidirectional imaging unit to achieve essentially the same functionality as the 360-degree imaging device 8.

[0039] As shown in Figure 4, the 360-degree imaging device 8 consists of an imaging unit 801, an image processing unit 804, an imaging control unit 805, a microphone 808, a sound processing unit 809, a CPU (Central Processing Unit) 811, a ROM (Read Only Memory) 812, an SRAM (Static Random Access Memory) 813, a DRAM (Dynamic Random Access Memory) 814, an operation unit 815, an external device connection interface 816, a communication circuit 817, an antenna 817a, an acceleration / direction sensor 818, and a recessed terminal 819 for Micro USB.

[0040] Of these, the imaging unit 801 includes wide-angle lenses (so-called fisheye lenses) 802a and 802b, each having a field of view of 180° or more for forming a hemispherical image, and two image sensors 803a and 803b, each corresponding to the wide-angle lens. The image sensors 803a and 803b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the wide-angle lenses 802a and 802b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the image sensors are set.

[0041] The image sensors 803a and 803b of the imaging unit 801 are each connected to the image processing unit 804 via a parallel I / F bus. On the other hand, the image sensors 803a and 803b of the imaging unit 801 are connected to the imaging control unit 805 via a serial I / F bus (such as an I2C bus). The image processing unit 804, the imaging control unit 805, and the sound processing unit 809 are connected to the CPU 811 via bus 810. Furthermore, ROM 812, SRAM 813, DRAM 814, operation unit 815, external device connection I / F (Interface) 816, communication circuit 817, and acceleration / direction sensor 818 are also connected to bus 810.

[0042] The image processing unit 804 receives image data output from image sensors 803a and 803b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create an equirectangular projection image. An equirectangular projection image is an image obtained by transforming a round sphere into a planar image, and in English, This image uses a projection method called equirectangular.

[0043] The imaging control unit 805 generally uses the I2C bus to set commands and other information in the registers of the image sensors 803a and 803b, with the 805 acting as the master device and the image sensors 803a and 803b as slave devices. It receives necessary commands and other information from the CPU 811. The imaging control unit 805 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 803a and 803b and send it to the CPU 811.

[0044] Furthermore, the imaging control unit 805 instructs the image sensors 803a and 803b to output image data when the shutter button on the operation unit 815 is pressed. Some 360-degree imaging devices 8 may also have functions to support preview display on a display (for example, a smartphone display) or video display. In this case, the image data output from the image sensors 803a and 803b is performed continuously at a predetermined frame rate (frames / minute).

[0045] Furthermore, as will be described later, the imaging control unit 805 also functions as a synchronization control means that works in cooperation with the CPU 811 to synchronize the output timing of image data from the image sensors 803a and 803b. In this embodiment, the 360-degree imaging device 8 is not provided with a display, but a display unit may be provided.

[0046] Microphone 808 converts sound into sound (signal) data. Sound processing unit 809 receives the sound data output from microphone 808 through the I / F bus and performs predetermined processing on the sound data.

[0047] The CPU 811 controls the overall operation of the 360-degree imaging device 8 and performs necessary processing. The ROM 812 stores various programs for the CPU 811. The SRAM 813 and DRAM 814 are work memories that store programs executed by the CPU 811 and data in progress. In particular, the DRAM 814 stores image data in progress of processing by the image processing unit 804 and data of completed equirectangular projection images.

[0048] The control unit 815 is a collective term for the operation buttons, such as the shutter button 815a. The user inputs various shooting modes and shooting conditions by operating the control unit 815.

[0049] The external device connection interface 816 is an interface for connecting various external devices. These external devices include, for example, USB (Universal Serial Bus) memory and PCs (Personal Computers). The equirectangular projection image data stored in the DRAM 814 can be recorded to an external recording medium via this network interface 816, or transmitted to an external terminal (device) such as a smartphone via the network interface 816 as needed.

[0050] The communication circuit 817 communicates with an external terminal (device) such as a smartphone via an antenna 817a provided on the 360-degree imaging device 8, using short-range wireless communication technologies such as Wi-Fi, NFC (Near Field Communication), or Bluetooth (registered trademark). This communication circuit 817 can also transmit equirectangular projection image data to an external terminal (device) such as a smartphone.

[0051] The acceleration / direction sensor 818 calculates the orientation of the 360-degree camera 8 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) in accordance with Exif and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was taken and the data size of the image data. The acceleration / direction sensor 818 is also a sensor that detects changes in angle (Roll angle, Pitch angle, Yaw angle) associated with the movement of the 360-degree camera 8. Changes in angle are an example of related information (metadata) in accordance with Exif and are used for image processing such as image correction of captured images. Furthermore, the acceleration / direction sensor 818 is a sensor that detects acceleration in three axes. The 360-degree camera 8 calculates its own orientation (angle relative to the direction of gravity) based on the acceleration detected by the acceleration / direction sensor 818. The installation of the acceleration / direction sensor 818 in the 360-degree camera 8 improves the accuracy of image correction.

[0052] The recessed terminal 819 for Micro USB is a terminal for connecting a USB cable or similar when communicating with the smartphone 9 shown in Figure 1 via a wired connection rather than wirelessly using Wi-Fi or similar methods.

[0053] <Smartphone Hardware Configuration> Next, we will explain the electrical hardware configuration diagram of a smartphone using Figure 5. Figure 5 is an electrical hardware configuration diagram of a smartphone.

[0054] As shown in Figure 5, the smartphone 9 includes a CPU 901, ROM 902, RAM 903, EEPROM 904, CMOS sensor 905, image sensor I / F 906, accelerometer / compass sensor 907, media I / F 909, and GPS receiver 911.

[0055] Of these, the CPU 901 controls the overall operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901, such as the CPU 901 and IPL. The RAM 903 is used as the work area for the CPU 901. The EEPROM 904 reads or writes various data, such as smartphone programs, according to the control of the CPU 901. The CMOS (Complementary Metal Oxide Semiconductor) sensor 905 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) according to the control of the CPU 901 to obtain image data. Note that it may be an imaging means such as a CCD (Charge Coupled Device) sensor instead of a CMOS sensor. The image sensor I / F 906 is a circuit that controls the driving of the CMOS sensor 905. The acceleration / direction sensor 907 is a type of sensor such as an electronic magnetic compass, gyrocompass, or acceleration sensor that detects the Earth's magnetic field. The media I / F 909 controls the reading or writing (storage) of data to or from the recording medium 908, such as flash memory. The GPS receiver 911 receives GPS signals from GPS satellites.

[0056] The smartphone 9 also includes a long-range communication circuit 912, a CMOS sensor 913, an image sensor interface 914, a microphone 915, a speaker 916, an audio input / output interface 917, a display 918, an external device connection interface 919, a short-range communication circuit 420, an antenna 420a for the short-range communication circuit 420, and a touch panel 421.

[0057] Of these, the long-distance communication circuit 912 is a circuit that communicates with other devices via the communication network 100. The CMOS sensor 913 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 901. The image sensor interface 914 is a circuit that controls the driving of the CMOS sensor 913. The microphone 915 is a built-in circuit that converts sound into electrical signals. The speaker 916 is a built-in circuit that converts electrical signals into physical vibrations to produce sound such as music and speech. The sound input / output interface 917 is a circuit that processes the input and output of sound signals between the microphone 915 and the speaker 916 according to the control of the CPU 901. The display 918 is a type of display means such as liquid crystal or organic EL (Electro Luminescence) that displays images of subjects and various icons. The external device connection interface 919 is an interface for connecting various external devices. The short-distance communication circuit 420 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 421 is a type of input method that allows the user to operate the smartphone 9 by pressing the display 918.

[0058] Furthermore, the smartphone 9 is equipped with a bus line 910. The bus line 910 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 901 shown in Figure 6.

[0059] Furthermore, recording media such as CD-ROMs on which the above programs are stored, as well as hard drives on which these programs are stored, may be provided domestically or internationally as program products.

[0060] [Functional configuration of the embodiment] Next, the functional configuration of this embodiment will be described using Figures 6 to 18. Figures 6 to 8 are functional block diagrams of the communication system.

[0061] <Functional Configuration of Communication Devices> Here, we will describe the functional configuration of communication device 1a, which is part of communication device 1. The functions of communication device 1a are essentially realized by PC3a and brightness depth imaging device 5.

[0062] As shown in Figure 6, the communication device 1a has a transmitting / receiving unit 31a, a receiving unit 32a, an image / sound processing unit 33a, a display control unit 34a, a judgment unit 35a, a sound collection unit 38a, and a storage / reading processing unit 39a, all of which are functions realized by the PC 3a. Each of these units is a function or means realized by any of the components shown in Figure 2 operating according to instructions from the CPU 301 that follow a program for the communication device deployed from the HD 304 onto the RAM 303. The communication device 1a also has a storage unit 3000a constructed from the ROM 302, RAM 303, and HD 304 shown in Figure 3.

[0063] Furthermore, the communication device 1a includes a brightness image acquisition unit 36a, a depth image acquisition unit 37a, and a sound collection unit 38a, as functions realized by the brightness depth imaging device 5. The functions realized by the brightness depth imaging device 5 are enclosed in dashed lines.

[0064] (Image type management table) Figure 9 is a conceptual diagram showing the image type management table. The storage unit 3000a has an image type management DB 3003a constructed, which is composed of the image type management table. In this image type management table, image data ID (Identification), source connection ID, and image type information are stored and managed in association. Of these, the image data ID is an example of image data identification information used to identify image data when performing video communication. Image data transmitted from the same source is assigned the same image data ID. This allows the destination (receiving communication device) to identify the source of the received image data. The source connection ID is a user ID used to identify the communication device or the user of the communication device. The user ID is an identifier that can be used within the authority granted by the communication management server 6. Note that the connection ID may also be a device ID used to identify the communication device 1a. The connection ID is an example of connection identification information, the user ID is an example of user identification information, and the device ID is an example of device identification information.

[0065] Image type information is a name used to identify the image related to the image data indicated by the associated image data ID, and is, for example, the source name. Image type information is created by PC3a according to a predetermined naming convention. Here, "Video_Depth", "Video_Omni", and "Video" are shown as image type information. These indicate, respectively, that the image type is a video consisting of a "luminance image including a depth image", a video consisting of a "luminance image that is a 360-degree image", and a video consisting of a "luminance image that is a planar image". Note that a 360-degree image is an example of a special image, and a planar image is an example of a general image.

[0066] For example, communication device 1a with connection ID "01aa" is shown to be transmitting image data indicated by image data ID "RS001". Furthermore, the image type indicated by the image type information is "Video_Depth".

[0067] Furthermore, data other than image data may also be managed in association with image data IDs. Examples of data other than image data include audio data and document data used during screen sharing.

[0068] (Distribution management table) Figure 10 is a conceptual diagram showing the placement management table. The storage unit 3000a contains a placement management DB 3006a, which is composed of the placement management table. In this placement management table, the virtual space type ID, number of participants, role, conversion parameters from the modeling coordinate system to the world coordinate system, and connection ID are stored and managed in association with each other.

[0069] Of these, the virtual space type ID is an example of virtual space type classification information used to identify the type of virtual space used for video calls. The number of participants indicates the number of people participating in the video call. Based on this number of participants, their positions in the virtual space are predetermined. In Figure 10, the number of participants is shown as 2 to 4, but it is not limited to these numbers. For example, a single-person room for one person or a virtual room for 5 or more people can be selected. Virtual rooms also include virtual booths set up in a part of the virtual room. Indoor or outdoor (outdoor) virtual event venues are also included. An event indicates, for example, that it will be held at a predetermined place and time, or that it will be specially planned and held.

[0070] Roles are defined according to the type of virtual space and are the actions or activities assigned, required, or expected of an individual (or group). In addition to those shown in Figure 10, roles may also include those of a customer service guide, a customer service representative, a salesperson in a business negotiation, a customer in a business negotiation, etc. Roles may also include positions such as president or department head.

[0071] The transformation parameters from the modeling coordinate system to the world coordinate system are the transformation parameters used when transforming an object from the modeling coordinate system in the first virtual space to the world coordinate system in the second virtual space, as shown in Figure 33 below. The connection ID is the same concept as the connection ID in the image type management table, so its explanation is omitted.

[0072] Furthermore, virtual worlds depicted using computer graphics (CG) have a single coordinate system that serves as the basis for the world. This is called the world coordinate system. If the shape of an object is directly given in the world coordinate system, the shape of the object must be redefined every time the object moves within the world coordinate system. Therefore, a coordinate system is defined (called modeling) in which the shape of an object is determined in the 3D image of the computer graphics. This coordinate system is called the modeling coordinate system.

[0073] Here, we will explain world coordinates and modeling coordinates using different terminology. The world coordinate system represents the entire space in which objects are displayed in computer graphics. It is a coordinate system used to indicate the position of objects in space and is used to handle the placement and movement of objects. It can describe the position and orientation of each solid. If all objects were defined from the beginning in one large space called the world coordinate system, it would be difficult to handle the display process. Therefore, modeling coordinates are used to handle the shape and deformation of individual solids placed in space. The origin is taken at one of the vertices or in the vicinity of the solid, and the position and orientation of the points, lines, faces, etc. that make up the solid are described. By placing all display objects, which are freely defined in individual modeling coordinate systems, onto a unified world coordinate system, it is possible to create a graphics space, or a virtual 3D world.

[0074] Here, we will explain the optimal type space for each virtual space using Figures 11 to 13.

[0075] ((Video call between the two)) Figure 11 is a conceptual diagram showing the placement of elements in a world coordinate system, illustrating the virtual space for a video call between two people. Figure 11 shows, for example, the placement of each role in a virtual meeting as a video call. The roles of host, participant (1), and observer in Figure 10 are placed at host position a11, participant position a14, and observer position a15 in Figure 11, respectively. Similarly, the display positions d11, d14, and d15 are placed to correspond to host position a11, participant position a14, and observer position a15, respectively. Note that display positions d11 and d14 represent the positions of the two displays of communication devices 1a and 1d as shown in Figure 1, while display position d15 represents the position of a single display such as a smartphone 9.

[0076] ((Video call with 3 people)) Figure 12 is a conceptual diagram showing the placement of elements in a world coordinate system, illustrating a virtual space for a video call with three participants. Figure 12 shows, for example, the placement of each role in a virtual meeting as a video call. The roles of organizer, participant (1), participant (2), and observer in Figure 10 are placed at organizer position a21, participant position a22, participant position a24, and observer position a25 in Figure 12, respectively. Similarly, the display positions d21, d22, d24, and d25 correspond to organizer position a21, participant position a22, participant position a24, and observer position a25, respectively. Note that display positions d21, d22, and d24 represent the positions of the three displays of communication devices 1a, 1b, and 1d as shown in Figure 1, while display position d25 represents the position of a single display such as a smartphone 9.

[0077] ((Video call with 4 people)) Figure 13 is a conceptual diagram showing the placement of elements in a world coordinate system, illustrating a virtual space for a four-person meeting. Figure 13 shows, for example, the placement of each role in a virtual meeting as a video call. The roles of organizer, participant (1), participant (2), participant (3), and observer in Figure 10 are placed at organizer position a11, participant position a12, participant position a13, participant position a14, and observer position a15 in Figure 13, respectively. Similarly, the display positions d11 to d15 are placed to correspond to organizer position a11, participant position a12, participant position a13, participant position a14, and observer position a15, respectively. Note that display positions d11 to d14 represent the positions of the four displays of communication devices 1a to 1d as shown in Figure 1, while display position d15 represents the position of a single display such as a smartphone 9.

[0078] In Figures 10 to 13, there are two to five users and positions in the virtual space, but this is not limited to these; there can be any number of users. Similarly, there can be multiple observer positions, or multiple observers can enter a virtual conference room and be positioned at the location of a single observer. Furthermore, examples of video calls include not only conferences but also interviews, lectures, etc., which are also examples of events.

[0079] (Functional configuration of each communication device) Next, we will explain in more detail the functional configurations of the communication device 1a using Figures 2 and 6.

[0080] The transmitting / receiving unit 31a of the communication device 1a is realized by the processing of the CPU 301 to the network I / F 309, and transmits and receives various data (or information) with other devices (for example, the communication management server 6) via the communication network 100.

[0081] The reception unit 32a is mainly implemented by the processing of the keyboard 12, mouse 13, and pointing device 312 by the CPU 301, and accepts various selections or inputs from the user.

[0082] The image and sound processing unit 33a is mainly implemented by the CPU 301 and performs various image and sound processing. For example, the image and sound processing unit 33a acts as a generation unit and generates a 3D object of the subject in virtual space based on luminance image and depth image data.

[0083] The display control unit 34a is mainly implemented by the processing of the CPU 301 and displays various images on each of the displays 4a to 4c.

[0084] The decision unit 35a is mainly implemented by the processing of the CPU 301 and performs various decisions.

[0085] The luminance image acquisition unit 36a is mainly implemented by the processing of the luminance camera 501 by the CPU 301, and acquires luminance image data by photographing the subject. Here, it is referred to as a luminance image, but this term is used for the purpose of contrasting it with depth images, and a luminance image is not an image that contains only luminance information, but also color information.

[0086] The depth image acquisition unit 37a is mainly implemented by the processing of the depth camera 502 by the CPU 301, and acquires depth image data by photographing the subject.

[0087] The sound collection unit 38a is mainly implemented by the processing of the microphone 505 by the CPU 301, and collects sound to acquire sound data.

[0088] The memory and read processing unit 39a is mainly implemented by the CPU 301 and stores various data (or information) in the memory unit 3000a and reads various data (or information) from the memory unit 3000a.

[0089] Furthermore, communication devices 1b, 1c, and 1d have the same functions, storage, and database as communication device 1a. The functions of communication devices 1b, 1c, and 1d are essentially implemented by PCs 3b, 3c, and 3d, respectively. In Figures 6 and 7, the only difference is that the last digit of the code changes from 'a' to 'b', 'c', and 'd', respectively, so a detailed explanation of these is omitted.

[0090] <Functional Configuration of the Communication Management Server> Next, the functional configuration of the communication management server 6 will be explained using Figures 3 and 6. Figure 6 is a functional configuration diagram of the communication management server.

[0091] As shown in Figure 6, the communication management server 6 includes a transmitting / receiving unit 61, an image / sound processing unit 63, a determination unit 65, a creation unit 66, and a storage / reading processing unit 69. Each of these units is a function or means realized by any of the components shown in Figure 3 operating according to instructions from the CPU 601 that follow a communication management (control) program deployed from the HD 604 onto the RAM 603. The communication management server 6 also has a storage unit 6000 constructed from the ROM 602, RAM 603, and HD 604 shown in Figure 3.

[0092] (Communication management table) Figure 14(a) is a conceptual diagram showing a communication management table. The storage unit 6000 has a communication management DB 6001 constructed, which is composed of a communication management table as shown in Figure 14(a). In this communication management table, each connection ID of all communication devices 1 managed by the communication management server 6 is associated with and managed with a password and an IP address. For example, in the communication management table shown in Figure 14, it is shown that the connection ID of communication device 1a is "01aa", the password is "aaaa", and the IP address is "1.2.1.3" as IPv4. Note that the IP address is shown in a simplified manner. Also, the IP address may be IPv6 instead of IPv4. Also, instead of an IP address, it may be a domain name (a string of characters that indicates a name used as an email address or homepage address).

[0093] (Session management table) Figure 14(b) is a conceptual diagram showing the session management table. The memory unit 6000 has a session management DB 6002 constructed, which consists of the session management table shown in Figure 14(b). In this session management table, the session ID, virtual space type ID, and connection ID of the communication device that participated in the video call of the same communication session are stored and managed in association with each other. Of these, the session ID is an example of session identification information for identifying the communication session that realizes the video call, and is generated for each virtual space such as a virtual conference room. The session ID may be replaced with a meeting ID. The meeting ID indicates an identifier for managing meetings (conversations between two or more people) on the communication management server. Note that the virtual space type ID and the connection ID of the participating communication device are the same concepts as the virtual space type ID and connection ID managed in the placement management table shown in Figure 10, so their explanation is omitted.

[0094] (Image type management table) The memory unit 6000 has an image type management DB 6003 constructed, which consists of an image type management table as shown in Figure 14(c). This image type management DB 6003 has an additional item (attribute) for session ID (communication ID) compared to the image type management DB 3003a mentioned above. Since each item (attribute) has already been explained, the explanation will be omitted.

[0095] (Display management table) Figure 15 is a conceptual diagram showing the display management table. The memory unit 6000 has a display management DB 6004 constructed, which consists of the display management table shown in Figure 15. This display management table stores and manages the vendor ID and product ID of the display's GUID (Globally Unique Identifier), the vertical and horizontal dimensions (position of each corner) and resolution of the display in physical space (real coordinate system), and the position of each corner of the display in virtual space (modeling coordinate system), all associated with each other. The real coordinate system refers to a coordinate system based on an arbitrary position in the three-dimensional world, which is the real world.

[0096] Here, using Figure 16, we will visually explain the positions indicated by the data managed in the display management table as points on the display. Figure 16 is a conceptual diagram that visually shows the positions indicated by the data managed in the display management table as points on the display.

[0097] For example, if the GUID of the display is "vid_10ca&pid_0001", it indicates the points representing the positions of each of the four corners of display 118 in physical space (real coordinate system). Similarly, if the GUID of the communication device is "vid_10ca&pid_0001", it indicates the points representing the positions of each of the four corners of the three displays in virtual space (modeling coordinate system).

[0098] Furthermore, if the GUID of the display is "vid_11ca&pid_0010", the points indicating the positions of each of the four corners of one of the displays of smartphone 9 in physical space (real coordinate system) are shown. Similarly, if the GUID of the smartphone is "vid_11ca&pid_0010", the points indicating the positions of the four corners of one of the displays of smartphone in virtual space (modeling coordinate system) are shown.

[0099] Furthermore, if the display's GUID is "vid_12ca&pid_0100", it indicates the positions of the four corners of a typical laptop's display in physical space (real coordinate system). Similarly, if the display's GUID is "vid_12ca&pid_0100", it indicates the positions of the four corners of a laptop's display in virtual space (modeling coordinate system).

[0100] (Virtual Space Information Management Table) Figure 17 is a conceptual diagram showing the virtual space information management table. The memory unit 6000 has a virtual space information management DB 6005 constructed, which is composed of the virtual space information management table shown in Figure 17. In this virtual space information management table, the session ID (communication ID), the number of participants in the video call, and the role name are stored and managed in association with each other. The session ID (communication ID) is the same concept as the "Session ID" item (attribute) in the session management DB 6002 described above, so its explanation is omitted. The number of participants is managed using the "Number of Participants" item (attribute) in the placement management DB 3006a described above as the name.

[0101] (Functional configuration of the communication management server) Next, we will explain in more detail the functional configurations of the communication management server 6 using Figures 3 and 6.

[0102] The transmitting / receiving unit 61 of the communication management server 6 is implemented by the processing of the CPU 601 to the network I / F 609, and transmits and receives various data (or information) with other devices (for example, communication device 1, smartphone 9) via the communication network 100.

[0103] The image and sound processing unit 63 is primarily implemented by the CPU 601 and generates 3D objects of subjects at each location in the virtual space based on the luminance image data and depth image data of each location. Therefore, the image and sound processing unit 63 functions as a generation unit. The image and sound processing unit 63 will be described in detail later.

[0104] The decision unit 65 is mainly implemented by the processing of the CPU 601 and performs various decisions.

[0105] The creation unit 66 is mainly implemented by the CPU 601 and creates image data IDs.

[0106] The memory and read processing unit 69 is mainly implemented by the CPU 601's processing of the HDD 605 shown in Figure 3, and stores various data (or information) in the memory unit 5000 and reads various data (or information) from the memory unit 5000.

[0107] <Functional Configuration of a 360° Spherical Imaging System> Next, the functional configuration of the 360-degree imaging device 8 will be explained using Figures 4 and 8.

[0108] As shown in Figure 8, the 360-degree imaging device 8 includes a reception unit 82, an imaging unit 83, a sound collection unit 84, a judgment unit 85, a communication unit 88, and a storage / reading processing unit 89. Each of these units is a function or means realized by any of the components shown in Figure 4 operating according to instructions from the CPU 811 that follow a program for the imaging device deployed on SRAM 813 to DRAM 814.

[0109] Furthermore, the 360-degree imaging device 8 has a memory unit 8000 constructed from ROM 512, SRAM 513, and DRAM 514, as shown in Figure 4. The GUID of the device is stored in the memory unit 8000.

[0110] (Functional configuration of a 360-degree imaging device) Next, we will explain in more detail the functional configurations of the 360-degree imaging device 8 using Figures 4 and 8.

[0111] The reception unit 82 of the 360-degree imaging device 8 is mainly realized by the processing of the CPU 811 to the operation unit 515 shown in Figure 4, and receives operation input from the user.

[0112] The imaging unit 83 is mainly realized by the processing performed by the CPU 811 on the imaging unit 801, image processing unit 804, and imaging control unit 805 shown in Figure 4. It captures images of subjects, landscapes, etc., and obtains data for two hemispherical images that constitute a full-sphere image.

[0113] The sound collection unit 84 is realized by the processing of the microphone 808 and sound processing unit 809 shown in Figure 4 by the CPU 811, and collects sounds from the surroundings of the 360-degree imaging device 8.

[0114] The decision unit 85 is mainly implemented by the processing of the CPU 801 and performs various decisions.

[0115] The communication unit 88 is mainly implemented by the processing of the communication circuit 817 by the CPU 811, and can communicate with the communication unit 98 of the smartphone 9 (described later) using short-range wireless communication technologies such as NFC, Bluetooth, and Wi-Fi.

[0116] The memory and read processing unit 89 is mainly implemented by the CPU 811 shown in Figure 4, and stores various data (or information) in the memory unit 8000 and reads various data (or information) from the memory unit 8000.

[0117] <Smartphone Functionality Configuration> Next, the functional configuration of the smartphone 9 will be described in detail using Figures 5 and 8. As shown in Figure 8, the smartphone 9 has a transmitting / receiving unit 91, a receiving unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a calculation unit 97, a communication unit 98, and a storage / reading processing unit 99. Each of these units is a function or means realized by any of the components shown in Figure 5 operating according to instructions from the CPU 901 that follow a program for the smartphone 9 deployed from the EEPROM 904 onto the RAM 903.

[0118] Furthermore, the smartphone 9 has a storage unit 9000 constructed from ROM 902, RAM 903, and EEPROM 904, as shown in Figure 5.

[0119] (Imaging equipment management table) Figure 18 is a conceptual diagram showing the imaging device management table. The memory unit 8000 has an imaging device management DB 9001 constructed, which is composed of the imaging device management table shown in Figure 18. This imaging device management table stores and manages the vendor ID and product ID of the GUID of the imaging device that can obtain the two hemispherical images that form the basis of the 360-degree spherical image. As GUIDs, for example, the vendor ID (VID) and product ID (PID) used in USB devices can be used. The smartphone 9 obtains the vendor ID and product ID by downloading them from a designated server, etc.

[0120] (Image type management table) The memory unit 9000 has an image type management DB 9003, which has the same data structure as the image type management DB 3003a. The image type management tables that make up the image type management DB 6003 have already been explained in Figure 9, so we will omit their explanation here.

[0121] (Placement management table) The storage unit 9000 has a placement management DB 9006 constructed within it, which has a data structure similar to that of placement management DB 3006a. The placement management tables that make up placement management DB 9006 have already been explained in Figures 10 to 13, so their explanation will be omitted here.

[0122] [Processing or operation of the embodiment] Next, the processing or operation of this embodiment will be described using Figures 19 to 39.

[0123] <Joining a video call> First, the process of joining a specific virtual space will be explained using Figures 19 to 22. Figure 19 is a sequence diagram showing the process of joining a specific virtual space. Figure 20(a) is an example of the login screen displayed on a communication device, and Figure 20(b) is an example of the login screen displayed on a smartphone. Figure 22 is a diagram showing the participant selection screen for selecting the number of people attending a video call. It is a diagram showing the virtual space type selection screen. In Figure 22, (a) is the role selection screen when the virtual space type is a conference type (type of meeting), (b) is the role selection screen when the virtual space type is an interview type, and (c) is the role selection screen when the virtual space type is a lecture type.

[0124] First, a user at site A (for example, user A1) enters their connection ID and password on the login screen displayed on the display 4 by the display control unit 34a (see Figure 20(a)) and presses the "OK" button. The reception unit 32a then receives the connection ID and password, and the transmission / reception unit 31a sends a login request to the communication management server 6 (S21). This login request includes the connection ID, password, and the IP address of the source (communication device 1a). As a result, the transmission / reception unit 61 of the communication management server 6 receives the login request.

[0125] Next, the storage and retrieval processing unit 69 of the communication management server 6 searches the communication management DB 6001 (see Figure 14(a)) using the connection ID and password included in the login request as search keys, and performs authentication by determining whether the same connection ID and password combination is managed (step S22). Here, we will continue the explanation assuming that the same connection ID and password combination is managed by the storage and retrieval processing unit 69. In this case, the storage and retrieval processing unit 69 appends the source IP address received in step S21 to the record of the connection ID and password combination used for authentication.

[0126] Next, the storage and retrieval processing unit 69 reads the virtual space information, which is the data for all records, from the virtual space information management DB 6005 (S23). Then, the transmitting and receiving unit 61 sends a response to the communication device 1a indicating that authentication was successful (S24). This response contains the virtual space information. As a result, the transmitting and receiving unit 11a of the communication device 1a receives the response.

[0127] Next, in the communication device 1a, the display control unit 34a displays a participant selection screen (see Figure 22) on the display 4 based on the virtual space information (S25). This participant selection screen displays selection buttons b1, b2, and b3, each indicating the number of participants to be selected. Here, the selection buttons b1, b2, and b3 indicate the types of participants: 2 people, 3 people, and 4 people. In addition, each selection button b1, etc., is associated with a session ID.

[0128] Here, when user A1 selects the desired selection button (in this case, "selection button b1"), the reception unit 32a accepts the selection of the number of participants "2 people" (communication session) (step S26).

[0129] Furthermore, in the communication device 1a, the display control unit 34a displays a role selection screen (see Figure 22) on the display 4 based on the virtual space information, indicating the role to be played when entering the virtual conference room (S27). This role selection screen displays selection buttons b11, b12, and b13, each indicating the role to be selected. When user A1 selects the desired selection button (in this case, "selection button b11"), the reception unit 32a accepts the role selection (step S28). Then, the transmitting / receiving unit 31a sends a request to the communication management server 6 to join the virtual conference room as the host (step S29). This participation request includes the session ID and virtual space type ID indicating the communication session selected in step S26, the role selected in step S28, and the connection ID of the requesting communication device 1a. As a result, the transmitting / receiving unit 51 of the communication management server 6 receives the participation request.

[0130] Next, the storage and retrieval processing unit 69 performs a virtual space participation process in the session management DB 6002 (see Figure 14(b)) (S30). Specifically, the storage and retrieval processing unit 69 manages the record with the same session ID as the session ID received in step S29 in the session management DB 6002 by associating it with the virtual space type ID received in step S29 and the connection ID of the participating communication device 1a.

[0131] Next, the storage and retrieval processing unit 69 registers the connection ID for the role in the virtual space type selected in step 28 with the placement management DB 6006 (S31). Here, the storage and retrieval processing unit 69 registers "01aa", which is the connection ID of user A1 of communication device 1a, in the connection ID item (attribute) of the record for the virtual space type ID and connection ID that indicates the organizer in the virtual conference room.

[0132] Next, the transmitting / receiving unit 61 of the communication management server 6 sends a participation request response to the communication device 1a for step S29 (S32). This participation request response includes the placement management information, which is all the data in the placement management DB 6006 that was updated by being registered in step S31. As a result, the transmitting / receiving unit 11a of the communication device 1a receives the participation request response. Note that the placement management information may not be all the data in the placement management DB 6006, but rather the data of the updated record portion (the virtual space type ID, virtual space type, role, conversion parameters from modeling coordinate system to world coordinate system, and connection ID data). Then, in the communication device 1a, the storage / reading processing unit 19a overwrites the placement management information received in step S32 with the placement management DB 3006a (S33).

[0133] Furthermore, if there are other communication devices participating in the same virtual conference room, the communication management server 6 needs to share placement management information by informing these devices that communication device 1a has newly joined with a specific role. Therefore, the transmitting / receiving unit 61 of the communication management server 6 sends participation information to a communication device of a user already participating in the virtual conference room (in this case, communication device 1b) indicating that user A1 of communication device 1a has joined (S34). This participation information includes placement management information with the same content as the placement management information sent in step S32. As a result, the transmitting / receiving unit 11b of communication device 1b receives the participation information. The destination of the transmitting / receiving unit 61 is another connection ID associated with the same session ID as the connection ID of communication device 1a in the session management DB 6002 (see Figure 14(b)). In other words, the destination is another communication device in the same virtual conference room as communication device 1a.

[0134] Next, in the communication device 1b, the storage and reading processing unit 19b overwrites the placement management information received in step S34 with the placement management DB 3006b (S35).

[0135] Furthermore, if user A1 selects selection button b2 in Figure 21 in step S26, in step S27, the display control unit 34a will display a role selection screen as shown in Figure 22. Also, if user A1 selects selection button b3 in Figure 22 in step S26, in step S27, the display control unit 34a will display a role selection screen as shown in Figure 23(c).

[0136] In the above example, a total of two selections are made: first, selecting the selection button b1, etc., that indicates the type of virtual space shown in Figure 22, and then selecting the selection button b11, etc., that indicates the role shown in Figure 22. However, this is not the only way. For example, in Figure 22, a total of nine selection buttons may be displayed, each indicating the purpose of communication (type of virtual space and role). The nine selection buttons could be displayed as, for example, Organizer in a conference room, Participant in a conference room, Bystander in a conference room, Interviewer in an interview room, Interviewee in an interview room, Bystander in an interview room, Instructor (teacher) in a lecture hall, Student in a lecture hall, and Bystander in a lecture hall. Communication refers to the transmission of information such as intentions, feelings, or thoughts, and is carried out through language, writing, or gestures.

[0137] <Image type information management process> Next, we will explain the image type information management process using Figures 23 and 24. Figure 23 is a sequence diagram showing the image type information management process for a communication device. Figure 24 is a sequence diagram showing the image type information management process for a smartphone.

[0138] (Management processing of image type information for communication devices) First, we will explain the image type information management process for communication devices using Figure 23.

[0139] The transmitting / receiving unit 31a of the communication device 1a sends an information addition request to the communication management server 6 (S41). This information addition request includes the connection ID of the transmitting device (communication device 1a), image type information, and display information. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the information addition request.

[0140] Next, the storage and read processing unit 69 of the communication management server 6 retrieves the corresponding session ID by searching the session management DB 6002 (see Figure 14(b)) using the connection ID received in step S41 as the search key (S42).

[0141] Next, the creation unit 66 creates a unique image data ID (S43). Then, the storage and reading processing unit 69 stores the session ID read in step S42, the image data ID generated in step S43, and the source connection ID and image type information received in step S41 as a new record in the image type management DB 6003 (see Figure 14(c)) (S44). Furthermore, the storage and reading processing unit 69 stores the display information, which is one record of data, in the display management DB 6004 (see Figure 15) (S45). This enables centralized management of the display information of each communication device 1.

[0142] Next, the transmitting / receiving unit 61 of the communication management server 6 transmits the image data ID generated in step S43 to the communication device 1a (S46). As a result, the transmitting / receiving unit 31a of the communication device 1a receives the image data ID.

[0143] Next, the storage and reading processing unit 39a of the communication device 1a stores the image data ID received in step S46 in the image type management DB 3003a (see Figure 9) in association with the connection ID and image type information of its own device (communication device 1a) (S47).

[0144] Furthermore, if there are other communication devices participating in the same virtual conference room, the communication management server 6 needs to share information with these communication devices by providing them with the image data ID, connection ID, and image type information of communication device 1a. Therefore, the transmitting / receiving unit 61 of the communication management server 6 sends an information addition notification to other communication devices (for example, communication device 1b) participating in the same virtual conference room as communication device 1a (S48). This information addition notification includes the image data ID generated in step S43, as well as the connection ID and image type information of the source communication device 1a received in step S51. As a result, the transmitting / receiving unit 11b of communication device 1b receives the information addition notification. The destination of the transmitting / receiving unit 61 is another connection ID associated with the same session ID as the connection ID of communication device 1a in the session management DB 6002 (see Figure 14(b)). In other words, the destination is another communication device in the same virtual conference room as communication device 1a. Also, in step S48, the communication management server 6 does not transmit the display information received in step 41. This is because the display information is not used by other communication devices.

[0145] Next, the storage and reading processing unit 39b of the communication device 1b stores the image data ID received in step S48, the connection ID of the source communication device 1a, and the image type information as a new record in the image type management DB 3003 (see Figure 9) (S49).

[0146] Furthermore, after sending the information addition notification, the communication management server 6 will send the connection ID and image type information of other communication devices that are already participating in the same virtual conference room to communication device 1 that has joined the same virtual conference room as communication device 1a, at timings such as step S46 above.

[0147] Therefore, all communication devices participating in the same virtual space can share image type information with other communication devices.

[0148] (Management of image type information for smartphones) Next, we will explain the image type information management process for smartphones using Figure 24.

[0149] First, user G1 at site G enters their connection ID and password on the login screen displayed on the display 918 by the display control unit 94 (see Figure 20(b)) and presses the "OK" button. The reception unit 92 then receives the connection ID and password, and the transmission / reception unit 91 sends a login request to the communication management server 6 (S61). This login request includes the connection ID, password, and the IP address of the source (smartphone 9). As a result, the transmission / reception unit 61 of the communication management server 6 receives the login request.

[0150] Next, the storage and retrieval processing unit 69 of the communication management server 6 searches the communication management DB 6001 (see Figure 14(a)) using the connection ID and password included in the login request as search keys, and performs authentication by determining whether the same connection ID and password combination is managed (S62). Here, we will continue the explanation assuming that the same connection ID and password combination is managed by the storage and retrieval processing unit 69. In this case, the storage and retrieval processing unit 69 appends the source IP address received in step S61 to the record of the connection ID and password combination used for authentication.

[0151] Next, the transmitting / receiving unit 61 sends a response to the smartphone 9 indicating that authentication was successful (S63). This response includes virtual space information. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the response.

[0152] Next, when user G1 at base G operates the control unit 815 of the 360-degree imaging device 8, the reception unit 82 receives the operation, the storage / reading processing unit 89 reads the GUID of the device (360-degree imaging device 8) stored in the storage unit 8000, and the communication unit 88 transmits the GUID of the device to the communication unit 98 of the smartphone 9 (S64). As a result, the communication unit 98 of the smartphone 9 receives the GUID of the 360-degree imaging device 8.

[0153] Next, the determination unit 95 of the smartphone 9 determines the image type (S65) by checking whether the vendor ID and product ID in the GUID received in step S64 are managed in the imaging device management DB 9001 (see Figure 18). Specifically, if the same vendor ID and product ID are managed in the imaging device management DB 9001, the determination unit 95 determines that the sender's imaging device (in this case, the 360-degree imaging device 8) is an imaging device that acquires 360-degree images. On the other hand, if the same vendor ID and product ID are not managed in the imaging device management DB 9001, the determination unit 95 determines that the source imaging device is an imaging device that takes general images.

[0154] Next, the storage and retrieval processing unit 99 stores the connection ID of the source device (smartphone 9) and the image type information, which is the result of the determination made in step S65, in association with the image type management DB 9003 (see Figure 9) (S66). In this state, the image data ID is not associated.

[0155] From this point onward, the processing in steps S67 to S75 is essentially the same as in steps S41 to S49 described above, except that the communication device 1a is replaced by the smartphone 9; all other processing is the same.

[0156] Specifically, the transmitting / receiving unit 91 of the smartphone 9 sends an information addition request to the communication management server 6 (S67). This information addition request includes the connection ID of the transmitting device (smartphone 9), image type information, and display information. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the information addition request.

[0157] Next, the storage and read processing unit 69 of the communication management server 6 retrieves the corresponding session ID by searching the session management DB 6002 (see Figure 14(b)) using the connection ID received in step S67 as the search key (S68).

[0158] Next, the creation unit 66 creates a unique image data ID (S69). Then, the storage and reading processing unit 69 stores the session ID read in step S68, the image data ID generated in step S69, and the source connection ID and image type information received in step S67 as a new record in the image type management DB 6003 (see Figure 14(c)) (S70). Furthermore, the storage and reading processing unit 69 stores the display information, which is one record of data, in the display management DB 6004 (see Figure 15) (S71).

[0159] Next, the transmitting / receiving unit 61 of the communication management server 6 transmits the image data ID generated in step S69 to the smartphone 9 (S72). As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the image data ID.

[0160] Next, the memory and read processing unit 39a of the smartphone 9 stores the image data ID received in step S72 in the image type management DB 3003 (see Figure 9) in association with the connection ID and image type information of its own device (smartphone 9) (S73).

[0161] Furthermore, if there are other communication devices participating in the same virtual conference room, the communication management server 6 needs to share information with these devices by transmitting the image data ID, connection ID, and image type information of the smartphone 9. Therefore, the transmitting / receiving unit 61 of the communication management server 6 sends an information addition notification to other communication devices (for example, communication device 1b) participating in the same virtual conference room as the smartphone 9 (S74). This information addition notification includes the image data ID generated in step S69, as well as the connection ID and image type information of the source smartphone 9 received in step S51. As a result, the transmitting / receiving unit 11b of communication device 1b receives the information addition notification. The destination of the transmitting / receiving unit 61 is another connection ID associated with the same session ID as the connection ID of the smartphone 9 in the session management DB 6002 (see Figure 14(b)). In other words, the destination is another communication device in the same virtual conference room as the smartphone 9.

[0162] Next, the memory and read processing unit 39b of the communication device 1b stores the image data ID received in step S74, the connection ID of the source smartphone 9, and the image type information as a new record in the image type management DB 3003b (see Figure 9) (S75).

[0163] Furthermore, after sending the information addition notification, the communication management server 6 will, at the timing of step S72, etc., send the connection ID and image type information of other communication devices that are already participating in the same virtual conference room to communication device 1 that has joined the same virtual conference room as smartphone 9.

[0164] Therefore, all communication devices and smartphones 9 participating in the same virtual space can share image type information with other communication devices and smartphones 9.

[0165] <Transmission of image and sound data> Next, the transmission process of image data and sound data will be explained using Figures 25 to 27. Figure 25 is a sequence diagram showing the process from when the luminance image data and sound data transmitted from the 360-degree imaging device reach each communication device. Figure 26 is a sequence diagram showing the process from when the luminance image data, depth image data, and sound data transmitted from a predetermined communication device reach other communication devices. Figure 27 is a conceptual diagram of the transmitted luminance image data and depth image data.

[0166] (Transmission from the 360-degree imaging device to each communication device) First, using Figure 25, we will explain the process when image data (in this case, luminance image data) and sound data are transmitted from the 360-degree imaging device 8 to each communication device 1 via the communication management server 6.

[0167] First, the 360-degree camera 8 starts shooting and collecting sound at base G, causing the shooting unit 83 to acquire image data (in this case, two luminance image data of a hemispherical image) and the sound collection unit 84 to acquire sound data (S81). In this case, the 360-degree camera 8 performs video recording, but it may also perform still image recording. Then, the communication unit 88 transmits the two luminance image data and sound data to the communication unit 98 of the smartphone 9 (S82). As a result, the communication unit 98 of the smartphone 9 receives the two luminance image data and sound data.

[0168] Next, the transmitting / receiving unit 91 of the smartphone 9 transmits to the communication management server 6 the two brightness image data and sound data sent from the 360-degree camera 8, as well as the connection ID of the smartphone as the source (S83). In this case, the two brightness image data include the image data ID received in step S72. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the two brightness image data (and image data ID), sound data, and the smartphone's connection ID.

[0169] Next, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 14(b)) to transmit two luminance image data files (and image data IDs), sound data, and the smartphone's connection ID to each communication device 1a to 1d participating in the same video call as the smartphone 9 (S84 to S87). As a result, the transmitting / receiving units 31a to 31d of each communication device 1a to 1d receive the two luminance image data files (and image data IDs), sound data, and the smartphone's connection ID.

[0170] (Transmission from a designated communication device to other communication devices and smartphones) Next, using Figure 26, we will explain the process when communication device 1a transmits image data (in this case, luminance image data and depth image data) and sound data to multiple other communication devices 1b, 1c, and 1d via the communication management server 6.

[0171] First, the communication device 1a starts shooting and sound collection at base A, at which point the luminance image acquisition unit 36a acquires luminance image data, the depth image acquisition unit 37a acquires depth image data, and the sound collection unit 38a acquires sound data (S101). In this case, the communication device 1a performs video recording, but it may also perform still image recording.

[0172] Then, the transmitting / receiving unit 31a of the communication device 1a transmits to the communication management server 6 image data (luminance image data, depth image data) and sound data, as well as the connection ID of the communication device 1a as the source (S102). In this case, the image data includes the image data ID received in step S46 above. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of the communication device 1a as the source.

[0173] In the communication system, a shooting unit 105 for taking self-portraits is provided, so the image data includes data for the luminance image Lc1 and the depth image Dc1. The data for the luminance image Lc1 and the depth image Dc1 are acquired by the shooting unit 105.

[0174] For example, the resolution of the luminance image Lc1 and depth image Dc1 data is 4K each. However, as shown in Figure 27, the transmitting communication device 1 compresses the two data sets into a single image data, resulting in a single image data with a resolution of 4K. Note that 4K is just one example; HD, FHD, 8K, etc., may also be used.

[0175] Next, returning to Figure 26, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 14(b)) to transmit image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of the source communication device 1a to each of the other communication devices 1b to 1d participating in the same video call as communication device 1a (S103 to S105). As a result, the transmitting / receiving units 31b, 31c, and 31d of each of the communication devices 1b, 1c, and 1d receive the image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of the source communication device 1a.

[0176] Since smartphone 9 is for transmission only, if the source is communication device 1, the communication management server 6 will not send image data, sound data, or the connection ID of the source communication device 1a to smartphone 9. However, if the smartphone is used for display purposes in the same way as communication device 1, the communication management server 6 will not send image data, sound data, or the connection ID of the source communication device 1a to smartphone 9.

[0177] <Receiving image and audio data> Next, the processing of receiving image data and sound data will be explained using Figures 28 to 38. Figure 28 is a sequence diagram showing the processing of receiving image data and sound data by the communication device 1d.

[0178] As shown in Figure 28, the transmitting / receiving unit 31a of the communication device 1a transmits image data (luminance image data, depth image data, image data ID), sound data, and the connection ID of the communication device 1a as the source to the communication management server 6 (S121). As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data, sound data, and the connection ID of the communication device 1a. This step S121 is the same as step S102 in Figure 26. Then, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 14(b)) to transmit image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of the communication device 1a as the source to other communication devices 1d participating in the same video call as the communication device 1a (S122). As a result, the transmitting / receiving unit 31d of the communication device 1d receives the image data, sound data, and the connection ID of the communication device 1a. This process is the same as step S103 in Figure 26.

[0179] Similarly, the transmitting / receiving unit 31b of communication device 1b transmits image data (luminance image data, depth image data, image data ID), sound data, and the connection ID of communication device 1b as the source to the communication management server 6 (S123). As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data, sound data, and the connection ID of communication device 1b. Then, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 14(b)) to transmit image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of communication device 1b as the source to other communication devices 1d participating in the same video call as communication device 1b (S124). As a result, the transmitting / receiving unit 31d of communication device 1d receives the image data, sound data, and the connection ID of communication device 1b.

[0180] Furthermore, the transmitting / receiving unit 31c of communication device 1c transmits image data (luminance image data, depth image data, image data ID), sound data, and the connection ID of communication device 1c as the source to the communication management server 6 (S125). As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data, sound data, and the connection ID of communication device 1c. Then, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 14(b)) to transmit image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of communication device 1c as the source to other communication devices 1d participating in the same video call as communication device 1c (S126). As a result, the transmitting / receiving unit 31d of communication device 1d receives the image data, sound data, and the connection ID of communication device 1b.

[0181] As a result, the communication device 1d can acquire image data and sound data from other locations A, B, and C.

[0182] Next, in the communication device 1d, the image and sound processing unit 33d performs image display and sound output processing based on the data received in steps S122, S124, 126 and step S87 in Figure 25 (S127). In this case, sound output is a general process of converting each sound data (electrical signal) into vibration, so the image display process will be explained in detail using Figure 29.

[0183] <Image display processing> Figure 29 is a flowchart illustrating the image display process.

[0184] First, the memory and read processing unit 39d searches the image type management DB 3003d using the image data ID received in step S87 in Figure 25 and steps S122, S124, and S126 in Figure 28 as a search key, and reads the corresponding image type information (S201). If the image type information is "Video_Depth" (S202; brightness and depth image), the process proceeds to step S203. If the image type information is "Video_Omni" (S202; 360-degree image), the process proceeds to step S211. If the image type information is "Video_Omni" (S202; planar image), the process proceeds to step S212.

[0185] (Display processing when the image type is a brightness / depth image) Here, we will explain the display process when the image type is a brightness / depth image.

[0186] ((Process for calculating the position of an object in the world coordinate system)) First, the communication device 1d calculates the position of the object in the world coordinate system (S203). Here, the display process when the image type is a luminance / depth image will be explained in detail using Figures 30 to 33. Figure 30 is a flowchart showing the process of calculating the position of the object in the world coordinate system.

[0187] First, the image and sound processing unit 33d creates a 3D image of an object based on the luminance image data and depth image data of each location, including its own location, in the virtual space (modeling coordinate system) (S221).

[0188] Here, we will use Figure 31 to explain the 3D rendering of an object's image. Figure 31 is a conceptual diagram of the transformation from the camera coordinate system to the modeling coordinate system, and from the modeling coordinate system to the world coordinate system. The camera coordinate system is a coordinate system that defines how the world appears on the projection plane, with the physical camera as the origin, representing the 3D world of the real world. It is necessary to transform the coordinates given in real coordinates into the coordinate system seen in the camera's coordinate system.

[0189] In order to display images of users from multiple physical locations in a single virtual space such as a virtual conference room, the image and sound processing unit 33d must first define the objects in the physical space of each location individually in a Modeling Coordinate System as the first virtual space, thereby creating a 3D model, and then define the image of each object in a single World Coordinate System.

[0190] Therefore, first, the image / sound processing unit 33d performs 3D modeling by transforming five sets of images, such as the luminance image La and depth image Da, defined in the Camera Coordinate System as shown in Figure 27, into a single first virtual space (modeling coordinate system) using (Equation 1). In Figure 31, user A1 is represented as an object. In this case, the image of user A1 in the modeling coordinate system is represented by a collection of many points, as shown in the enlarged view ev of Figure 31. In this embodiment, since images are taken from five angles by five luminance / depth imaging devices 5a to 5e, the image of user A1 is represented as a collection of many overlapping points. In this way, users A1, B1, C1, and D1 are each defined independently in separate modeling coordinate systems (3D modeled). Also, since user A1 is photographed from one angle by one imaging unit 105, it is 3D modeled, but it is not represented as a collection of many overlapping points. As a result, at each location's communication device 1, the image of user A1 will be displayed as a frontal view of the face, but the side view of the face will not be displayed.

[0191]

number

[0192] Vector P is a projection matrix, and vector P-1 is a matrix for transforming from the camera coordinate system to the real coordinate system. Vector M is a matrix for transforming from the real coordinate system to the modeling coordinate system. This output value shows the point (coordinates) and color information in the modeling coordinate system.

[0193] Next, returning to Figure 30, the memory / read processing unit 39d reads the display information of its own device (communication device 1d) stored in the memory unit 3000d (S222). Then, the image / sound processing unit 33d places the virtual display (also called a virtual screen) dd of its own location in the modeling coordinate system based on the "information on the position of each corner of the display in the virtual space (modeling coordinate system)" in the display information (S223). Figure 32 is a conceptual diagram showing the image of user D1 and the virtual display placed in the modeling coordinate system. For example, as shown in Figure 32, the image / sound processing unit 33d further places (defines) the virtual display dd on the image of user D1 at its own location. In this case, the virtual display dd1 is rotated in pan, tilt, and roll based on the rotation matrix (0) data output by the orientation sensor (detection unit 30a). In other words, the direction in which user A1 rotates the actual communication device 1a is reflected in the virtual display dd1.

[0194] Returning to Figure 30, the memory / reading processing unit 39d reads the "transformation parameters from modeling coordinate system to world coordinate system" associated with the connection IDs of users (communication devices) participating in the same virtual conference room from the placement management DB 3006d (see Figure 10) (S224). Then, the image / sound processing unit 33d performs a modeling transformation of the images of each object and the virtual display of its own location from the modeling coordinate system to the world coordinate system in the virtual space (S225).

[0195] Here, we will again use Figure 31 to explain the transformation from the modeling coordinate system to the world coordinate system.

[0196] As shown in Figure 31, the image and sound processing unit 33d performs a Modeling Transformation on the image of user A1 in the modeling coordinate system, using (Equation 2) to transfer it to a common second virtual space (world coordinate system). Note that the Modeling Transformation is a transformation process for placing display objects defined in the modeling coordinate system onto the world coordinate system, and involves a combination of four transformation processes: scaling, rotation, shearing, and translation of the display objects.

[0197]

number

[0198] Vector W is a matrix used to transform from the modeling coordinate system to the world coordinate system. Its output represents the point (coordinates) and color information in the world coordinate system. Vector O is a matrix used for rotation (pan, tilt, roll).

[0199] Figure 33 is an illustrative diagram showing the transformation from the modeling coordinate system to the world coordinate system. As shown in Figure 33, the image and sound processing unit 33d transforms the images of users A1 to D1 in each modeling coordinate system and the virtual display dd4 at its own location into a single common world coordinate system. This allows the images of users A1 to D1 at each location to participate in a common virtual conference room.

[0200] As shown in Figure 33, the world coordinate system of this embodiment includes a virtual spherical wall cw for pasting a 360-degree image and a virtual planar wall pw for pasting a 3D planar image. These will be discussed later.

[0201] ((Display of images visible to the user)) Next, returning to Figure 29, we will explain the process of displaying an image that user D1 can see on communication device 1.

[0202] As shown in Figure 29, the image and sound processing unit 33a detects skeletal information such as the eyes, nose, and shoulders of user D1 based on the luminance image data acquired by the luminance image acquisition unit 36d and the luminance image data acquired by the depth image acquisition unit 37d (S204).

[0203] Next, the image and sound processing unit 33a uses the center of both eyes in the skeletal information as the viewpoint position and, based on this viewpoint position, creates images of other users such as user A1 that user D1 can see on the virtual display dd in the world coordinate system, as shown in the center of Figure 31 (S205). In other words, the image and sound processing unit 33a converts the displayed objects in the world coordinate system into images that can be displayed on the virtual display dd based on the viewpoint position. This conversion means a conversion from the world coordinate system to the viewpoint coordinate system and is called a field of view conversion. The viewpoint coordinate system is a coordinate system that has the camera position (viewpoint) in the virtual space as its origin and extends in the direction that the user wants to see (the direction in which the subject exists).

[0204] In the world coordinate system, the intersection points of rays (rays) and polygons (virtual triangles, polygons, etc. that make up a 3D model) are calculated using ray tracing, which travels from the viewpoint to pixels on the virtual display dd. These rays are traced backward from the viewpoint, following the light rays traveling from the image direction. This method is called ray tracing and is also called reverse line of sight tracing because it traces the line of sight backward. The intersection point closest to the viewpoint, i.e., the object (called a "polygon" in the field of computer graphics) that first intersects the ray, is found, and the pixels on the virtual display are painted with the color of that object. If no object (polygon) intersects the ray, the pixel is painted with the background color. By performing this process for all pixels on the virtual display dd, the image displayed on the virtual display dd is created.

[0205] In addition to ray tracing, other methods such as the Z-buffer method also exist, and the methods for creating images displayed on the virtual display dd are not limited to those described above.

[0206] Next, the image / sound processing unit 33a converts the image created in the virtual space (world coordinate system) into an image for display on the display 4 of the communication device 1d in physical space (S206). Then, the display control unit 34d displays the image converted in step S206 on the display 4 of the communication device 1d in physical space (S207). In this case, the space in which the image is displayed on the display 4 is a two-dimensional planar image, and the resolution depends on the performance of the computer, especially on devices such as the display memory installed in the video board (or graphics board), and is therefore called the device coordinate system. (Display processing when the image type is a 360-degree image) Next, we will explain the display process when the image type is a 360-degree image.

[0207] The image / sound processing unit 33d creates a spherical image from the two luminance image data received by the transmitting / receiving unit 11d, and pastes this spherical image onto the virtual space (world coordinate system) (S211). In this case, the image / sound processing unit 33d combines the two luminance images, which are hemispherical images, to create a single equirectangular projection image EC. Then, the image / sound processing unit 33a uses OpenGL ES (Open Graphics Library for Embedded Systems) to paste the equirectangular projection image onto the virtual sphere (in this case, the inner wall surface of the virtual spherical wall cw in Figure 33) to create the spherical image data. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data.

[0208] Then, after processing in step S211, the process proceeds to step S206.

[0209] (Display processing when the image type is a flat image) Next, the display process for images of a planar type will be explained. In this case, even though it is a planar image, it needs to be placed on a virtual space, so the image / sound processing unit 33a converts the planar image into a three-dimensional image (S213). After the processing in step S213, the process proceeds to step S206.

[0210] <Example of image display> Next, various display examples from step S207 in Figure 29 will be explained using Figures 34 to 38.

[0211] (Example of display shown to a participant (user D1) in a video call with 4 people) ((Initial display)) In Figure 34, (a) shows the line of sight of user D1 in the world coordinate system when four people are making a video call, and (b) is an example of the image that the communication device 1d shows to user D1 in the case of (a).

[0212] As shown in Figure 34(a), in the virtual space (in this case, the world coordinate system), user A1 is located slightly behind and to the left of user D1's line of sight, user B1 is directly in front, and user C1 is to the left. In addition, a planar image PE is displayed behind user A1. Note that, in principle, only one user exists at a single location in the virtual space (world coordinate system).

[0213] Then, when user D1 in the physical space views this state in the virtual space on display 4, as shown in Figure 34(b), the display control unit 34d displays on displays 4a to 4c a composite image on which the 360-degree spherical image CE as a background image and images of each user A1, B1, and C1 with different sizes and display directions are combined.

[0214] The images of each user are those displayed by the display control unit 34d in step S207 of Figure 29. The 360-degree image CE is also displayed by the display control unit 34d in step 211 of Figure 29. In practice, the images of each user and the 360-degree image CE are displayed as a composite image, as described above. The display of the images of each user and the 360-degree image CE is the same hereafter.

[0215] Specifically, when user D1 views user A1's image, user A1's image is displayed in a predetermined area on display 4 such that it is positioned slightly behind and to the left and in front of user B1, is smaller in size than user B1's image, and has a larger display area on the left shoulder than on the right shoulder. Similarly, when user D1 views user B1's image, user B1's image is positioned closer than user A1's image which is directly in front, is larger in size than user A1's image, and is displayed in a predetermined area on display 4 such that user B1's face is directly in front of user D1. Furthermore, when user D1 views user C1's image, user C1's image is positioned to the left and is displayed in a predetermined area on display 4 such that user C1's face is directly in front of user D1.

[0216] ((Displayed after moving)) Next, we will explain an example of the display when user A1 moves slightly to the left (in the direction of the arrow) in physical space, as shown in Figure 35(b), from the state shown in Figure 34(b). In Figure 35, (a) is a diagram showing the line of sight of user D1 after movement in the world coordinate system in the case of a virtual meeting, and (b) is an example of the display of the image that communication device 1d shows to user D1 in the case of (a).

[0217] As shown in Figure 35(a), in the virtual space (world coordinate system), in the line of sight of user D1, user A1 appears to have moved slightly to the right compared to the case in Figure 34(a), user B1 appears to have moved slightly to the right of the front compared to the case in Figure 34(a), and user C1 appears to have moved slightly to the left of the left side compared to the case in Figure 34(a).

[0218] Then, when user D1 in the physical space views this state in the virtual space on the display 4, as shown in Figure 35(b), the display control unit 34d displays on the display 4 a 360-degree image CE as a background image that has been moved slightly to the right compared to Figure 34(b), as well as a composite image which is a combination of images of each user A1, B1, and C1 that are slightly different in size and display area compared to Figure 34(b).

[0219] Specifically, when user D1 views the 360-degree image CE, the 360-degree image CE is displayed on display 4 in a way that it is slightly shifted to the right compared to the state shown in Figure 34(a). Similarly, when user D1 views user A1's image, user A1's image is displayed in a predetermined area on display 4 in a way that it is slightly positioned to the right and slightly larger compared to the state shown in Figure 34(a). When user D1 views user B1's image, user B1's image is displayed in a predetermined area on display 4 in a way that it is slightly shifted to the right from directly in front and slightly smaller compared to the state shown in Figure 34(a). When user D1 views user C1's image, user D1's image is displayed in a predetermined area on display 4 in a way that it is slightly shifted to the left from the left side and slightly larger compared to the state shown in Figure 34(a).

[0220] (Example of a display to show to bystanders of a meeting) In Figure 36, (a) shows the viewpoint of an observer in the world coordinate system in the case of a virtual meeting, and (b) is an example of the image displayed on a smartphone in the case of (a).

[0221] As shown in Figure 36(a), in the virtual space (in this case, the world coordinate system), beyond the line of sight of the bystander (camera) G2, user A1 is located in the background to the right and slightly ahead, user B1 is located further to the right and slightly ahead, user C1 is located in the background to the left and slightly ahead, and user D1 is located further to the left and slightly ahead, slightly ahead. Note that in the case of a bystander, their own image is not displayed on each communication device. Therefore, in the case of a bystander, multiple users may exist at a single location in the virtual space (world coordinate system).

[0222] Then, as shown in Figure 36(b), the state in this virtual space is displayed on the display 918 by the display control unit 94 as a composite image, which is a combination of the 360-degree spherical image CE as the background image and images of each user A1, B1, C1, and D1 with different sizes and display directions. In the case of a bystander, their image is not captured like that of users A1, B1, C1, and D1, so the processing in steps S204 to S207 in Figure 29 is not executed. That is, even if bystander G2 moves the smartphone 9 up, down, left, or right, or moves their viewpoint up, down, left, or right relative to the smartphone 9, the display control unit 94 displays the same image.

[0223] Specifically, on display 918, the image of user A1 is located in a predetermined area at the back, diagonally to the right, smaller in size than user B1, and larger on the left shoulder than on the right shoulder. Also on display 918, the image of user B1 is located in a predetermined area at the front, diagonally to the right, further to the right than user A1, larger in size than user A1, and larger on the left shoulder than on the right shoulder. Furthermore, on display 918, the image of user C1 is located in a predetermined area at the back, diagonally to the left, smaller in size than user D1, and larger on the right shoulder than on the left shoulder. Also on display 918, the image of user D1 is located in a predetermined area at the front, diagonally to the left, further to the left than user C1, larger in size than user C1, and larger on the right shoulder than on the left shoulder.

[0224] Note that Bystander G2 may be User G1 in Figure 1, or it may be a different user. Also, if there are multiple bystanders, multiple users will be participating in the virtual event, such as a virtual meeting, as bystanders in the same position.

[0225] (Example of display shown to a participant (user D1) in a video call with two people) In Figure 37, (a) shows the line of sight of user D1 in the world coordinate system in the case of a video call between two people, and (b) is an example of the image that the communication device 1d shows to user D1 in the case of (a).

[0226] As shown in Figure 37(a), in the virtual space (in this case, the world coordinate system), user A1 is positioned directly in front of user D1's line of sight. Note that, in principle, only one user exists at a single location in the virtual space (world coordinate system).

[0227] Then, when user D1 in the physical space views this state in the virtual space on the display 4, as shown in Figure 37(b), the display control unit 34d displays a composite image on the display 4, which is a combination of the 360-degree spherical image CE as the background image and the images of each user A1.

[0228] (Example of display shown to a participant (user D1) in a video call with three people) In Figure 38, (a) shows the line of sight of user D1 in the world coordinate system in the case of a video call with three people, and (b) is an example of the image that the communication device 1d shows to user D1 in the case of (a).

[0229] As shown in Figure 38(a), in the virtual space (in this case, the world coordinate system), user A1 is positioned diagonally to the left and user B1 diagonally to the right, beyond user D1's line of sight. Note that, in principle, only one user exists at a single location in the virtual space (world coordinate system).

[0230] Then, when user D1 in the physical space views this state in the virtual space on the display 4, as shown in Figure 38(b), the display control unit 34d displays a composite image on the display 4, which is a combination of the 360-degree spherical image CE as the background image and the images of each user A1 and B1.

[0231] Specifically, when user D1 views user A1's image, user A1's image is displayed in a predetermined area on display 4 that allows it to be viewed from a position diagonally to the left and in front of the user. Similarly, when user D1 views user B1's image, user B1's image is displayed in a predetermined area on display 4 that allows it to be viewed from a position diagonally to the right and in front of the user.

[0232] (Brief explanation of the example display) Here, I will briefly explain the above display example.

[0233] The display control unit 34d displays a composite image on the display 4, which is a combination of the first image and the second image, with at least one of the first image (e.g., image of user A1 / image of user B1) based on the first image data transmitted by the first other communication device 1a and the second image (e.g., image of user B1 / image of user A1) based on the second image data transmitted by the second other communication device 1b having different sizes.

[0234] Furthermore, the second image data is composed of image data obtained by taking photographs from multiple positions (angles), and the display control unit 34d displays a composite image on the display 4, which is a composite image in which the second image shows a second object (e.g., user B1 / user A1) in a state where the display direction is different from that of the first object (e.g., user A1 / user B1) represented in the first image.

[0235] Furthermore, the display positions of the first and second images of the composite image displayed on the display 4 are changed in accordance with a change in the predetermined position of the person on the communication device side.

[0236] [Main effects of the embodiment] As described above, according to this embodiment, for example, communication device 1a displays images based on each image data transmitted by other communication devices 1b, 1c, and 1d in a predetermined area on the display 4, according to a predetermined position (for example, the viewpoint position between both eyes) of an object (user D1) in the virtual space (world coordinate system) on the side of its own device (communication device 1d) and the changeable positional relationship (also called "arrangement relationship") between multiple communication devices in the virtual space (world coordinate system) (see Figures 11 to 13) (see Figures 34(b), 35(b), 37(b), and 38(b)). This has the effect of resolving as much as possible the problem that it is difficult for users at multiple locations to obtain a sense of being in the same room according to the number of locations and the purpose of communication.

[0237] Furthermore, for example, communication device 1d displays multiple types of information with different positional relationships between multiple communication devices in the virtual space (see Figure 21), and accepts the selection of a specific number of attendees (e.g., 2 people) from among the number of attendees in a video call with different positional relationships between multiple communication devices in the virtual space. Then, communication device 1d displays an image based on image data transmitted by other communication devices in a predetermined area on the display 4 corresponding to the specific number of attendees (for example, see Figure 34(b)). This has the effect of making it easier for users to select a virtual space according to the purpose of communication.

[0238] 〔supplement〕 The above-mentioned communication management server 6 may simply be called a "server." The communication management server 6 may also be provided in the form of cloud computing. Cloud computing refers to a usage model in which computer resources are provided as a service via a computer network such as the internet, and the provided computer resources are servers. The form of service provided can be SaaS (Software as a Service), PaaS (Platform as a Service), HaaS or IaaS (Hardware / Infrastructure as a Service), but the form of service provision is not limited.

[0239] Furthermore, in the above embodiment, the virtual space and role are determined by the user selecting selection buttons b1 to b3 in Figure 21 and selection buttons b11, b12, and b13 in Figure 22, but this is not the only way. For example, by pre-setting the virtual space type and role for each connection ID in the communication management server 6, user A1 may not have to specifically select selection buttons b1 to b3 and selection buttons b11, b12, and b13, and the communication management server 6 may automatically determine the virtual space type and role.

[0240] Furthermore, in the above embodiment, as shown in Figure 32, each communication device 1 first defines the objects in the physical space of each location individually in a modeling coordinate system as a first virtual space to create a 3D model, and then converts the image of each object into a single world coordinate system. However, this is not the only method. For example, the communication management server 6 may first define the objects in the physical space of each location individually in a modeling coordinate system as a first virtual space to create a 3D model, and then convert the image of each object into a single world coordinate system. In this case, the communication management server 6 transmits the image data converted to the world coordinate system to the communication device 1 at each location. That is, a communication system comprising a first communication device (e.g., communication device 1d), a second communication device (e.g., communication device 1a), and a server (e.g., communication management server 6) includes a providing means that provides a display image generated according to a changeable positional relationship between the position of a subject in the first communication device in a virtual space and the position of a subject in the second communication device in a virtual space, using image data captured by the first communication device and image data captured by the second communication device, and a display control means that displays the display image provided by the providing means.

[0241] Multiple CPUs, as described in the above embodiment, may be provided in a single device.

[0242] Furthermore, each of the functions of the above embodiments can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), SOCs (System on a chip), GPUs (Graphics Processing Units), and conventional circuit modules designed to execute each of the functions described above.

[0243] Furthermore, in communication between each communication device 1a to 1d, the communication management server 6, the 360-degree imaging device 8, and the smartphone 9, other servers, routers, etc., may relay the data.

[0244] 〔summary〕 The following is a summary of this embodiment. Note that the statements in parentheses are not intended to limit the scope of the claims, but are merely illustrative examples.

[0245] (1) Communicate via video call with other communication devices at one or more other locations. The communication device of the model in which the display means is arranged acquires data of the luminance image and depth image of the subject at the home location by photographing the subject at the home location (refer to the case of communication device 1d in S101), a transmission / reception unit that transmits data of the luminance image and depth image of the home location to other communication devices at the one or more other locations (refer to the case of communication device 1d in S102 - S1105), a reception unit that receives data of the luminance image and depth image of one or more other locations obtained by other communication devices at the one or more other locations by photographing the subject at the other locations (refer to S121, S122), a generation unit that generates a 3D object of the subject at the home location in the virtual space based on the data of the luminance image and depth image of the home location, and generates a 3D object of the subject at the one or more other locations in the virtual space based on the data of the luminance image and depth image of the one or more other locations (refer to S127), and a display control unit that arranges the 3D object of the subject at the home location and the 3D objects of the subjects at the one or more other locations at arrangement positions in the virtual space corresponding to the number of locations where the video call is made, and displays an image of the 3D object of the subject at the other location on the display means according to the positional relationship in the virtual space between the 3D object of the subject at the home location and the 3D objects of the subjects at the one or more other locations (refer to S207, FIG. 35, etc.). The communication device is characterized by having the above components.

[0246] (2) In the communication device of (1) above, the display control unit displays information indicating the number of locations where the video call is made (refer to S25, FIG. 22), and has a reception unit that accepts selection of a specific number of locations among the number of locations (refer to S26). The display control unit arranges each 3D object generated by the generation unit at an arrangement position in the virtual space corresponding to the specific number of locations, and displays an image of each 3D object on the display means (refer to S207, FIG. 35, etc.). The communication device is characterized by the above.

[0247] (3) The information indicating the number of locations is characterized by indicating the number of locations and the arrangement relationship of each 3D object. The communication device is as described in (2) above.

[0248] (4) The communication device according to (2) or (3) above, wherein the display control unit displays information indicating a plurality of roles in the video call (see FIG. 22), the reception unit receives a selection of a specific role among the plurality of roles, and the display control unit arranges each 3D object generated by the generation unit at an arrangement position on the virtual space corresponding to the specific number of points and the specific role, and displays an image of each 3D object on the display means (S207, FIG. 35, etc.). The communication device is characterized by this.

Explanation of Signs

[0249] 1a Communication device (an example of another communication device, an example of a first other communication device) 1b Communication device (an example of another communication device, an example of a second other communication device) 1c Communication device (an example of another communication device) 1d Communication device (an example of the own device) 4 Display (an example of display means) 6 Communication management server 8 Omnidirectional imaging device 9 Smartphone) 31a Transmission / reception unit (an example of transmission means, an example of reception means, an example of a second transmission means, an example of a second reception means) 32d Reception unit (an example of reception means) 34d Display control unit (an example of display control means) 61 Transmission / reception unit (an example of a first transmission means, an example of a first reception means) b1 Selection button (an example of a specific type of information among a plurality of types of information with different positional relationships) b2 Selection button (an example of a specific type of information among a plurality of types of information with different positional relationships) b3 Selection button (an example of a specific type of information among a plurality of types of information with different positional relationships)

Claims

1. A desk-type communication device equipped with display means for making video calls with other communication devices at one or more other locations, One or more shooting means that acquires brightness image and depth image data of a site by photographing a subject at the site, A transmission means for transmitting the brightness image and depth image data of the local site to one or more other communication devices at other sites, A receiving means for receiving data of brightness images and depth images of one or more other locations obtained by other communication devices at one or more other locations photographing subjects at those locations, from other communication devices at one or more other locations. A generation means that generates a three-dimensional object of a subject at the local site in a virtual space based on the brightness image and depth image data of the local site, and generates a three-dimensional object of a subject at the one or more other sites in a virtual space based on the brightness image and depth image data of one or more other sites, A display control means that, in accordance with the number of locations conducting the video call, positions the 3D object of the subject at the local location and the 3D object of the subject at one or more other locations in a virtual space, and displays an image of the 3D object of the subject at the other locations and information indicating multiple roles in the video call on the display means, according to the positional relationship in the virtual space between the 3D object of the subject at the local location and the 3D object of the subject at one or more other locations. A reception means for accepting the selection of a specific role from among the aforementioned multiple roles, It has, The display control means is When a participant's selection for a specific role is accepted, each 3D object is placed in the virtual space using a virtual space placement position corresponding to the number of locations, and an image of each 3D object is displayed on the display means from the participant's line of sight. When a bystander is selected as a specific role, the system is characterized by placing each 3D object in the virtual space using virtual space placement positions corresponding to the number of locations, and displaying images of each 3D object on the display means from the bystander's perspective. Communication device.

2. A communication device according to claim 1, The display control means displays information indicating the number of locations where the video call is to be conducted. The reception means accepts the selection of a specific number of locations from the multiple number of locations, The display control means places each three-dimensional object generated by the generation means at a location in the virtual space corresponding to the specific number of locations, and displays an image of each three-dimensional object on the display means. A communication device characterized by the following.

3. The communication device according to claim 2, characterized in that the information indicating the number of multiple locations indicates the number of locations and the arrangement relationship of each three-dimensional object.

4. The communication device according to any one of claims 1 to 3, characterized in that the luminance image and depth image data include a plurality of luminance image data and depth image data obtained by capturing images from a plurality of positions in physical space.

5. A communication system comprising a desk-type communication device equipped with a display means for making video calls with other communication devices at one or more other locations, and a server, The aforementioned server, A first receiving means that receives data of brightness images and depth images of one or more other locations obtained by other communication devices at one or more other locations photographing subjects at those locations, from other communication devices at one or more other locations. A first transmitting means that transmits the brightness image and depth image data of one or more other locations received by the first receiving means to the communication device, It has, The aforementioned communication device is One or more shooting means that acquires brightness image and depth image data of a site by photographing a subject at the site, A second transmission means for transmitting the brightness image and depth image data of the local site to the server, A second receiving means for receiving brightness image and depth image data from one or more other locations from the server, A generation means that generates a three-dimensional object of a subject at the local site in a virtual space based on the brightness image and depth image data of the local site, and generates a three-dimensional object of a subject at the one or more other sites in a virtual space based on the brightness image and depth image data of one or more other sites, A display control means that, in accordance with the number of locations conducting the video call, positions the 3D object of the subject at the local location and the 3D object of the subject at one or more other locations in a virtual space, and displays an image of the 3D object of the subject at the other locations and information indicating multiple roles in the video call on the display means, according to the positional relationship in the virtual space between the 3D object of the subject at the local location and the 3D object of the subject at one or more other locations. A reception means for accepting the selection of a specific role from among the aforementioned multiple roles, It has, The display control means is When a participant's selection for a specific role is accepted, each 3D object is placed in the virtual space using a virtual space placement position corresponding to the number of locations, and an image of each 3D object is displayed on the display means from the participant's line of sight. When a bystander is selected as a specific role, the system is characterized by placing each 3D object in the virtual space using virtual space placement positions corresponding to the number of locations, and displaying images of each 3D object on the display means from the bystander's perspective. Communication system.

6. A method of displaying when making a video call with other communication devices at one or more other locations, One or more shooting steps to obtain brightness image and depth image data of the site by photographing a subject at the site, A transmission step of transmitting the brightness image and depth image data of the local site to one or more other communication devices at other sites, A receiving step in which one or more communication devices at one or more other locations receive data of brightness images and depth images of one or more other locations obtained by other communication devices at one or more other locations photographing subjects at those other locations, A generation step of generating a 3D object of a subject at the local site in a virtual space based on the brightness image and depth image data of the local site, and generating a 3D object of a subject at the one or more other sites in a virtual space based on the brightness image and depth image data of one or more other sites, A display control step in which a 3D object of the subject at the local location and a 3D object of the subject at one or more other locations are placed in a virtual space corresponding to the number of locations conducting the video call, and an image of the 3D object of the subject at the other locations and information indicating multiple roles in the video call are displayed on a display means according to the positional relationship in the virtual space between the 3D object of the subject at the local location and the 3D object of the subject at one or more other locations, A reception step that accepts the selection of a specific role from among the aforementioned multiple roles, Execute, The aforementioned display control step is: When a participant's selection for a specific role is accepted, each 3D object is placed in the virtual space using a virtual space placement position corresponding to the number of locations, and an image of each 3D object is displayed on the display means from the participant's line of sight. When a bystander is selected as a specific role, the system is characterized by placing each 3D object in the virtual space using virtual space placement positions corresponding to the number of locations, and displaying images of each 3D object on the display means from the bystander's perspective. Display method.

7. A program that causes a computer to perform the method described in claim 6.

Images