Communication device, communication system, display method, and program

Abstract

This invention provides a communication device, communication system, display method, and program that, even when displaying incomplete images of subjects at other locations, deliberately displays images of three-dimensional objects in virtual space within a virtual viewing window, utilizing the fixed idea that it is difficult to see the entire subject through the viewing window, thereby suppressing the loss of a sense of reality. [Solution] A communication device 1d that makes a video call with another communication device 1b at another location receives brightness image and depth image data of the other location obtained by the other communication device at the other location taking a picture of a subject at the other location, when the other communication device transmits this data (S121, S122). Based on the brightness image and depth image data of the other location, it generates a 3D object of the subject at the other location in a virtual space (S127), and displays the image of the 3D object in a virtual viewing window on the display.

Description

Technical Field

[0001] The present disclosure relates to a communication device, a communication system, a display method, and a program.

Background Art

[0002] In recent years, telework that makes effective use of time and place by using ICT (Information and Communication Technology) has been spreading. This makes it possible to achieve work-life balance, improve business efficiency, and improve customer satisfaction. In addition, along with the spread of telework, the performance of systems such as video (TV) conferencing via the Internet has been improving (see FIG. 38).

[0003] Also, in communication, the sense of distance between people is very important because it affects ease of conversation and tension (Edward Hall, The Hidden Dimension, 2000, Misuzu Shobo). However, in video conferencing, there is a problem that it is difficult to obtain a feeling of being in the same space as the conversation partner in reality (hereinafter referred to as "reality") like communication in the physical space (or real space).

[0004] Therefore, in Patent Document 1, a technique is disclosed that realizes motion parallax that changes the video of the conversation partner at another base according to the viewpoint (eye position) of a person in the physical space of one base, and displays the conversation partner on the display on the side of one base with a person size corresponding to the distance, thereby giving each user a sense of reality with the conversation partner.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, when attempting to display images of subjects taken at other locations from one's own location, images from other angles, such as the side of the subject at the other location, may not be displayed. This results in an incomplete image of the subject at the other location, which diminishes the sense of reality (the feeling of actually seeing it, rather than it being virtual). [Means for solving the problem]

[0007] The invention according to claim 1 is a communication device for making video calls with another communication device at another location, characterized by comprising: a receiving means for receiving data of brightness images and depth images of the other location obtained by the other communication device at the other location photographing a subject at the other location, which is transmitted by the other communication device; a generating means for generating a three-dimensional object of the subject at the other location in a virtual space based on the brightness image and depth image data of the other location; and a display control means for displaying an image of the three-dimensional object in a virtual viewing window on a display means. [Effects of the Invention]

[0008] As explained above, according to the present invention, even when displaying an incomplete image of a subject at another location, by deliberately displaying an image of a 3D object in a virtual space within a virtual viewing window and utilizing the fixed idea that it is difficult to see the entire subject through the viewing window, it is possible to suppress the loss of a sense of reality. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of a communication system according to an embodiment of the present invention. [Figure 2] This is an electrical hardware configuration diagram of a communication device and a smartphone. [Figure 3] This is an electrical hardware configuration diagram of the communication management server. [Figure 4] This is an electrical hardware configuration diagram of a 360-degree imaging device. [Figure 5] This is a functional block diagram of a communication system. [Figure 6] This is a functional block diagram of a communication system. [Figure 7] This is a functional block diagram of a communication system. [Figure 8] This is a conceptual diagram of the image type management table. [Figure 9] This is a conceptual diagram of the placement management table. [Figure 10] This is a conceptual diagram showing the placement in the world coordinate system, representing the optimal spatial configuration for a virtual conference room. [Figure 11] This is a conceptual diagram showing the placement in the world coordinate system, and represents the optimal spatial configuration of a virtual interview room. [Figure 12] This is a conceptual diagram showing the placement in the world coordinate system, representing the optimal spatial configuration of a virtual lecture hall. [Figure 13] (a) is a conceptual diagram of the communication management table, (b) is a conceptual diagram of the session management table, and (c) is a conceptual diagram of the image type management table. [Figure 14] This is a conceptual diagram of the display management table. [Figure 15] This is a conceptual diagram that visually represents the data managed by the display management table as points on the display. [Figure 16] This is a conceptual diagram of a virtual space information management table. [Figure 17] This is a conceptual diagram of the imaging equipment management table. [Figure 18] This is a sequence diagram showing the process of joining a specific communication session. [Figure 19] (a) is an example of a login screen displayed on a communication device, and (b) is an example of a login screen displayed on a smartphone. [Figure 20] This diagram shows the selection screen for the type of virtual space in a communication device. [Figure 21] (a) is the role selection screen when the virtual space type is meeting type, (b) is the role selection screen when the virtual space type is interview type, and (c) is the role selection screen when the virtual space type is lecture type. [Figure 22]This is a sequence diagram showing the management process of image type information for a communication device. [Figure 23] This is a sequence diagram showing the management process of image type information for a smartphone. [Figure 24] This is a sequence diagram showing the process until the luminance image data and sound data transmitted from the omnidirectional imaging device reach each communication device. [Figure 25] This is a sequence diagram showing the process until the luminance image data, depth image data, and sound data transmitted from a predetermined communication device reach another communication device. [Figure 26] This is a conceptual diagram of the transmitted luminance image data and depth image data. [Figure 27] This is a sequence diagram showing the process of receiving image data and sound data by communication device 1d. [Figure 28] This is a flowchart showing the process of image display. [Figure 29] This is a flowchart showing the calculation process of the position of an object in the world coordinate system. [Figure 30] This is a conceptual diagram showing the three-dimensionalization by conversion from the physical space (camera coordinate system) to the first virtual space (modeling coordinate system), and the conversion from the first virtual space (modeling coordinate system) to the second virtual space (world coordinate system). [Figure 31] This is a conceptual diagram showing the arrangement of the image of user D1 and the virtual display in the modeling coordinate system. [Figure 32] This is an image diagram showing the conversion from the modeling coordinate system to the world coordinate system. [Figure 33] (a) is a diagram showing the line of sight of user D1 in the world coordinate system in the case of a virtual conference, and (b) is an example of the image displayed by communication device 1d to user D1 in the case of (a). [Figure 34] (a) is a diagram showing the line of sight of user D1 after movement in the world coordinate system in the case of a virtual conference, and (b) is an example of the image displayed by communication device 1d to user D1 in the case of (a). [Figure 35](a) is a diagram showing the viewpoint of an observer in the world coordinate system in the case of a virtual meeting, and (b) is an example of the image displayed on a smartphone in the case of (a). [Figure 36] (a) is a diagram showing the line of sight of user D1 in the world coordinate system in the case of a virtual interview, and (b) is an example of the image that the communication device 1d shows to user D1 in the case of (a). [Figure 37] (a) is a diagram showing the line of sight of user D1 in the world coordinate system in the case of a virtual lecture, and (b) is an example of the image that communication device 1d shows to user D1 in the case of (a). [Figure 38] This is an example of how a video conference would look on a display. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings.

[0011] [Outline of the communication system] The communication system of this embodiment is a system for making video calls in a virtual space (also called a "virtual area").

[0012] First, the general configuration of the communication system of this embodiment will be explained using Figure 1. Figure 1 is a schematic diagram of the configuration of the communication system according to an embodiment of the present invention.

[0013] As shown in Figure 1, the communication system of this embodiment is constructed using a plurality of communication devices 1a, 1b, 1c, 1d, a communication management server 6, a 360-degree imaging device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection method of the communication network 100 may be either wireless or wired.

[0014] Communication devices 1a, 1b, 1c, and 1d are located at bases A, B, C, and D, respectively. Hereafter, any of the communication devices 1a, 1b, 1c, and 1d will be referred to as "communication device 1". Communication device 1 can take images and collect sound at its own base, and output images obtained from images taken at other bases and sounds obtained from sound collected at other bases. Communication device 1 will be explained in detail later.

[0015] Figure 1 also shows the case where communication devices 1a, 1b, 1c, and 1d are used by users A1, B1, C1, and D1, respectively. For example, communication device 1a takes pictures and records audio of its own location A and transmits the image and audio data to communication devices 1b, 1c, and 1d at other locations B, C, D, and G, as well as to a smartphone 9, via the communication network 100 and the communication management server 6. Similarly, communication devices b1, 1c, and 1d take pictures and record audio of their own locations and transmit the image and audio data to communication devices at other locations via the communication network 100 and the communication management server 6. The smartphone 9 will be described later.

[0016] The communication management server 6 is a computer that manages and controls communication between the communication device 1 and the smartphone 9, and manages the types of image data transmitted and received (general images and special images). Therefore, the communication management server is also a communication control server. The communication management server 6 is composed of one or more computers. If it is composed of multiple computers, the communication management server is also a communication management system (or communication control system).

[0017] The 360-degree camera 8 and smartphone 9 are being used by user G at base G. User G1 takes the 360-degree camera 8 to base G to photograph and record the area around base G, and transmits the image data and sound data to each communication device 1a, 1b, 1c, and 1d via smartphone 9, communication network 100, and communication management server 6.

[0018] The 360-degree camera 8 is a special digital camera used to capture subjects, landscapes, etc., and obtain two hemispherical images that form the basis of a 360-degree panoramic image. In this case, the 360-degree camera 8 cannot connect to the communication network 100 on its own, so it first transmits image data and sound data to the smartphone 9 using wireless communication technology such as Wi-Fi (Wireless Fidelity) or Bluetooth (registered trademark). The smartphone 9 then transmits the image data and sound data to other devices and servers such as the communication management server 6 via the communication network 100. Note that the 360-degree camera 8 may also be a device that can connect to the communication network 100 on its own.

[0019] As described above, the smartphone 9 acquires image data and sound data from the 360-degree camera 8 and transmits the image data and sound data to the communication management server 6 and the communication device 1 via the communication network 100. Furthermore, the smartphone 9 plays back the image data and sound data acquired from the communication device 1 at another location.

[0020] Furthermore, OpenGL ES is installed on the communication device 1 and the smartphone 9, enabling them to create predetermined region information that indicates a portion of a 360-degree image, or to create predetermined region images that are a portion of a 360-degree image from a 360-degree image sent from another communication device.

[0021] Furthermore, communication device 1 also includes digital televisions, smartphones (a separate device from smartphone 9), smartwatches, car navigation systems, medical devices, etc.

[0022] [Hardware configuration] Next, the hardware configurations of the communication device 1, communication management server 6, 360-degree imaging device 8, and smartphone 9 of this embodiment will be described in detail with reference to Figures 2 to 4.

[0023] <Hardware configuration of the communication management server> First, let's explain the electrical hardware configuration diagram of the communication management server using Figure 2. Figure 2 is the electrical hardware configuration diagram of the communication management server.

[0024] As shown in Figure 2, the communication management server 6, as a computer, is equipped with a CPU 601, ROM 602, RAM 603, HD 604, HDD (Hard Disk Drive) controller 605, display 606, external device connection I / F (Interface) 608, network I / F 609, bus line 610, pointing device 612, and media I / F 614.

[0025] Of these, the CPU 601 controls the operation of the entire communication management server 6. The ROM 602 stores programs used to drive the CPU 601, such as the IPL. The RAM 603 is used as the work area for the CPU 601. The HD 604 stores various data such as programs. The HDD controller 605 controls the reading or writing of various data to the HD 604 according to the control of the CPU 601. The display 606 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 608 is an interface for connecting various external devices. In this case, external devices include, for example, USB memory or printers. The network I / F 609 is an interface for data communication using the communication network 100. The bus line 610 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 601 shown in Figure 2.

[0026] Furthermore, the pointing device 612 is a type of input means used for selecting and executing various instructions, selecting processing targets, and moving the cursor. The media interface 614 controls the reading or writing (storage) of data to or from the recording medium 613, such as flash memory. The recording medium 613 includes DVDs and Blu-ray Discs.

[0027] <Hardware configuration of the 360-degree imaging system> Next, we will explain the electrical hardware configuration diagram of the 360-degree imaging device using Figure 3. Figure 3 is an electrical hardware configuration diagram of the 360-degree imaging device.

[0028] The 360-degree imaging device 8 is a 360-degree (omnidirectional) 360-degree imaging device using two image sensors, but it can use two or more image sensors. Furthermore, it does not necessarily have to be a device dedicated to omnidirectional imaging; a standard digital camera or smartphone can be fitted with an aftermarket omnidirectional imaging unit to achieve essentially the same functionality as the 360-degree imaging device 8.

[0029] As shown in Figure 3, the 360-degree imaging device 8 consists of an imaging unit 801, an image processing unit 804, an imaging control unit 805, a microphone 808, a sound processing unit 809, a CPU (Central Processing Unit) 811, a ROM (Read Only Memory) 812, an SRAM (Static Random Access Memory) 813, a DRAM (Dynamic Random Access Memory) 814, an operation unit 815, an external device connection interface 816, a communication circuit 817, an antenna 817a, an acceleration / direction sensor 818, and a recessed terminal 819 for Micro USB.

[0030] Of these, the imaging unit 801 includes wide-angle lenses (so-called fisheye lenses) 802a and 802b, each having a field of view of 180° or more for forming hemispherical images, and two image sensors 803a and 803b, each corresponding to the wide-angle lens. The image sensors 803a and 803b include image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert the optical image from the fisheye lenses 802a and 802b into electrical signal image data and output it, timing generation circuits that generate horizontal or vertical synchronization signals and pixel clocks for these image sensors, and a group of registers in which various commands and parameters necessary for the operation of these image sensors are set.

[0031] The image sensors 803a and 803b of the imaging unit 801 are each connected to the image processing unit 804 via a parallel I / F bus. On the other hand, the image sensors 803a and 803b of the imaging unit 801 are connected to the imaging control unit 805 via a serial I / F bus (such as an I2C bus). The image processing unit 804, the imaging control unit 805, and the sound processing unit 809 are connected to the CPU 811 via bus 810. Furthermore, ROM 812, SRAM 813, DRAM 814, operation unit 815, external device connection I / F (Interface) 816, communication circuit 817, and acceleration / direction sensor 818 are also connected to bus 810.

[0032] The image processing unit 804 receives image data output from image sensors 803a and 803b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create an equirectangular projection image. An equirectangular projection image is an image obtained by transforming a round sphere into a planar image, and in English, it refers to an image that uses a projection method called equirectangular.

[0033] The imaging control unit 805 generally uses the I2C bus to set commands and other information in the registers of the image sensors 803a and 803b, with the 805 acting as the master device and the image sensors 803a and 803b as slave devices. It receives necessary commands and other information from the CPU 811. The imaging control unit 805 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 803a and 803b and send it to the CPU 811.

[0034] Furthermore, the imaging control unit 805 instructs the image sensors 803a and 803b to output image data when the shutter button on the operation unit 815 is pressed. Some 360-degree imaging devices 8 may also have functions to support preview display on a display (for example, a smartphone display) or video display. In this case, the image data output from the image sensors 803a and 803b is performed continuously at a predetermined frame rate (frames / minute).

[0035] Furthermore, as will be described later, the imaging control unit 805 also functions as a synchronization control means that works in cooperation with the CPU 811 to synchronize the output timing of image data from the image sensors 803a and 803b. In this embodiment, the 360-degree imaging device 8 is not provided with a display, but a display unit may be provided.

[0036] Microphone 808 converts sound into sound (signal) data. Sound processing unit 809 receives the sound data output from microphone 808 through the I / F bus and performs predetermined processing on the sound data.

[0037] The CPU 811 controls the overall operation of the 360-degree imaging device 8 and performs necessary processing. The ROM 812 stores various programs for the CPU 811. The SRAM 813 and DRAM 814 are work memories that store programs executed by the CPU 811 and data in progress. In particular, the DRAM 814 stores image data in progress of processing by the image processing unit 804 and data of completed equirectangular projection images.

[0038] The control unit 815 is a collective term for the operation buttons, such as the shutter button 815a. The user inputs various shooting modes and shooting conditions by operating the control unit 815.

[0039] The external device connection interface 816 is an interface for connecting various external devices. These external devices include, for example, USB (Universal Serial Bus) memory and PCs (Personal Computers). The equirectangular projection image data stored in the DRAM 814 can be recorded to an external recording medium via the external device connection interface 816, or transmitted to an external terminal (device) such as a smartphone via the external device connection interface 816 as needed.

[0040] The communication circuit 817 communicates with an external terminal (device) such as a smartphone via an antenna 817a provided on the 360-degree imaging device 8, using short-range wireless communication technologies such as Wi-Fi, NFC (Near Field Communication), or Bluetooth (registered trademark). This communication circuit 817 can also transmit equirectangular projection image data to an external terminal (device) such as a smartphone.

[0041] The acceleration / direction sensor 818 calculates the orientation of the 360-degree camera 8 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) in accordance with Exif and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was taken and the data size of the image data. The acceleration / direction sensor 818 is also a sensor that detects changes in angle (Roll angle, Pitch angle, Yaw angle) associated with the movement of the 360-degree camera 8. Changes in angle are an example of related information (metadata) in accordance with Exif and are used for image processing such as image correction of captured images. Furthermore, the acceleration / direction sensor 818 is a sensor that detects acceleration in three axes. The 360-degree camera 8 calculates its own orientation (angle relative to the direction of gravity) based on the acceleration detected by the acceleration / direction sensor 818. The installation of the acceleration / direction sensor 818 in the 360-degree camera 8 improves the accuracy of image correction.

[0042] The recessed terminal 819 for Micro USB is a terminal for connecting a USB cable or similar when communicating with the smartphone 9 shown in Figure 1 via a wired connection rather than wirelessly using Wi-Fi or similar methods.

[0043] <Hardware configuration of communication equipment> Next, we will explain the electrical hardware configuration diagrams of the communication device and smartphone using Figure 4. Figure 4 is an electrical hardware configuration diagram of the communication device and smartphone.

[0044] Communication device 1 is, for example, a tablet terminal. As shown in Figure 4, communication device 1 includes a CPU 101, ROM 102, RAM 103, EEPROM 104, CMOS sensor 105, image sensor I / F 106, acceleration / direction sensor 107, media I / F 109, and GPS receiver 111.

[0045] Of these, the CPU 101 controls the operation of the entire communication device 1. The ROM 102 stores programs used to drive the CPU 101, such as the CPU 101 and IPL. The RAM 103 is used as the work area for the CPU 101. The EEPROM 104 reads or writes various data, such as smartphone programs, according to the control of the CPU 101. The shooting unit 105 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) according to the control of the CPU 101 to obtain image data. The image sensor I / F 106 is a circuit that controls the driving of the shooting unit 105. The acceleration / direction sensor 107 is a type of sensor such as an electronic magnetic compass, gyrocompass, or acceleration sensor that detects the Earth's magnetic field. The media I / F 109 controls the reading or writing (storage) of data to or from the recording medium 108, such as flash memory. The GPS receiver 111 receives GPS signals from GPS satellites.

[0046] Furthermore, the communication device 1 includes a long-distance communication circuit 112, a shooting unit 113, an image sensor interface 114, a microphone 115, a speaker 116, an audio input / output interface 117, a display 118, an external device connection interface 119, a short-distance communication circuit 120, an antenna 420a for the short-distance communication circuit 120, and a touch panel 421.

[0047] Of these, the long-distance communication circuit 112 is a circuit that communicates with other devices via the communication network 100. The shooting unit 113 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 101, and details will be described later. The image sensor I / F 114 is a circuit that controls the driving of the shooting unit 113. The microphone 115 is a built-in circuit that converts sound into electrical signals. The speaker 116 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and speech. The sound input / output I / F 117 is a circuit that processes the input and output of sound signals between the microphone 115 and the speaker 116 according to the control of the CPU 101. The display 118 is a type of display means such as liquid crystal or organic EL (Electro Luminescence) that displays images of the subject and various icons. The external device connection I / F 119 is an interface for connecting various external devices. The short-distance communication circuit 420 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 421 is a type of input means that allows the user to operate the communication device 1 by pressing the display 118.

[0048] Furthermore, the communication device 1 is equipped with a bus line 110. The bus line 110 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 101 shown in Figure 4.

[0049] The imaging unit 113 will now be described. The imaging unit 105 includes a luminance camera 105a, a depth camera 105b, and a bus line 105e that electrically connects these to the image sensor I / F. Of these, the luminance camera 105a is a device that uses an image sensor such as a CCD or CMOS to convert received light into an electrical signal and obtain image data. For example, an RGB color camera is included.

[0050] The depth camera 105b acquires depth image data by imaging subjects such as people and objects and measuring the distance from the subject to the depth camera 105b using a LiDAR laser method. The depth camera 105b is a camera that can capture the shape of a subject in three dimensions and detect various movements such as a person's skeleton, and reflect this in real time in subsequent processing. For this reason, the depth camera 105b is composed of a laser transmitter 105b1 and a laser receiver 105b2. The laser transmitter 113b1 emits laser light. The laser light emitted by the laser transmitter 105b1 is reflected by the subject, and the laser receiver 113b2 receives this reflected laser light, converts it into an electrical signal, and extracts the necessary information. Note that the above describes a method of measuring depth using LiDAR, but it is not limited to this. For example, a method of measuring depth using the TOF method may also be used. Furthermore, depth may be measured using a stereo measurement method with a standard RGB color camera, and the method of measuring depth is not limited to those described above.

[0051] The imaging unit 113 includes a luminance camera 113a, a depth camera 113b, and a bus line 113e that electrically connects these to the imaging sensor I / F. The depth camera 113b further includes a laser transmitter 113b1 and a laser receiver 113b2. The luminance camera 113a, depth camera 113b, bus line 113e, laser transmitter 113b1, and laser receiver 113b2 have the same configuration as the luminance camera 105a, depth camera 105b, bus line 105e, laser transmitter 105b1, and laser receiver 105b2 in the imaging unit 105, respectively, so their descriptions are omitted.

[0052] Furthermore, recording media such as CD-ROMs on which the above programs are stored, as well as hard drives on which these programs are stored, may be provided domestically or internationally as program products.

[0053] Furthermore, since smartphone 9 has the same configuration as communication device 1, its explanation will be omitted. Note that in Figure 4, each component of smartphone 9 is indicated by a symbol in parentheses.

[0054] [Functional configuration of the embodiment] Next, the functional configuration of this embodiment will be described using Figures 5 to 17. Figures 5 to 7 are functional block diagrams of the communication system.

[0055] <Functional Configuration of Communication Devices> Here, we will explain the functional configuration of communication device 1a, which is part of communication device 1.

[0056] As shown in Figure 5, the communication device 1a includes a detection unit 30a, a transmitting / receiving unit 31a, a receiving unit 32a, an image / sound processing unit 33a, a display control unit 34a, a judgment unit 35a, a sound collection unit 38a, a storage / reading processing unit 39a, and a detection unit 40. Each of these units is a function or means realized by any of the components shown in Figure 4 operating according to instructions from the CPU 101 that follow a program for the communication device deployed from the EEPROM 104 onto the RAM 103. The communication device 1a also includes a storage unit 3000a constructed from the ROM 102, RAM 103, and EEPROM 104 shown in Figure 4.

[0057] Furthermore, the communication device 1a has a brightness image acquisition unit 36a, a depth image acquisition unit 37a, and a sound collection unit 38a, which are functions realized by operating the imaging unit 105 in accordance with instructions from the CPU 101. The functions realized by the imaging unit 105 are enclosed in dashed lines.

[0058] (Image type management table) Figure 8 is a conceptual diagram showing the image type management table. The storage unit 3000a has an image type management DB 3003a constructed, which is composed of the image type management table. In this image type management table, image data ID (Identification), source connection ID, and image type information are stored and managed in association with each other. Of these, the image data ID is an example of image data identification information used to identify image data when performing video communication. Image data transmitted from the same source is assigned the same image data ID. This allows the destination (receiving communication device) to identify the source of the received image data. The source connection ID is an ID used to identify the communication device or the user of the communication device. The connection ID is an identifier that can be used within the authority granted by the communication management server 6. Note that the connection ID may also be a device ID used to identify the communication device 1a. The connection ID is an example of connection identification information, the user ID is an example of user identification information, and the device ID is an example of device identification information.

[0059] Image type information is a name used to identify the image related to the image data indicated by the associated image data ID, and is, for example, the source name. Image type information is created by PC3a according to a predetermined naming convention. Here, "Video_Depth", "Video_Omni", and "Video" are shown as image type information. These indicate, respectively, that the image type is a video consisting of a "luminance image including a depth image", a video consisting of a "luminance image that is a 360-degree image", and a video consisting of a "luminance image that is a planar image". Note that a 360-degree image is an example of a special image, and a planar image is an example of a general image.

[0060] For example, communication device 1a with connection ID "01aa" is shown to be transmitting image data indicated by image data ID "RS001". Furthermore, the image type indicated by the image type information is "Video_Depth".

[0061] Furthermore, data other than image data may also be managed in association with image data IDs. Examples of data other than image data include audio data and document data used during screen sharing.

[0062] (Distribution management table) Figure 9 is a conceptual diagram showing the placement management table. The storage unit 3000a contains a placement management DB 3006a, which is composed of the placement management table. In this placement management table, the virtual space type ID, virtual space type, role, conversion parameters from the modeling coordinate system to the world coordinate system, and connection ID are stored and managed in association with each other.

[0063] Among these, the virtual space type ID is an example of virtual space type classification information used to identify the type of virtual space used for video calls. The virtual space type is information that indicates the type of virtual space. In other words, a virtual space (virtual area) refers to a specific region, area, or range within a virtual space. In Figure 9, the virtual space types are shown as a virtual conference room, a virtual interview room, and a virtual lecture hall, but are not limited to these. For example, it also includes virtual booths set up in a part of a virtual room. It also includes indoor or outdoor (outdoor) virtual event venues, etc. An event indicates, for example, that it will be held at a predetermined place and time, or that it will be specially planned and held.

[0064] Roles are defined according to the type of virtual space and are the actions or activities assigned, required, or expected of an individual (or group). In addition to those shown in Figure 9, roles may also include those of a customer service guide, a customer service representative, a salesperson in a business negotiation, a customer in a business negotiation, etc. Roles may also include official positions such as president or department head.

[0065] The transformation parameters from the modeling coordinate system to the world coordinate system, as shown in Figure 32 below, represent the transformation parameters and rotation parameters used when transforming an object from the modeling coordinate system in the first virtual space to the world coordinate system in the second virtual space. These rotation parameters are data output from the acceleration / direction sensor 107 and represent (pan, tilt, roll).

[0066] The connection ID is the same concept as the connection ID in the image type management table, so we will omit its explanation.

[0067] Furthermore, virtual worlds depicted using computer graphics (CG) have a single coordinate system that serves as the basis for the world. This is called the world coordinate system. If the shape of an object is directly given in the world coordinate system, the shape of the object must be redefined every time the object moves within the world coordinate system. Therefore, a coordinate system is defined (called modeling) in which the shape of an object is determined in the 3D image of the computer graphics. This coordinate system is called the modeling coordinate system.

[0068] Here, we will explain world coordinates and modeling coordinates using different terminology. The world coordinate system represents the entire space in which objects are displayed in computer graphics. It is a coordinate system used to indicate the position of objects in space and is used to handle the placement and movement of objects. It can describe the position and orientation of each solid. If all objects were defined from the beginning in one large space called the world coordinate system, it would be difficult to handle the display process. Therefore, modeling coordinates are used to handle the shape and deformation of individual solids placed in space. The origin is taken at one of the vertices or in the vicinity of the solid, and the position and orientation of the points, lines, faces, etc. that make up the solid are described. By placing all display objects, which are freely defined in individual modeling coordinate systems, onto a unified world coordinate system, it is possible to create a graphics space, or a virtual 3D world.

[0069] Here, we will explain the optimal type space for each virtual space using Figures 10 to 12.

[0070] ((Virtual meeting room)) Figure 10 is a conceptual diagram showing the placement of elements in a world coordinate system, representing the optimal spatial configuration of a virtual meeting room. Figure 10 shows the placement of each role in the virtual meeting. The roles of the users in the virtual meeting room in Figure 9—organizer, participant (1), participant (2), participant (3), and observer—are placed in Figure 10 at organizer position a11, participant position a12, participant position a13, participant position a14, and observer position a15, respectively. Similarly, the virtual display positions d11-d15 are placed to correspond to organizer position a11, participant position a12, participant position a13, participant position a14, and observer position a15, respectively. Note that display positions d11-d14 represent the display positions of communication devices 1a-1d as shown in Figure 1, while display position d15 represents the position of a single display such as a smartphone 9.

[0071] ((Virtual interview room)) Figure 11 is a conceptual diagram showing the placement positions in the world coordinate system, representing the optimal spatial configuration of a virtual interview room. Figure 11 shows the placement positions for each role in the virtual interview. In Figure 9, the roles of each user in the virtual interview room—interviewer (1), interviewer (2), interviewer (3), interviewee, and bystander—are placed at interviewer positions a21, a22, a23, interviewee position a24, and bystander position a25, respectively, in Figure 11. Similarly, the display positions d21 to d25 are positioned to correspond to interviewer positions a21, a22, a23, interviewee position a24, and bystander position a25, respectively. Note that display positions d21 to d24 represent the display positions of communication devices 1a to 1d as shown in Figure 1, while display position d25 represents the position of a single display such as a smartphone 9.

[0072] ((Virtual lecture hall)) Figure 12 is a conceptual diagram showing the placement of elements in the world coordinate system, representing the optimal spatial configuration of a virtual lecture hall. Figure 12 shows the placement of each role in the virtual lecture. The roles of the users in the virtual conference room in Figure 9—lecturer (teacher), student (1), student (2), student (3), and observer—are placed in Figure 12 at lecturer position a31, student position a32, student position a33, student position a34, and observer position a35, respectively. Similarly, the display positions d31 to d35 are positioned to correspond to lecturer position a31, student position a32, student position a33, student position a34, and observer position a35, respectively. Note that display positions d31 to d34 represent the positions of the three displays of communication devices 1a to 1d as shown in Figure 1, while display position d35 represents the position of a single display such as a smartphone 9.

[0073] In Figures 9 through 14, there are a total of five user locations and virtual space locations, but this is not limited to any number. Similarly, there may be multiple bystander locations, or multiple bystanders may enter a virtual conference room and be positioned at the location of a single bystander. Furthermore, meetings, interviews, and lectures are examples of events.

[0074] (Functional configuration of each communication device) Next, we will explain in more detail the functional configurations of the communication device 1a using Figures 4 and 5.

[0075] The detection unit 30a of the communication device 1a is implemented by commands from the CPU 301 to the acceleration / direction sensor 107, and detects the direction of movement and rotation of the communication device 1a.

[0076] The transmitting / receiving unit 31a is implemented by the processing of the network interface 309 by the CPU 301, and transmits and receives various data (or information) with other devices (for example, the communication management server 6) via the communication network 100.

[0077] The reception unit 32a is mainly implemented by the processing of the keyboard 12, mouse 13, and pointing device 312 by the CPU 301, and accepts various selections or inputs from the user.

[0078] The image and sound processing unit 33a is mainly implemented by the CPU 301 and performs various image and sound processing. For example, the image and sound processing unit 33a acts as a generation unit and generates a 3D object of the subject in virtual space based on luminance image and depth image data.

[0079] The display control unit 34a is mainly implemented by the CPU 301 and displays various images on each of the displays 4a to 4c.

[0080] The decision unit 35a is mainly implemented by the processing of the CPU 301 and performs various decisions.

[0081] The luminance image acquisition unit 36a is mainly implemented by the processing of the luminance camera 501 by the CPU 301, and acquires luminance image data by photographing the subject. Here, it is referred to as a luminance image, but this term is used for the purpose of contrasting it with depth images, and a luminance image is not an image that contains only luminance information, but also color information.

[0082] The depth image acquisition unit 37a is mainly implemented by the processing of the depth camera 502 by the CPU 301, and acquires depth image data by photographing the subject.

[0083] The sound collection unit 38a is mainly implemented by the processing of the microphone 505 by the CPU 301, and collects sound to acquire sound data.

[0084] The memory and read processing unit 39a is mainly implemented by the CPU 301 and stores various data (or information) in the memory unit 3000a and reads various data (or information) from the memory unit 3000a.

[0085] Furthermore, communication devices 1b, 1c, and 1d have the same functions, storage, and database as communication device 1a. The functions of communication devices 1b, 1c, and 1d are essentially implemented by PCs 3b, 3c, and 3d, respectively. In Figures 5 and 6, the only difference is that the transceiver 31a in communication device 1a corresponds to the transceiver 31b in communication device 1b; therefore, a detailed explanation of these is omitted.

[0086] <Functional Configuration of the Communication Management Server> Next, the functional configuration of the communication management server 6 will be explained using Figures 2 and 5. Figure 5 is a functional configuration diagram of the communication management server.

[0087] As shown in Figure 5, the communication management server 6 has a transmitting / receiving unit 61, a determination unit 65, a creation unit 66, and a storage / reading processing unit 69. Each of these units is a function or means realized by any of the components shown in Figure 2 operating according to instructions from the CPU 601 that follow a communication management (control) program deployed from HD 604 onto RAM 603. The communication management server 6 also has a storage unit 6000 constructed from ROM 602, RAM 603, and HD 604 as shown in Figure 2.

[0088] (Communication management table) Figure 13(a) is a conceptual diagram showing a communication management table. The storage unit 6000 has a communication management DB 6001 constructed, which consists of a communication management table as shown in Figure 13(a). In this communication management table, each connection ID of all communication devices 1 managed by the communication management server 6 is associated with and managed with a password and an IP address. For example, in the communication management table shown in Figure 13, the connection ID of communication device 1a is "01aa", the password is "aaaa", and the IP address is "1.2.1.3" as IPv4. Note that the IP address is shown in a simplified manner. Also, the IP address may be IPv6 instead of IPv4. Also, instead of an IP address, a domain name (a string of characters that indicates a name used as an email address or homepage address) may be used.

[0089] (Session management table) Figure 13(b) is a conceptual diagram showing the session management table. The memory unit 6000 has a session management DB 6002 constructed, which consists of the session management table shown in Figure 13(b). In this session management table, the session ID, virtual space type ID, and connection ID of the communication device that participated in the video call of the same communication session are stored and managed in association with each other. Of these, the session ID is an example of session identification information for identifying the communication session that realizes the video call, and is generated for each virtual space such as a virtual conference room. The session ID may be replaced with a meeting ID. The meeting ID indicates an identifier for managing meetings (conversations between two or more people) on the communication management server. Note that the virtual space type ID and the connection ID of the participating communication device are the same concepts as the virtual space type ID and connection ID managed in the placement management table shown in Figure 9, so their explanation is omitted.

[0090] (Image type management table) The memory unit 6000 has an image type management DB 6003 constructed, which consists of an image type management table as shown in Figure 13(c). This image type management DB 6003 has an additional item (attribute) for session ID (communication ID) compared to the image type management DB 3003a mentioned above. Since each item (attribute) has already been explained, the explanation will be omitted.

[0091] (Display management table) Figure 14 is a conceptual diagram showing the display management table. The memory unit 6000 has a display management DB 6004 constructed, which consists of the display management table shown in Figure 14. This display management table stores and manages the vendor ID and product ID of the display's GUID (Globally Unique Identifier), the vertical and horizontal dimensions (position of each corner) and resolution of the display in physical space (real coordinate system), and the position of each corner of the display in virtual space (modeling coordinate system), all associated with each other. The real coordinate system refers to a coordinate system based on an arbitrary position in the three-dimensional world, which is the real world.

[0092] Here, Figure 15 is used to visually explain the positions indicated by the data managed in the display management table as points on the display. Figure 15 is a conceptual diagram that visually shows the positions indicated by the data managed in the display management table as points on the display.

[0093] For example, if the GUID of the display is "vid_10ca&pid_0001", it indicates the points representing the positions of each of the four corners of display 118 in physical space (real coordinate system). Similarly, if the GUID of the communication device is "vid_10ca&pid_0001", it indicates the points representing the positions of each of the four corners of the three displays in virtual space (modeling coordinate system).

[0094] Furthermore, if the GUID of the display is "vid_11ca&pid_0010", the points indicating the positions of each of the four corners of one of the displays of smartphone 9 in physical space (real coordinate system) are shown. Similarly, if the GUID of the smartphone is "vid_11ca&pid_0010", the points indicating the positions of the four corners of one of the displays of smartphone in virtual space (modeling coordinate system) are shown.

[0095] Furthermore, if the display's GUID is "vid_12ca&pid_0100", it indicates the positions of the four corners of a typical laptop's display in physical space (real coordinate system). Similarly, if the display's GUID is "vid_12ca&pid_0100", it indicates the positions of the four corners of a laptop's display in virtual space (modeling coordinate system).

[0096] (Virtual Space Information Management Table) Figure 16 is a conceptual diagram showing the virtual space information management table. The memory unit 6000 has a virtual space information management DB 6005 constructed, which is composed of the virtual space information management table shown in Figure 16. In this virtual space information management table, session ID (communication ID), virtual space type, and role name are stored and managed in association with each other. The session ID (communication ID) and virtual space type are the same concepts as the "Session ID (communication ID)" and "Virtual Space Type" items (attributes) of the session management DB 6002 described above, so their explanation is omitted. In addition, the role name is managed using the "Role" item (attribute) of the deployment management DB 3006a described above as the name.

[0097] (Functional configuration of the communication management server) Next, we will explain the functional configuration of the communication management server 6 in more detail using Figures 2 and 5.

[0098] The transmitting / receiving unit 61 of the communication management server 6 is implemented by the processing of the CPU 601 to the network I / F 609, and transmits and receives various data (or information) with other devices (for example, communication device 1, smartphone 9) via the communication network 100.

[0099] The decision unit 65 is mainly implemented by the processing of the CPU 601 and performs various decisions.

[0100] The creation unit 66 is mainly implemented by the CPU 601 and creates image data IDs.

[0101] The memory and read processing unit 69 is mainly implemented by the CPU 601's processing of the HDD 605 shown in Figure 2, and stores various data (or information) in the memory unit 5000 and reads various data (or information) from the memory unit 5000.

[0102] <Functional Configuration of a 360° Spherical Imaging System> Next, the functional configuration of the 360-degree imaging device 8 will be explained using Figures 3 and 7.

[0103] As shown in Figure 7, the 360-degree imaging device 8 includes a reception unit 82, an imaging unit 83, a sound collection unit 84, a judgment unit 85, a communication unit 88, and a storage / reading processing unit 89. Each of these units is a function or means realized by any of the components shown in Figure 3 operating according to instructions from the CPU 811 that follow a program for the imaging device deployed on SRAM 813 to DRAM 814.

[0104] Furthermore, the 360-degree imaging device 8 has a memory unit 8000 constructed from ROM 512, SRAM 513, and DRAM 514, as shown in Figure 3. The GUID of the device is stored in the memory unit 8000.

[0105] (Functional configuration of a 360-degree imaging device) Next, we will explain in more detail the functional configurations of the 360-degree imaging device 8 using Figures 3 and 7.

[0106] The reception unit 82 of the 360-degree imaging device 8 is mainly realized by the processing of the CPU 811 to the operation unit 515 shown in Figure 3, and receives operation input from the user.

[0107] The imaging unit 83 is mainly realized by the processing performed by the CPU 811 on the imaging unit 801, image processing unit 804, and imaging control unit 805 shown in Figure 3, and captures images of subjects, landscapes, etc., and obtains data for two hemispherical images that constitute a full-sphere image.

[0108] The sound collection unit 84 is realized by the processing of the microphone 808 and sound processing unit 809 shown in Figure 3 by the CPU 811, and collects sounds from the surroundings of the 360-degree imaging device 8.

[0109] The decision unit 85 is mainly implemented by the CPU 811 and performs various decisions.

[0110] The communication unit 88 is mainly implemented by the processing of the communication circuit 817 by the CPU 811, and can communicate with the communication unit 98 of the smartphone 9 (described later) using short-range wireless communication technologies such as NFC, Bluetooth, and Wi-Fi.

[0111] The memory and read processing unit 89 is mainly implemented by the CPU 811 shown in Figure 3, and stores various data (or information) in the memory unit 8000 and reads various data (or information) from the memory unit 8000.

[0112] <Smartphone Functionality Configuration> Next, the functional configuration of the smartphone 9 will be described in detail using Figures 4 and 6. As shown in Figure 6, the smartphone 9 has a transmitting / receiving unit 91, a receiving unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a calculation unit 97, a communication unit 98, and a storage / reading processing unit 99. Each of these units is a function or means realized by any of the components shown in Figure 4 operating according to instructions from the CPU 901 that follow a program for the smartphone 9 deployed from the EEPROM 904 onto the RAM 903.

[0113] Furthermore, the smartphone 9 has a storage unit 9000 constructed from ROM 902, RAM 903, and EEPROM 904, as shown in Figure 4.

[0114] (Imaging equipment management table) Figure 17 is a conceptual diagram showing the imaging device management table. The memory unit 8000 has an imaging device management DB 9001 constructed, which is composed of the imaging device management table shown in Figure 17. This imaging device management table stores and manages the vendor ID and product ID of the GUID of the imaging device that can obtain the two hemispherical images that form the basis of the 360-degree spherical image. As GUIDs, for example, the vendor ID (VID) and product ID (PID) used in USB devices can be used. The smartphone 9 obtains the vendor ID and product ID by downloading them from a designated server.

[0115] (Image type management table) The memory unit 9000 has an image type management DB 9003, which has the same data structure as the image type management DB 3003a. The image type management tables that make up the image type management DB 6003 have already been explained in Figure 8, so we will omit their explanation here.

[0116] (Placement management table) The storage unit 9000 has a placement management DB 9006 constructed within it, which has a data structure similar to that of placement management DB 3006a. The placement management tables that make up placement management DB 9006 have already been explained in Figures 9 to 12, so their explanation will be omitted here.

[0117] [Processing or operation of the embodiment] Next, the processing or operation of this embodiment will be described using Figures 18 to 37.

[0118] <Joining a video call> First, the process of joining a specific virtual space will be explained using Figures 18 to 21. Figure 18 is a sequence diagram showing the process of joining a specific virtual space. Figure 19 is an example of the login screen displayed on a communication device. The login screen on a smartphone is similar. Figure 20 is a diagram showing the screen for selecting the type of virtual space. In Figure 21, (a) is the role selection screen when the type of virtual space is a meeting type (type of meeting), (b) is the role selection screen when the type of virtual space is an interview type, and (c) is the role selection screen when the type of virtual space is a lecture type.

[0119] First, a user at site A (for example, user A1) enters their connection ID and password on the login screen (see Figure 19) displayed on the display 118 by the display control unit 34a and presses the "OK" button. The reception unit 32a then receives the connection ID and password, and the transmission / reception unit 31a sends a login request to the communication management server 6 (S21). This login request includes the connection ID, password, and the IP address of the source (communication device 1a). As a result, the transmission / reception unit 61 of the communication management server 6 receives the login request.

[0120] Next, the storage and retrieval processing unit 69 of the communication management server 6 searches the communication management DB 6001 (see Figure 13(a)) using the connection ID and password included in the login request as search keys, and performs authentication by determining whether the same connection ID and password combination is managed (step S22). Here, we will continue the explanation assuming that the same connection ID and password combination is managed by the storage and retrieval processing unit 69. In this case, the storage and retrieval processing unit 69 appends the source IP address received in step S21 to the record of the connection ID and password combination used for authentication.

[0121] Next, the storage and retrieval processing unit 69 reads the virtual space information, which is the data for all records, from the virtual space information management DB 6005 (S23). Then, the transmitting and receiving unit 61 sends a response to the communication device 1a indicating that authentication was successful (S24). This response contains the virtual space information. As a result, the transmitting and receiving unit 11a of the communication device 1a receives the response.

[0122] Next, in the communication device 1a, the display control unit 34a displays a virtual space selection screen (see Figure 20) on the display 118 based on the virtual space information (S25). This virtual space selection screen displays selection buttons b1, b2, and b3, each indicating the type of virtual space to be selected. In this case, the selection buttons b1, b2, and b3 indicate virtual conference rooms, virtual interview rooms, and virtual lecture rooms. Each selection button b1, etc., is associated with a session ID.

[0123] Here, when user A1 selects the desired selection button (in this case, "selection button b1"), the reception unit 32a accepts the selection of the type of virtual space (communication session) (step S26).

[0124] Furthermore, in the communication device 1a, the display control unit 34a displays a role selection screen (see Figure 21(a)) on the display 118 based on the virtual space information, indicating the role to be played when entering the virtual conference room (S27). This role selection screen displays selection buttons b11, b12, and b13, each indicating the role to be selected. When user A1 selects the desired selection button (in this case, "selection button b11"), the reception unit 32a accepts the role selection (step S28). Then, the transmitting / receiving unit 31a sends a request to the communication management server 6 to join the virtual conference room as the host (step S29). This participation request includes the session ID and virtual space type ID indicating the communication session selected in step S26, the role selected in step S28, and the connection ID of the requesting communication device 1a. As a result, the transmitting / receiving unit 51 of the communication management server 6 receives the participation request.

[0125] Next, the storage and retrieval processing unit 69 performs a virtual space participation process on the session management DB 6002 (see Figure 13(b)) (S30). Specifically, the storage and retrieval processing unit 69 manages the session management DB 6002 by associating the virtual space type ID received in step S29 and the connection ID of the participating communication device 1a with the record of the same session ID as the session ID received in step S29.

[0126] Next, the storage and retrieval processing unit 69 registers the connection ID for the role in the virtual space type selected in step 28 with the placement management DB 6006 (S31). Here, the storage and retrieval processing unit 69 registers "01aa", which is the connection ID of user A1 of communication device 1a, in the connection ID item (attribute) of the record for the virtual space type ID and connection ID that indicates the organizer in the virtual conference room.

[0127] Next, the transmitting / receiving unit 61 of the communication management server 6 sends a participation request response to the communication device 1a for step S29 (S32). This participation request response includes the placement management information, which is all the data in the placement management DB 6006 that was updated by being registered in step S31. As a result, the transmitting / receiving unit 11a of the communication device 1a receives the participation request response. Note that the placement management information may not be all the data in the placement management DB 6006, but rather the data of the updated record portion (the virtual space type ID, virtual space type, role, conversion parameters from modeling coordinate system to world coordinate system, and connection ID data). Then, in the communication device 1a, the storage / reading processing unit 19a overwrites the placement management information received in step S32 with the placement management DB 3006a (S33).

[0128] Furthermore, if there are other communication devices participating in the same virtual conference room, the communication management server 6 needs to share placement management information by informing these devices that communication device 1a has newly joined with a specific role. Therefore, the transmitting / receiving unit 61 of the communication management server 6 sends participation information to a communication device of a user already participating in the virtual conference room (in this case, communication device 1b) indicating that user A1 of communication device 1a has joined (S34). This participation information includes placement management information with the same content as the placement management information sent in step S32. As a result, the transmitting / receiving unit 11b of communication device 1b receives the participation information. The destination of the transmitting / receiving unit 61 is another connection ID associated with the same session ID as the connection ID of communication device 1a in the session management DB 6002 (see Figure 13(b)). In other words, the destination is another communication device that is in the same virtual conference room as communication device 1a.

[0129] Next, in the communication device 1b, the storage and reading processing unit 19b overwrites the placement management information received in step S34 with the placement management DB 3006b (S35).

[0130] Furthermore, if user A1 selects selection button b2 in Figure 20 in step S26, in step S27, the display control unit 34a will display a role selection screen as shown in Figure 21(b). Also, if user A1 selects selection button b3 in Figure 20 in step S26, in step S27, the display control unit 34a will display a role selection screen as shown in Figure 21(c).

[0131] In the above example, a total of two selections are made: first, selecting the selection buttons b1 etc. that indicate the type of virtual space shown in Figure 20, and then selecting the selection buttons b11 etc. that indicate the role shown in Figure 21. However, this is not the only way. For example, in Figure 20, a total of nine selection buttons indicating the purpose of communication (for each type of virtual space and role) may be displayed. The nine selection buttons could be displayed as, for example, Organizer in a conference room, Participant in a conference room, Bystander in a conference room, Interviewer in an interview room, Interviewee in an interview room, Bystander in an interview room, Instructor (teacher) in a lecture room, Student in a lecture room, and Bystander in a lecture room. Communication refers to the transmission of information such as intentions, feelings, or thoughts, and is carried out through the medium of language, writing, or gestures.

[0132] <Image type information management process> Next, we will explain the image type information management process using Figures 22 and 23. Figure 22 is a sequence diagram showing the image type information management process for a communication device. Figure 23 is a sequence diagram showing the image type information management process for a smartphone.

[0133] (Management processing of image type information for communication devices) First, we will explain the image type information management process for communication devices using Figure 22.

[0134] The transmitting / receiving unit 31a of the communication device 1a sends an information addition request to the communication management server 6 (S41). This information addition request includes the connection ID of the transmitting device (communication device 1a), image type information, and display information. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the information addition request.

[0135] Next, the storage and read processing unit 69 of the communication management server 6 retrieves the corresponding session ID by searching the session management DB 6002 (see Figure 13(b)) using the connection ID received in step S41 as the search key (S42).

[0136] Next, the creation unit 66 creates a unique image data ID (S43). Then, the storage and reading processing unit 69 stores the session ID read in step S42, the image data ID generated in step S43, and the source connection ID and image type information received in step S41 as a new record in the image type management DB 6003 (see Figure 13(c)) (S44). Furthermore, the storage and reading processing unit 69 stores the display information, which is one record of data, in the display management DB 6004 (see Figure 14) (S45). This enables centralized management of the display information of each communication device 1.

[0137] Next, the transmitting / receiving unit 61 of the communication management server 6 transmits the image data ID generated in step S43 to the communication device 1a (S46). As a result, the transmitting / receiving unit 31a of the communication device 1a receives the image data ID.

[0138] Next, the storage and reading processing unit 39a of the communication device 1a stores the image data ID received in step S46 in the image type management DB 3003a (see Figure 8) in association with the connection ID and image type information of its own device (communication device 1a) (S47).

[0139] Furthermore, if there are other communication devices participating in the same virtual conference room, the communication management server 6 needs to share information with these devices by transmitting the image data ID, connection ID, and image type information of communication device 1a. Therefore, the transmitting / receiving unit 61 of the communication management server 6 sends an information addition notification to other communication devices (for example, communication device 1b) participating in the same virtual conference room as communication device 1a (S48). This information addition notification includes the image data ID generated in step S43, as well as the connection ID and image type information of the source communication device 1a received in step S51. As a result, the transmitting / receiving unit 11b of communication device 1b receives the information addition notification. The destination of the transmitting / receiving unit 61 is another connection ID associated with the same session ID as the connection ID of communication device 1a in the session management DB 6002 (see Figure 13(b)). In other words, the destination is another communication device in the same virtual conference room as communication device 1a. Also, in step S48, the communication management server 6 does not transmit the display information received in step 41. This is because the display information is not used by other communication devices.

[0140] Next, the storage and reading processing unit 39b of the communication device 1b stores the image data ID received in step S48, the connection ID of the source communication device 1a, and the image type information as a new record in the image type management DB 3003b (see Figure 8) (S49).

[0141] Furthermore, after sending the information addition notification, the communication management server 6 will send the connection ID and image type information of other communication devices that are already participating in the same virtual conference room to communication device 1 that has joined the same virtual conference room as communication device 1a, at timings such as step S46 above.

[0142] Therefore, all communication devices participating in the same virtual space can share image type information with other communication devices.

[0143] (Management of image type information for smartphones) Next, we will explain the image type information management process for smartphones using Figure 23.

[0144] First, user G1 at site G enters their connection ID and password on the login screen displayed on the display 918 by the display control unit 94 (see Figure 19(b)) and presses the "OK" button. The reception unit 92 then receives the connection ID and password, and the transmission / reception unit 91 sends a login request to the communication management server 6 (S61). This login request includes the connection ID, password, and the IP address of the source (smartphone 9). As a result, the transmission / reception unit 61 of the communication management server 6 receives the login request.

[0145] Next, the storage and retrieval processing unit 69 of the communication management server 6 searches the communication management DB 6001 (see Figure 13(a)) using the connection ID and password included in the login request as search keys, and performs authentication by determining whether the same connection ID and password combination is managed (S62). Here, we will continue the explanation assuming that the same connection ID and password combination is managed by the storage and retrieval processing unit 69. In this case, the storage and retrieval processing unit 69 appends the source IP address received in step S61 to the record of the connection ID and password combination used for authentication.

[0146] Next, the transmitting / receiving unit 61 sends a response to the smartphone 9 indicating that authentication was successful (S63). This response includes virtual space information. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the response.

[0147] Next, when user G1 at base G operates the control unit 815 of the 360-degree imaging device 8, the reception unit 82 receives the operation, the storage / reading processing unit 89 reads the GUID of the device (360-degree imaging device 8) stored in the storage unit 8000, and the communication unit 88 transmits the GUID of the device to the communication unit 98 of the smartphone 9 (S64). As a result, the communication unit 98 of the smartphone 9 receives the GUID of the 360-degree imaging device 8.

[0148] Next, the determination unit 95 of the smartphone 9 determines the image type (S65) by determining whether the same vendor ID and product ID as those in the GUID received in step S64 are managed in the imaging device management DB 9001 (see Figure 17). Specifically, if the same vendor ID and product ID are managed in the imaging device management DB 9001, the determination unit 95 determines that the sender's imaging device (in this case, the 360-degree imaging device 8) is an imaging device that acquires 360-degree images. On the other hand, if the same vendor ID and product ID are not managed in the imaging device management DB 9001, the determination unit 95 determines that the source imaging device is an imaging device that takes general images.

[0149] Next, the storage and retrieval processing unit 99 stores the connection ID of the source device (smartphone 9) and the image type information, which is the result of the determination made in step S65, in association with the image type management DB 9003 (see Figure 8) (S66). In this state, the image data ID is not associated.

[0150] From this point onward, the processing in steps S67 to S75 is essentially the same as in steps S41 to S49 described above, except that the communication device 1a is replaced by the smartphone 9; all other processing is the same.

[0151] Specifically, the transmitting / receiving unit 91 of the smartphone 9 sends an information addition request to the communication management server 6 (S67). This information addition request includes the connection ID of the transmitting device (smartphone 9), image type information, and display information. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the information addition request.

[0152] Next, the storage and read processing unit 69 of the communication management server 6 retrieves the corresponding session ID by searching the session management DB 6002 (see Figure 13(b)) using the connection ID received in step S67 as the search key (S68).

[0153] Next, the creation unit 66 creates a unique image data ID (S69). Then, the storage and reading processing unit 69 stores the session ID read in step S68, the image data ID generated in step S69, and the source connection ID and image type information received in step S67 as a new record in the image type management DB 6003 (see Figure 13(c)) (S70). Furthermore, the storage and reading processing unit 69 stores the display information, which is one record of data, in the display management DB 6004 (see Figure 14) (S71).

[0154] Next, the transmitting / receiving unit 61 of the communication management server 6 transmits the image data ID generated in step S69 to the smartphone 9 (S72). As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the image data ID.

[0155] Next, the memory and read processing unit 39a of the smartphone 9 stores the image data ID received in step S72 in the image type management DB 3003a (see Figure 8), associating it with the connection ID and image type information of its own device (smartphone 9) (S73).

[0156] Furthermore, if there are other communication devices participating in the same virtual conference room, the communication management server 6 needs to share information with these devices by transmitting the image data ID, connection ID, and image type information of the smartphone 9. Therefore, the transmitting / receiving unit 61 of the communication management server 6 sends an information addition notification to other communication devices (for example, communication device 1b) participating in the same virtual conference room as the smartphone 9 (S74). This information addition notification includes the image data ID generated in step S69, as well as the connection ID and image type information of the source smartphone 9 received in step S51. As a result, the transmitting / receiving unit 11b of communication device 1b receives the information addition notification. The destination of the transmitting / receiving unit 61 is another connection ID associated with the same session ID as the connection ID of the smartphone 9 in the session management DB 6002 (see Figure 13(b)). In other words, the destination is another communication device in the same virtual conference room as the smartphone 9.

[0157] Next, the memory and read processing unit 39b of the communication device 1b stores the image data ID received in step S74, the connection ID of the source smartphone 9, and the image type information as a new record in the image type management DB 3003b (see Figure 8) (S75).

[0158] Furthermore, after sending the information addition notification, the communication management server 6 will, at the timing of step S72, etc., send the connection ID and image type information of other communication devices that are already participating in the same virtual conference room to communication device 1 that has joined the same virtual conference room as smartphone 9.

[0159] Therefore, all communication devices and smartphones 9 participating in the same virtual space can share image type information with other communication devices and smartphones 9.

[0160] <Transmission of image data and sound data> Next, the transmission process of image data and sound data will be explained using Figures 24 to 26. Figure 24 is a sequence diagram showing the process from when the luminance image data and sound data transmitted from the 360-degree imaging device reach each communication device. Figure 25 is a sequence diagram showing the process from when the luminance image data, depth image data, and sound data transmitted from a predetermined communication device reach other communication devices. Figure 26 is a conceptual diagram of the transmitted luminance image data and depth image data.

[0161] (Transmission from smartphone to each communication device) First, using Figure 24, we will explain the process when image data (in this case, luminance image data) and sound data are transmitted from the smartphone 9 to each communication device 1 via the communication management server 6.

[0162] First, the 360-degree camera 8 starts shooting and collecting sound at base G, causing the shooting unit 83 to acquire image data (in this case, two luminance image data of a hemispherical image) and the sound collection unit 84 to acquire sound data (S81). In this case, the 360-degree camera 8 performs video recording, but it may also perform still image recording. Then, the communication unit 88 transmits the two luminance image data and sound data to the communication unit 98 of the smartphone 9 (S82). As a result, the communication unit 98 of the smartphone 9 receives the two luminance image data and sound data.

[0163] Next, the transmitting / receiving unit 91 of the smartphone 9 transmits to the communication management server 6 the two brightness image data and sound data sent from the 360-degree camera 8, as well as the connection ID of the smartphone as the source (S83). In this case, the two brightness image data include the image data ID received in step S72. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the two brightness image data (and image data ID), sound data, and the smartphone's connection ID.

[0164] Next, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 13(b)) to transmit two luminance image data files (and image data IDs), sound data, and the smartphone's connection ID to each communication device 1a to 1d participating in the same video call as the smartphone 9 (S84 to S87). As a result, the transmitting / receiving units 31a to 31d of each communication device 1a to 1d receive the two luminance image data files (and image data IDs), sound data, and the smartphone's connection ID.

[0165] (Transmission from a designated communication device to other communication devices and smartphones) Next, using Figure 25, we will explain the process when communication device 1a transmits image data (in this case, luminance image data and depth image data) and sound data to multiple other communication devices 1b, 1c, and 1d via the communication management server 6.

[0166] First, the communication device 1a starts shooting and sound collection at base A, at which point the luminance image acquisition unit 36a acquires luminance image data, the depth image acquisition unit 37a acquires depth image data, and the sound collection unit 38a acquires sound data (S101). In this case, the communication device 1a performs video recording, but it may also perform still image recording.

[0167] Then, the transmitting / receiving unit 31a of the communication device 1a transmits to the communication management server 6 image data (luminance image data, depth image data) and sound data, as well as the connection ID of the communication device 1a as the source (S102). In this case, the image data includes the image data ID received in step S46 above. As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of the communication device 1a as the source.

[0168] Here, the transmitted luminance image data and depth image data will be explained using Figures 4 and 26. In Figure 4, since a shooting unit 105 for taking self-portraits is provided, the image data includes data for the luminance image Lc1 and data for the depth image Dc1. The data for the luminance image Lc1 and depth image Dc1 are acquired by the shooting unit 105.

[0169] For example, the resolution of the luminance image Lc1 and depth image Dc1 data is 4K each. However, as shown in Figure 26, the transmitting communication device 1 compresses the two data sets into a single image data, resulting in a single image data with a resolution of 4K. Note that 4K is just one example; HD, FHD, 8K, etc., may also be used.

[0170] Next, returning to Figure 25, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 13(b)) to transmit image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of the source communication device 1a to each of the other communication devices 1b to 1d participating in the same video call as communication device 1a (S103 to S105). As a result, the transmitting / receiving units 31b, 31c, and 31d of each of the communication devices 1b, 1c, and 1d receive the image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of the source communication device 1a.

[0171] Since smartphone 9 is for transmission only, if the source is communication device 1, the communication management server 6 will not send image data, sound data, or the connection ID of the source communication device 1a to smartphone 9. However, if the smartphone is used for display purposes in the same way as communication device 1, the communication management server 6 will not send image data, sound data, or the connection ID of the source communication device 1a to smartphone 9.

[0172] <Receiving image and audio data> Next, the processing of receiving image data and sound data will be explained using Figures 27 to 37. Figure 27 is a sequence diagram showing the processing of receiving image data and sound data by the communication device 1d.

[0173] As shown in Figure 27, the transmitting / receiving unit 31a of the communication device 1a transmits image data (luminance image data, depth image data, image data ID), sound data, and the connection ID of the communication device 1a as the source to the communication management server 6 (S121). As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data, sound data, and the connection ID of the communication device 1a. This step S121 is the same as step S102 in Figure 25. Then, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 13(b)) to transmit image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of the communication device 1a as the source to other communication devices 1d participating in the same video call as the communication device 1a (S122). As a result, the transmitting / receiving unit 31d of the communication device 1d receives the image data, sound data, and the connection ID of the communication device 1a. This process is the same as step S103 in Figure 25.

[0174] Similarly, the transmitting / receiving unit 31b of communication device 1b transmits image data (luminance image data, depth image data, image data ID), sound data, and the connection ID of communication device 1b as the source to the communication management server 6 (S123). As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data, sound data, and the connection ID of communication device 1b. Then, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 13(b)) to transmit image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of communication device 1b as the source to other communication devices 1d participating in the same video call as communication device 1b (S124). As a result, the transmitting / receiving unit 31d of communication device 1d receives the image data, sound data, and the connection ID of communication device 1b.

[0175] Furthermore, the transmitting / receiving unit 31c of communication device 1c transmits image data (luminance image data, depth image data, image data ID), sound data, and the connection ID of communication device 1c as the source to the communication management server 6 (S125). As a result, the transmitting / receiving unit 61 of the communication management server 6 receives the image data, sound data, and the connection ID of communication device 1c. Then, the transmitting / receiving unit 61 of the communication management server 6 uses the session management DB 6002 (see Figure 13(b)) to transmit image data (luminance image data, depth image data, and image data ID), sound data, and the connection ID of communication device 1c as the source to other communication devices 1d participating in the same video call as communication device 1c (S126). As a result, the transmitting / receiving unit 31d of communication device 1d receives the image data, sound data, and the connection ID of communication device 1b.

[0176] As a result, the communication device 1d can acquire image data and sound data from other locations A, B, and C.

[0177] Next, in the communication device 1d, the image and sound processing unit 33d performs image display and sound output processing based on the data received in steps S122, S124, 126 and step S87 in Figure 24 (S127). In this case, sound output is a general process of converting each sound data (electrical signal) into vibration, so the image display process will be explained in detail using Figure 28.

[0178] <Image display processing> Figure 28 is a flowchart illustrating the image display process.

[0179] First, the memory and read processing unit 39d searches the image type management DB 3003d using the image data ID received in step S87 in Figure 24 and steps S122, S124, and S126 in Figure 27 as a search key, and reads the corresponding image type information (S201). If the image type information is "Video_Depth" (S202; brightness and depth image), the process proceeds to step S203. If the image type information is "Video_Omni" (S202; 360-degree image), the process proceeds to step S211. If the image type information is "Video_Omni" (S202; planar image), the process proceeds to step S203.

[0180] (Display processing when the image type is a brightness / depth image) Here, we will explain the display process when the image type is a brightness / depth image.

[0181] ((Process for calculating the position of an object in the world coordinate system)) First, the communication device 1d calculates the position of the object in the world coordinate system (S203). Here, the display process when the image type is a luminance / depth image will be explained in detail using Figures 29 to 32. Figure 29 is a flowchart showing the process of calculating the position of the object in the world coordinate system.

[0182] First, the image and sound processing unit 33d creates a 3D image of an object based on the luminance image data and depth image data of each location, including its own location, in the virtual space (modeling coordinate system) (S221).

[0183] Here, we will use Figure 30 to explain the 3D rendering of an object's image. Figure 30 is a conceptual diagram of the transformation from the camera coordinate system to the modeling coordinate system, and from the modeling coordinate system to the world coordinate system. The camera coordinate system is a coordinate system that defines how the world appears on the projection plane, with the physical camera as the origin, representing the 3D world of the real world. It is necessary to transform the coordinates given in real coordinates into the coordinate system seen in the camera's coordinate system.

[0184] In order to display images of users from multiple physical locations in a single virtual space such as a virtual conference room, the image and sound processing unit 33d must first define the objects in the physical space of each location individually in a Modeling Coordinate System as the first virtual space, thereby creating a 3D model, and then define the image of each object in a single World Coordinate System.

[0185] Therefore, first, the image / sound processing unit 33d performs 3D modeling by transforming five sets of images, such as the luminance image La and depth image Da, defined in the Camera Coordinate System as shown in Figure 26, into a single first virtual space (modeling coordinate system) using (Equation 1). In Figure 30, user A1 is represented as an object. In this case, the image of user A1 in the modeling coordinate system is represented by a collection of many points, as shown in the enlarged view ev of Figure 30. In this embodiment, since images are captured from five angles by five luminance / depth imaging devices 5a to 5e, the image of user A1 is represented as a collection of many overlapping points. In this way, users A1, B1, C1, and D1 are each defined independently in separate modeling coordinate systems (3D modeled).

[0186]

number

[0187] Vector P is a projection matrix, and vector P-1 is a matrix for transforming from the camera coordinate system to the real coordinate system. Vector M is a matrix for transforming from the real coordinate system to the modeling coordinate system. This output value shows the point (coordinates) and color information in the modeling coordinate system.

[0188] Next, returning to Figure 29, the memory / read processing unit 39d reads the display information of its own device (communication device 1d) stored in the memory unit 3000d, and the orientation sensor (detection unit 30a) outputs rotation matrix (0) data (S222). Then, the image / sound processing unit 33d places the virtual display (also called a virtual screen) dd of its own location in the modeling coordinate system based on the "information on the position of each corner of the display in the virtual space (modeling coordinate system)" in the display information (S223). Figure 31 is a conceptual diagram showing the image of user D1 and the virtual display placed in the modeling coordinate system. For example, as shown in Figure 31, the image / sound processing unit 33d further places (defines) a virtual display dd1 on the image of user D1 at its own location. In this case, the virtual display dd1 is rotated in pan, tilt, and roll based on the rotation matrix (0) data output by the orientation sensor (detection unit 30a). In other words, this indicates that the direction in which user A1 rotates the actual communication device 1a is reflected on the virtual display dd1.

[0189] Returning to Figure 29, the memory / reading processing unit 39d reads the "transformation parameters from modeling coordinate system to world coordinate system" associated with the connection IDs of users (communication devices) participating in the same virtual conference room from the placement management DB 3006d (see Figure 9) (S224). Then, the image / sound processing unit 33d performs a modeling transformation on the images of objects at each location and the virtual displays (virtual peepholes) at each location, from the modeling coordinate system to the world coordinate system in the virtual space (S225).

[0190] Here, we will again use Figure 30 to explain the transformation from the modeling coordinate system to the world coordinate system.

[0191] As shown in Figure 30, the image and sound processing unit 33d performs a Modeling Transformation on the image of user A1 in the modeling coordinate system to a common second virtual space (world coordinate system) using (Equation 2). Note that the Modeling Transformation is a transformation process for placing display objects defined in the modeling coordinate system onto the world coordinate system, and represents a combination of four transformation processes: scaling, rotation, shearing, and translation of the display objects.

[0192]

number

[0193] Vector W is a matrix used to transform from the modeling coordinate system to the world coordinate system. Its output represents the point (coordinates) and color information in the world coordinate system. Vector O is a matrix used for rotation (pan, tilt, roll).

[0194] Figure 32 is an illustrative diagram showing the transformation from the modeling coordinate system to the world coordinate system. As shown in Figure 32, the image and sound processing unit 33d transforms the images of users A1 to D1 in each modeling coordinate system and the virtual display dd4 at its own location into a single common world coordinate system. This allows the images of users A1 to D1 at each location to participate in a common virtual conference room.

[0195] As shown in Figure 32, the world coordinate system of this embodiment includes a virtual spherical wall cw for pasting a 360-degree image and a virtual planar wall pw for pasting a 3D planar image. These will be discussed later.

[0196] ((Display of images visible to the user)) Next, returning to Figure 28, we will explain the process of displaying an image that user D1 can see on communication device 1.

[0197] As shown in Figure 28, the image and sound processing unit 33a detects skeletal information such as the eyes, nose, and shoulders of user D1 based on the luminance image data acquired by the luminance image acquisition unit 36d and the luminance image data acquired by the depth image acquisition unit 37d (S204).

[0198] Next, the image and sound processing unit 33a uses the center of both eyes in the skeletal information as the viewpoint position and, based on this viewpoint position, creates images of other users such as user A1 that user D1 can see on the virtual display dd in the world coordinate system, as shown in the center of Figure 32 (S205). In other words, the image and sound processing unit 33a converts the displayed objects in the world coordinate system into images that can be displayed on the virtual display dd based on the viewpoint position. This conversion means a conversion from the world coordinate system to the viewpoint coordinate system and is called a field of view conversion. The viewpoint coordinate system is a coordinate system that has the camera position (viewpoint) in the virtual space as its origin and extends in the direction that the user wants to see (the direction in which the subject exists).

[0199] In the world coordinate system, the intersection points of rays (rays) and polygons (virtual triangles, polygons, etc. that make up a 3D model) are calculated using ray tracing, which travels from the viewpoint to pixels on the virtual display dd. These rays are traced backward from the viewpoint, following the light rays traveling from the image direction. This method is called ray tracing and is also called reverse line of sight tracing because it traces the line of sight backward. The intersection point closest to the viewpoint, i.e., the object (called a "polygon" in the field of computer graphics) that first intersects the ray, is found, and the pixels on the virtual display are painted with the color of that object. If no object (polygon) intersects the ray, the pixel is painted with the background color. By performing this process for all pixels on the virtual display dd, the image displayed on the virtual display dd is created.

[0200] In addition to ray tracing, other methods such as the Z-buffer method also exist, and the methods for creating images displayed on the virtual display dd are not limited to those described above.

[0201] Next, the image / sound processing unit 33a converts the image created in the virtual space (world coordinate system) into an image for display on the displays 4a-4c of the communication device 1d in physical space (S206). Then, the display control unit 34d displays the image converted in step S206 on the displays 4a-4c of the communication device 1d in physical space (S207). In this case, the space in which the image is displayed on displays 4a-4c is a two-dimensional planar image, and the resolution depends on the performance of the computer, especially on devices such as the display memory installed in the video board (or graphics board), and is therefore called the device coordinate system.

[0202] (Display processing when the image type is a 360-degree image) Next, we will explain the display process when the image type is a 360-degree image.

[0203] The image / sound processing unit 33d creates a spherical image from the two luminance image data received by the transmitting / receiving unit 11d, and places this spherical image onto the virtual space (world coordinate system) (S211). In this case, the image / sound processing unit 33d combines the two luminance images, which are hemispherical images, to create a single equirectangular projection image EC. Then, the image / sound processing unit 33a uses OpenGL ES (Open Graphics Library for Embedded Systems) to place the equirectangular projection image onto the virtual sphere (in this case, the inner wall surface of the virtual spherical wall cw in Figure 32) to create the spherical image data. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data.

[0204] Then, after processing in step S211, the process proceeds to step S206.

[0205] (Display processing when the image type is a flat image) Next, the display process when the image type is a planar image will be described. In this case, even if it is a planar image, the image / sound processing unit 33a performs the processing from step S203 onwards.

[0206] <Example of image display> Next, using Figures 34 to 37, we will explain various display examples from steps S207, S212, and S214 in Figure 28.

[0207] (Example of display shown to meeting participants (user D1)) ((Initial display)) In Figure 33, (a) shows the line of sight of user D1 in the world coordinate system in the case of a virtual meeting, and (b) is an example of the image that the communication device 1d shows to user D1 in the case of (a).

[0208] Here, we show a case where user D1 uses the communication device 1d directly in front of them, so that the self-portrait shooting unit 105 in the communication device 1d takes a picture from the direction directly in front of user D1 (an example of a second direction).

[0209] As shown in Figure 33(a), in the virtual space (in this case, the world coordinate system), user A1 is located in a virtual peephole wa slightly behind and to the left of user D1's line of sight, user B1 is located in a virtual peephole wb directly in front, and user C1 is located in a virtual peephole wc to the left. In principle, only one user exists at a single location in the virtual space (world coordinate system).

[0210] Then, when user D1 in the physical space views this state in the virtual space on the display 118, as shown in Figure 33(b), the display control unit 34d displays a composite image on displays 4a to 4c, which is a combination of the 360-degree spherical image CE as the background image, and the images of the virtual peepholes wa, wb, wc and users A1, B1, C1, which differ in size and display direction.

[0211] Note that each user's image is the image displayed by the display control unit 34d in step S207 of Figure 28. Similarly, the 360-degree image CE is the image displayed by the display control unit 34d in step 212 of Figure 28. In practice, each user's image and the 360-degree image CE are displayed as a composite image as described above. The display of each user's image and the 360-degree image CE is the same hereafter.

[0212] Specifically, when user D1 views user A1's image, user A1's image is displayed in a virtual viewing window wa located slightly behind and to the left and in front of the user. It is displayed in a predetermined area on display 118 such that it is smaller in size than user B1's image and the display area on the left shoulder is larger than that on the right shoulder. Similarly, when user D1 views user B1's image, user B1's image is displayed in a virtual viewing window wb located closer than user A1's image, which is directly in front of the user. It is displayed in a predetermined area on display 118 such that it is larger in size than user A1's image and the front of user B1's face is displayed. Furthermore, when user D1 views user C1's image, user C1's image is displayed in a virtual viewing window wc located to the left, in a predetermined area on 118 such that the left side of user C1's face (as viewed by user D1) is displayed. However, the entire left side of user C1's face is not displayed. As shown in Figure 1, because user C1 uses the communication device 1c directly in front of him, when the self-portrait shooting unit in Figure 4 takes a picture from directly in front of user C1 (an example of the first direction), the entire left side of user C1's face cannot be captured. Therefore, when the display control unit 34d displays an image with the left side of the face cut off, user D1 viewing this image will feel uncomfortable. In this embodiment, however, an image of a 3D object in virtual space is deliberately displayed in a virtual peephole, and the fixed idea that it is difficult to see the entire subject through the peephole is utilized to suppress the viewer's discomfort and prevent a diminished sense of being in the same room.

[0213] ((Displayed after rotation)) Next, we will explain an example of the display when user D1 rotates (pans) slightly to the left (in the direction of the arrow) along with their face (line of sight) in physical space, as shown in Figure 34(b), starting from the state in Figure 33(b). In Figure 34, (a) is a diagram showing the line of sight of user D1 after movement in the world coordinate system in the case of a virtual meeting, and (b) is an example of the display of the image that communication device 1d shows to user D1 in the case of (a).

[0214] As shown in Figure 34(a), in the virtual space (world coordinate system), in the line of sight of user D1, user A1 appears to have moved slightly to the right compared to the case in Figure 33(a), user B1 appears to have moved slightly to the right of the front compared to the case in Figure 33(a), and user C1 appears to have moved slightly to the left of the left side compared to the case in Figure 33(a).

[0215] Then, when user D1 in the physical space views this state in the virtual space on the display 118, as shown in Figure 34(b), the display control unit 34d displays on the display 118 a 360-degree image CE as a background image that has been moved slightly to the right compared to Figure 33(b), as well as a composite image which is a combination of images of each user A1, B1, and C1 that are slightly different in size and display area compared to Figure 33(b).

[0216] Specifically, when user D1 views the 360-degree image CE, the 360-degree image CE is displayed on the display 118 in such a way that the image is slightly shifted to the right compared to the state shown in Figure 33(a). Similarly, when user D1 views user A1's image, user A1's image is displayed in a predetermined area on the display 118 in such a way that the image is slightly positioned to the right and slightly larger compared to the state shown in Figure 33(a). When user D1 views user B1's image, user B1's image is displayed in a predetermined area on the display 118 in such a way that the image is slightly shifted to the right from directly in front and slightly smaller compared to the state shown in Figure 33(a). When user D1 views user C1's image, user D1's image is displayed in a predetermined area on the display 118 in such a way that the image is slightly shifted to the left from the left side and slightly larger compared to the state shown in Figure 33(a).

[0217] In this case as well, the display of virtual viewing windows wa, wb, and wc helps to reduce any sense of unease for the viewer.

[0218] (Example of a display to show to bystanders of a meeting) In Figure 35, (a) shows the viewpoint of an observer in the world coordinate system in the case of a virtual meeting, and (b) is an example of the image displayed on a smartphone in the case of (a).

[0219] As shown in Figure 35(a), in the virtual space (world coordinate system), beyond the line of sight of the bystander (camera) G2, user A1 is located in the background to the right and slightly ahead, user B1 is located further to the right and slightly ahead, user C1 is located in the background to the left and slightly ahead, and user D1 is located further to the left and slightly ahead, slightly ahead. Note that in the case of a bystander, their own image is not displayed on each communication device. Therefore, in the case of a bystander, multiple users may exist at a single location in the virtual space (world coordinate system).

[0220] Then, as shown in Figure 35(b), the state in this virtual space is displayed on the display 918 by the display control unit 94 as a composite image, which is a combination of the 360-degree image CE as the background image and images of each user A1, B1, C1, and D1 with different sizes and display directions. In the case of a bystander, their image is not captured like that of users A1, B1, C1, and D1, so the processing in steps S204 to S207 in Figure 28 is not executed. That is, even if bystander G2 moves the smartphone 9 up, down, left, or right, or moves their viewpoint up, down, left, or right relative to the smartphone 9, the display control unit 94 displays the same image.

[0221] Specifically, on display 918, the image of user A1 is located in a predetermined area at the back, diagonally to the right, smaller in size than user B1, and larger on the left shoulder than on the right shoulder. Also on display 918, the image of user B1 is located in a predetermined area at the front, diagonally to the right, further to the right than user A1, larger in size than user A1, and larger on the left shoulder than on the right shoulder. Furthermore, on display 918, the image of user C1 is located in a predetermined area at the back, diagonally to the left, smaller in size than user D1, and larger on the right shoulder than on the left shoulder. Also on display 918, the image of user D1 is located in a predetermined area at the front, diagonally to the left, further to the left than user C1, larger in size than user C1, and larger on the right shoulder than on the left shoulder.

[0222] In this case as well, the display of virtual viewing windows wa, wb, wc, and wd helps to reduce the viewer's sense of unease. The image of user D1 is displayed within the virtual viewing window wd.

[0223] Furthermore, bystander G2 may be user G1 in Figure 1, or it may be a different user. Also, if there are multiple bystanders, multiple users will be participating in the virtual event, such as a virtual meeting, as bystanders at the same location.

[0224] (Example of a display shown to the interviewee (user D1) during the interview) In Figure 36, (a) shows the line of sight of user D1 in the world coordinate system in the case of a virtual interview, and (b) is an example of the image that the communication device 1d shows to user D1 in the case of (a).

[0225] As shown in Figure 36(a), in the virtual space (world coordinate system), user A1 is positioned diagonally to the left in front of user D1's line of sight, user B1 is directly in front, and user C1 is positioned diagonally to the right. Note that, in principle, only one user exists at a single location in the virtual space (world coordinate system).

[0226] Then, when user D1 in the physical space views this state in the virtual space on the display 118, as shown in Figure 36(b), the display control unit 34d displays on the display 118 a composite image which is a combination of the 360-degree spherical image CE as the background image and images of each user A1, B1, and C1 that differ in size and display direction.

[0227] Specifically, when user D1 views user A1's image, user A1's image is displayed in a predetermined area on display 118 such that it is positioned diagonally to the left and slightly smaller in size than user B1's image. Similarly, when user D1 views user B1's image, user B1's image is displayed in a predetermined area on display 118 such that it is positioned directly in front of user B1. Furthermore, when user D1 views user C1's image, user C1's image is displayed in a predetermined area on display 118 such that it is positioned diagonally to the right and slightly smaller in size than user B1's image.

[0228] In this case as well, the display of virtual viewing windows wa, wb, and wc helps to reduce any sense of unease for the viewer.

[0229] (Example of a display shown to a lecture participant (user D1)) In Figure 37, (a) shows the line of sight of user D1 in the world coordinate system in the case of a virtual lecture, and (b) is an example of the image that communication device 1d shows to user D1 in the case of (a).

[0230] As shown in Figure 37(a), in the virtual space (world coordinate system), user A1 is located to the left of user D1's line of sight, user B1 is slightly behind and to the left and in front of user D1, and user C1 is located to the right. In principle, only one user exists at a single location in the virtual space (world coordinate system).

[0231] Then, when user D1 in the physical space views this state in the virtual space on the display 118, as shown in Figure 37(b), the display control unit 34d displays on the display 118 a composite image which is a combination of the planar image PE as the background image and images of each user A1, B1, and C1 that differ in size and display direction.

[0232] The planar image PE is the image displayed by the display control unit 34d in step S214 of Figure 28. Each user's image and the planar image PE are actually displayed as a composite image, as described above. Note that the image used for the lecture may be a 360-degree spherical image CE instead of a planar image PE.

[0233] Specifically, when user D1 views user A1's image, user A1's image is displayed in a predetermined area on display 118, positioned to the left, so that the right side of the face is visible. Similarly, when user D1 views user B1's image, user B1's image is displayed in a predetermined area on display 118, positioned slightly behind and to the left, so that it is smaller in size than users A1 and C1's images. Furthermore, when user D1 views user C1's image, user C1's image is displayed in a predetermined area on display 118, positioned to the right, so that the left side of the face is visible.

[0234] In this case as well, the display of virtual viewing windows wa, wb, and wc helps to reduce any sense of unease for the viewer.

[0235] [Main effects of the embodiment] As explained above, according to this embodiment, even when displaying an incomplete image of a subject at another location, by deliberately displaying the image of a 3D object in virtual space within a virtual viewing window and utilizing the fixed idea that it is difficult to see the entire subject through the viewing window, it is possible to suppress the loss of a sense of reality.

[0236] Furthermore, for example, communication device 1a displays images based on the image data transmitted by other communication devices 1b, 1c, and 1d in a predetermined area on the display 4, according to a predetermined position (for example, the viewpoint position between both eyes) of an object (user D1) in the virtual space (world coordinate system) on the side of its own device (communication device 1d) and the changeable positional relationship (also called "arrangement relationship") between multiple communication devices in the virtual space (world coordinate system) (see Figures 10 to 12) (see Figures 33(b), 34(b), 36(b), and 37(b)). This has the effect of resolving as much as possible the problem that it is difficult for users at multiple locations to obtain a sense of being in the same room according to the number of locations and the purpose of communication.

[0237] Furthermore, for example, communication device 1d displays multiple types of information with different positional relationships between multiple communication devices in the virtual space (see Figure 20), and accepts the selection of a specific type (e.g., a conference room) from among these multiple types with different positional relationships between multiple communication devices in the virtual space. Then, communication device 1d displays an image based on image data transmitted by other communication devices in a predetermined area on the display 4 corresponding to the specific type (for example, see Figure 33(b)). This has the effect of making it easier for users to select a virtual space according to the purpose of communication.

[0238] 〔supplement〕 The above-mentioned communication management server 6 may simply be called a "server." The communication management server 6 may also be provided in the form of cloud computing. Cloud computing refers to a usage model in which computer resources are provided as a service via a computer network such as the internet, and the provided computer resources are servers. The form of service provided can be SaaS (Software as a Service), PaaS (Platform as a Service), HaaS or IaaS (Hardware / Infrastructure as a Service), but the form of service provision is not limited.

[0239] Furthermore, in the above embodiment, the virtual space and role are determined by the user selecting selection buttons b1 to b3 in Figure 20 and selection buttons b11 to b33 in Figures 21(a) to (c), but this is not the only way. For example, by pre-setting the virtual space type and role for each connection ID in the communication management server 6, user A1 may not need to specifically select selection buttons b1 to b3 and selection buttons b11 to b33, and the communication management server 6 may automatically determine the virtual space type and role.

[0240] Furthermore, in the above embodiment, as shown in Figure 30, each communication device 1 first defines the objects in the physical space of each location individually in a modeling coordinate system as a first virtual space to create a 3D model, and then converts the image of each object into a single world coordinate system. However, this is not the only method. For example, the communication management server 6 may first define the objects in the physical space of each location individually in a modeling coordinate system as a first virtual space to create a 3D model, and then convert the image of each object into a single world coordinate system. In this case, the communication management server 6 transmits the image data converted to the world coordinate system to the communication device 1 at each location. That is, a communication system comprising a first communication device (e.g., communication device 1d), a second communication device (e.g., communication device 1a), and a server (e.g., communication management server 6) includes a providing means that provides a display image generated according to a changeable positional relationship between the position of a subject in the first communication device in a virtual space and the position of a subject in the second communication device in a virtual space, using image data captured by the first communication device and image data captured by the second communication device, and a display control means that displays the display image provided by the providing means.

[0241] Furthermore, as shown in Figure 26, communication device 1a compresses and combines a total of 10 data points—multiple luminance images La to Le and depth images Da to De—into a single image data before transmitting it to the communication management server 6. However, it is not limited to this. For example, communication device 1 may transmit the total of 10 data points directly to the communication management server 6, and the communication management server 6 may compress and combine the total of 10 data points into a single image data before transmitting it to the other communication devices 1b to 1d (see S103 to S105). Note that the same processing is performed not only when the source is communication device 1a, but also when the source is communication devices 1b to 1d.

[0242] Multiple CPUs, as described in the above embodiment, may be provided in a single device.

[0243] Moreover, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the "processing circuit" in this specification refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit) designed to execute each function described above, a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU (Graphics Processing Unit), or a device such as a conventional circuit module.

[0244] Furthermore, in the communication between each communication device 1a to 1d, the communication management server 6, the omnidirectional imaging device 8, and the smartphone 9, other servers, routers, etc. may relay data.

Description of Reference Numerals

[0245] 1a Communication device (an example of another communication device, an example of a first other communication device) 1b Communication device (an example of another communication device, an example of a second other communication device) 1c Communication device (an example of another communication device) 1d Communication device (an example of the self-device) 4a Display (an example of display means) 4d Display (an example of display means) 4c Display (an example of display means) 6 Communication management server 8 Omnidirectional imaging device 9 Smartphone 31a Transmission / reception unit (an example of transmission means) 30d Detection unit (an example of detection means) 31d Transmission / reception unit (an example of reception means) 32d Reception unit (an example of reception means) 33d Image / audio processing unit [generation unit] (an example of generation means) 34d Display control unit (an example of display control means) 36d Brightness image acquisition unit (an example of acquisition means) 37d Depth image acquisition unit (an example of acquisition method) b1 Selection button (an example of a specific type of information among multiple types of information with different positional relationships) b2 Selection button (an example of a specific type of information among multiple types of information with different positional relationships) b3 Selection button (an example of a specific type of information among multiple types of information with different positional relationships)

Claims

1. A communication device that makes video calls with other communication devices at other locations, A receiving means that receives data of brightness images and depth images of the other location obtained by another communication device at the other location photographing a subject at the other location, which is then transmitted by the other communication device. A generation means for generating a three-dimensional object of a subject at the aforementioned other location in a virtual space based on luminance image and depth image data of the aforementioned other location, A display control means that displays an image of the three-dimensional object in a virtual viewing window on a display means, A communication device characterized by having the following features.

2. A communication device according to claim 1, A means of acquiring data of brightness and depth images of a location by photographing a subject at that location, A transmission means for transmitting the brightness image and depth image data of the said site to be received by the other communication device, It has, The generation means generates a three-dimensional object of the subject at the local site in the virtual space based on the luminance image and depth image data of the local site. The display control means displays an image of a three-dimensional object of a subject at another location on the display means such that the inside of the virtual peephole can be seen from the viewpoint of the three-dimensional object of the subject at the local location. A communication device characterized by the following.

3. A communication device according to claim 2, The communication device has a detection means for detecting the direction of the normal to the display means, A communication device characterized by displaying an image of a three-dimensional object of a subject at another location on the display means so that the inside of the virtual peephole can be seen from the viewpoint based on the detection result of the detection means.

4. The communication device according to any one of claims 1 to 3, characterized in that the communication device is a tablet terminal.

5. A communication system having other communication devices at other locations and a communication device that makes video calls with said other communication devices, The other communication device has a transmission means for transmitting data of brightness images and depth images of the other location obtained by the other communication device photographing the subject at the other location. The aforementioned communication device is A receiving means for receiving brightness image and depth image data from the aforementioned other location, A generation means for generating a three-dimensional object of a subject at the aforementioned other location in a virtual space based on luminance image and depth image data of the aforementioned other location, A display control means that displays an image of the three-dimensional object in a virtual viewing window on a display means, Having A communication system characterized by the following.

6. A display method performed by a communication device that makes a video call with another communication device at another location, A receiving step in which the other communication device at the other location receives data of brightness images and depth images of the other location obtained by another communication device at the other location taking a photograph of the subject at the other location, and transmits this data to the other communication device. A generation step of generating a three-dimensional object of the subject at the aforementioned other location in a virtual space based on the brightness image and depth image data of the aforementioned other location, A display control step of displaying an image of the three-dimensional object in a virtual viewing window on a display means, A display method characterized by performing the following:

7. A program that causes a computer to perform the method described in claim 6.

Images