Image communication system, communication terminal, communication method, providing method, and program

Abstract

To solve the problem in which: when a communication terminal which received image data obtained from a special imaging device displays the image data in accordance with a plane image, two hemispherical images that will be a source of a panoramic omnidirectional sphere image are displayed, resulting in an image which is very hard to see; and when a communication device which received image data obtained from a general imaging device performs the processing of creating a panoramic omnidirectional sphere image, the image is corrupted and becomes very hard to see.SOLUTION: A system receives image type information indicating the image type from the first communication device (S54), generates the image data identification information for identifying the image data to be transmitted from the first communication terminal to the second communication terminal on the basis of the receiving of the image type information (S56), transmits the image data identification information generated as above to the first communication terminal (S58) and transmits the generated image data identification information and the image type information received as above to the second communication terminal (S60).SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] The present invention relates to Image communication system, communication terminal, communication method, providing method, and programs. [Background technology]

[0002] Conference systems that hold remote conferences between remote locations via a communication network such as the Internet are becoming widespread. In this conference system, in a conference room where one of the parties, such as attendees, is present, images of the conference room of the parties and audio such as remarks are captured and collected using a communication terminal of the remote conference system, and these are converted into digital data and transmitted to the communication terminal of the other party. As a result, images can be displayed on a display and audio can be output from a speaker in the other party's conference room to conduct a video call, and a technology is known for holding a conference between remote locations in a state close to an actual conference (see Patent Document 1).

[0003] Meanwhile, a technology is known in which a communication terminal is connected to an image capture device capable of acquiring a full celestial sphere panoramic image in real time, the full celestial sphere panoramic image acquired from the image capture device is distributed to each communication terminal of the other party, and each communication terminal of the other party sequentially converts the full celestial sphere panoramic image into a rectangular flat image and displays the image on a display or the like (see Patent Document 2). Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the types of imaging devices used at each location are different, such as a general imaging device that captures a rectangular planar image and a special imaging device that captures two hemispherical images that are the basis of a celestial sphere panoramic image, the following problems arise on the communication terminal side that receives the image data.

[0005] That is, when a communication terminal that receives image data acquired from a special imaging device displays the two hemispherical images that are the basis of the omnidirectional panoramic image in combination with a planar image, the two hemispherical images that are the basis of the omnidirectional panoramic image are displayed, and the user A1, etc. is displayed facing downward in the spherical image as shown in Fig. 23(a), which makes the image very difficult to see. On the other hand, when a communication terminal that receives image data acquired from a general imaging device performs a process to create a omnidirectional panoramic image, the image becomes corrupted and very difficult to see. [Means for solving the problem]

[0006] The invention according to claim 1 is The images were taken by the imaging devices at each of the aforementioned locations. An image communication system for transmitting and receiving image data, comprising: 、 A planar image of a part of a special image generated from the image data; The image captured by the imaging device The image communication system has a display control means for displaying a general image on a display. Effect of the Invention

[0007] As described above, according to the present invention, the second communication terminal that receives the image data can perform image processing according to the image type information, thereby achieving the effect of displaying an image that is easy to see. [Brief description of the drawings]

[0008] [Figure 1] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. [Diagram 2] FIG. 13 is a conceptual diagram of the imaging device in use. [Diagram 3] (a) is a hemispherical image (front) taken by the photographing device, (b) is a hemispherical image (back) taken by the photographing device, and (c) is an image represented by the Mercator projection. [Figure 4] (a) A conceptual diagram showing how a sphere is covered with a Mercator image, and (b) a diagram showing a spherical panoramic image. [Diagram 5]FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere panoramic image is a three-dimensional solid sphere. [Figure 6] FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 7] 13 is a diagram showing the relationship between predetermined region information and an image of a predetermined region T. FIG. [Figure 8] 1 is a schematic diagram of an image communication system according to an embodiment of the present invention. [Figure 9] FIG. 2 is a diagram illustrating a hardware configuration of the imaging device. [Figure 10] FIG. 2 is a hardware configuration diagram of a video conference terminal. [Figure 11] FIG. 2 is a hardware configuration diagram of a communication management system and a PC. [Figure 12] FIG. 1 is a hardware configuration diagram of a smartphone. [Figure 13] FIG. 1 is a functional block diagram of an image communication system. [Figure 14] FIG. 13 is a conceptual diagram illustrating an image type management table. [Figure 15] FIG. 13 is a conceptual diagram illustrating a photographing device management table. [Figure 16] FIG. 13 is a conceptual diagram illustrating a session management table. [Figure 17] FIG. 13 is a conceptual diagram illustrating an image type management table. [Figure 18] FIG. 11 is a sequence diagram showing a process of joining a specific communication session. [Figure 19] FIG. 13 is a diagram showing a selection screen for a communication session (virtual conference room). [Figure 20] FIG. 11 is a sequence diagram showing a management process of image type information. [Figure 21] FIG. 11 is a sequence diagram showing a communication process of image data in a video call. [Figure 22] 1A shows the case where the photographing device 1a is not used, and FIG. 1B shows the case where the photographing device 1a is used. [Figure 23]23A shows a display example of the display at the site B, where FIG. 23A shows a case where the image data transmitted from the photographing devices 1a and 1b is directly displayed without creating a celestial sphere panoramic image and a predetermined area image, and FIG. 23B shows a case where the celestial sphere panoramic image and a predetermined area image are created from the image data transmitted from the photographing devices 1a and 1b. [Figure 24] FIG. 11 is a sequence diagram showing another example of the management process of image type information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] <<Outline of the embodiment>> <How to generate spherical panoramic images> A method for generating a spherical panoramic image will be described with reference to FIGS.

[0011] First, the appearance of the imaging device 1 will be described with reference to Fig. 1. The imaging device 1 is a digital camera for obtaining captured images that are the basis of a full celestial sphere (360°) panoramic image. Fig. 1(a) is a left side view of the imaging device, Fig. 1(b) is a front view of the imaging device, and Fig. 1(c) is a plan view of the imaging device.

[0012] As shown in FIG. 1(a), the photographing device 1 is sized so that a person can hold it in one hand. Also, as shown in FIG. 1(a), FIG. 1(b), and FIG. 1(c), an image sensor 103a is provided on the front side (front side) and an image sensor 103b is provided on the back side (rear side) at the top of the photographing device 1. These image sensors (image sensors) 103a and 103b are used in conjunction with optical members (for example, fisheye lenses 102a and 102b described later) capable of photographing hemispherical images (angle of view of 180° or more). Also, as shown in FIG. 1(b), an operation unit 115 such as a shutter button is provided on the surface opposite to the front side of the photographing device 1.

[0013] Next, the usage of the photographing device 1 will be described with reference to Fig. 2. Fig. 2 is an image diagram of the photographing device in use. As shown in Fig. 2, the photographing device 1 is used, for example, by a user holding it in his / her hand to photograph subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image sensor 103a and the image sensor 103b shown in Fig. 1, respectively.

[0014] Next, an outline of the process of creating a celestial sphere panoramic image from an image captured by the image capture device 1 will be described with reference to Fig. 3 and Fig. 4. Fig. 3(a) is a diagram showing a hemispherical image (front side) captured by the image capture device, Fig. 3(b) is a diagram showing a hemispherical image (rear side) captured by the image capture device, and Fig. 3(c) is a diagram showing an image expressed by the Mercator projection (hereinafter referred to as a "Mercator image"). Fig. 4(a) is a conceptual diagram showing a state in which a sphere is covered with a Mercator image, and Fig. 4(b) is a diagram showing a celestial sphere panoramic image.

[0015] As shown in Fig. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a fisheye lens 102a described below. Also, as shown in Fig. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a fisheye lens 102b described below. Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are combined by the photographing device 1 to create a Mercator image as shown in Fig. 3(c).

[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted to cover the sphere as shown in FIG. 4(a), and a celestial sphere panoramic image as shown in FIG. 4(b) is created. In this way, the celestial sphere panoramic image is represented as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere panoramic image may be a still image or a video.

[0017] As described above, since a celestial sphere panoramic image is an image pasted so as to cover a spherical surface, it gives a sense of incongruity to humans when viewed. Therefore, by displaying a predetermined region (hereinafter referred to as a "predetermined region image") of a celestial sphere panoramic image as a planar image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to humans. This will be described with reference to Figs. 5 and 6.

[0018] FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional panoramic image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user who views the omnidirectional panoramic image displayed as a three-dimensional sphere. FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing the predetermined area image when displayed on a display. In FIG. 6(a), the omnidirectional panoramic image shown in FIG. 4 is represented by a three-dimensional sphere CS. If the omnidirectional panoramic image generated in this way is a sphere CS, the virtual camera IC is located outside the omnidirectional panoramic image as shown in FIG. 5. The predetermined area T in the omnidirectional panoramic image is the shooting area of ​​the virtual camera IC, and is specified by predetermined area information indicating position coordinates (x(rH), y(rV), angle of view α(angle)) including the angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional panoramic image. Zooming of the predetermined area T can be expressed by widening or narrowing the range (arc) of the angle of view α. Zooming of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the spherical panoramic image.

[0019] Then, the predetermined area image, which is an image of the predetermined area T shown in FIG. 6(a), is displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined area image represented by the predetermined area information initially set (default). Note that the predetermined area information may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the position coordinates of the virtual camera IC. In the following, the position coordinates (x(rH), y(rV), and angle of view α(angle)) of the virtual camera IC are used for explanation.

[0020] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 7. FIG. 7 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 7, when the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC is 2L, the center point CP becomes the (x, y) parameter of the predetermined area information. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). In FIG. 7, the trigonometric function generally shown in the following formula (1) holds.

[0021] Lf = tan(α / 2) (Eq. 1) <Outline of the image communication system> Next, an outline of the configuration of the image communication system of this embodiment will be described with reference to Fig. 8. Fig. 8 is a schematic diagram of the configuration of the image communication system of this embodiment.

[0022] 8, the image communication system of this embodiment is composed of photographing devices 1a and 1b, a video conference terminal 3, a communication management system 5, a PC (Personal Computer) 7, a photographing device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

[0023] Of these, the photographing devices 1a and 1b are special digital cameras for photographing subjects, scenery, etc. to obtain two hemispherical images that are the basis of a celestial sphere panoramic image, as described above. On the other hand, the photographing device 8 is a general digital camera for photographing subjects, scenery, etc. to obtain a general planar image.

[0024] The video conference terminal 3 is a terminal dedicated to video conferences, and displays images of a video call (video call) on the display 4 via a wired cable such as a USB (Universal Serial Bus) cable. The video conference terminal 3 usually captures images with a camera 312 described later, but when the video conference terminal 3 is connected to a cradle 2a to which the photographing device 1a is attached via a wired cable, the photographing device 1a is given priority, and two hemispherical images that are the basis of a celestial sphere panoramic image can be obtained. When a wired cable is used, the cradle 2a not only performs communication between the photographing device 1a and the video conference terminal 3, but also plays a role of supplying power to the photographing device 1a and supporting the photographing device 1a. Here, the photographing device 1a, the cradle 2a, the video conference terminal 3, and the display 4 are located at the same base, base A. Also, at base A, four users A1, A2, A3, and A4 are participating in the video call.

[0025] The communication management system 5 manages the communications between the video conference terminals 3, the PCs 7, and the smartphones 9, and manages the types of image data (classification of general images and special images) transmitted and received. Here, the special images are spherical panoramic images. The communication management system 5 is installed in a service company that provides video communication services. The communication management system 5 may be constructed by a single computer, or may be constructed by multiple computers in which each section (function, means, or storage section) is divided and arbitrarily assigned.

[0026] The PC 7 is capable of video calling by having the photographing device 8 attached thereto. In this example, the PC 7 and the photographing device 8 are located at the same location, site C. Also, at site C, one user C is participating in the video call.

[0027] The smartphone 9 displays an image of the video call on a display 917 provided in the smartphone 9, which will be described later. The smartphone 9 usually captures images using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 (described later) provided in the smartphone 9, but can obtain two hemispherical image data that are the basis of the omnidirectional panoramic image obtained by the image capture device 1b by using wireless communication technology such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark). When using wireless communication technology, the cradle 2b only serves to supply power to the image capture device 1b and support the image capture device 1b. Here, the image capture device 1b, the cradle 2b, and the smartphone 9 are located at the same base, base B. At base B, two users B1 and B2 are participating in the video call.

[0028] The video conference terminal 3, the PC 7, and the smartphone 9 are examples of communication terminals. OpenGL ES is installed in each communication terminal, and the communication terminal can create predetermined area information indicating a partial area of ​​a celestial sphere panoramic image, or create a predetermined area image from a celestial sphere panoramic image transmitted from another communication terminal.

[0029] The arrangement of the terminals, devices, and users shown in FIG. 8 is one example, and other examples may be used. For example, at site C, instead of the camera device 8, a camera device capable of capturing images related to a celestial sphere panoramic image may be used. The communication terminal also includes a digital television, a smart watch, a car navigation device, and the like. In the following description, any one of the camera devices 1a and 1b will be referred to as "camera device 1."

[0030] <<Hardware configuration of the embodiment>> 9 to 12, the hardware configurations of the image capturing device 1, the video conference terminal 3, the communication management system 5, the PC 7, and the smartphone 9 of this embodiment will be described in detail. Note that the image capturing device 8 is a general camera, and therefore a detailed description thereof will be omitted.

[0031] <Hardware configuration of the imaging device 1> First, the hardware configuration of the photographing device 1 will be described with reference to Fig. 9. Fig. 9 is a hardware configuration diagram of the photographing device 1. In the following, the photographing device 1 is assumed to be an omnidirectional (all-directional) photographing device using two imaging elements, but the number of imaging elements may be two or more. In addition, the photographing device 1 does not necessarily have to be a device dedicated to omnidirectional photographing, and an omnidirectional photographing unit may be attached to a normal digital camera, smartphone, or the like to have substantially the same functions as the photographing device 1.

[0032] As shown in FIG. 9, the photographing device 1 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I / F 116, a communication unit 117, and an antenna 117a.

[0033] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each wide-angle lens. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the fisheye lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the imaging elements are set.

[0034] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are connected to a serial I / F bus (such as an I2C bus) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to a CPU 111 via a bus 110. Furthermore, ROM 112, SRAM 113, DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like are also connected to the bus 110.

[0035] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs a predetermined processing on each image data, and then synthesizes the image data to create Mercator image data such as that shown in Figure 3(c).

[0036] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0037] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 1 have a preview display function or a function corresponding to video display on a display (for example, a display of the video conference terminal 3). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / minute).

[0038] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. Note that, in this embodiment, the photographing device 1 is not provided with a display unit, but may be provided with a display unit.

[0039] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I / F bus, and performs a predetermined process on the sound data.

[0040] The CPU 111 controls the overall operation of the photographing device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and processed Mercator image data.

[0041] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. A user operates the operation buttons to input various shooting modes, shooting conditions, and the like.

[0042] The network I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. The Mercator image data stored in the DRAM 114 is recorded in an external medium via the network I / F 116, or transmitted to an external device such as the video conference terminal 3 via the network I / F 116 as necessary.

[0043] The communication unit 117 communicates with an external device such as the video conference terminal 3 by short-range wireless technology such as WiFi or NFC (Near Field Communication) via an antenna 117a provided in the image capturing device 1. The communication unit 117 can also transmit Mercator image data to the external device of the video conference terminal 3.

[0044] The electronic compass 118 calculates the direction and tilt (Roll rotation angle) of the image capturing device 1 from the earth's magnetism, and outputs the direction and tilt information. This direction and tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data such as the date and time the image was captured, and the data size of the image data.

[0045] <Hardware configuration of video conferencing terminal> Next, the hardware configuration of the video conference terminal 3 will be described with reference to Fig. 10. Fig. 10 is a diagram showing the hardware configuration of the video conference terminal. As shown in Fig. 10, the video conference terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, an operation button 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an image sensor I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319.

[0046] Of these, the CPU 301 controls the overall operation of the video conference terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as an IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and sound data. The SSD (Solid State Drive) 305 controls reading and writing of various data from the flash memory 304 under the control of the CPU 301. Note that a HDD may be used instead of the SSD. The media I / F 307 controls reading and writing (storing) of data from a recording medium 306, such as a flash memory. The operation button 308 is a button that is operated when selecting a destination of the video conference terminal 3, etc. The power switch 309 is a switch for switching the power of the video conference terminal 3 ON / OFF.

[0047] The network I / F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means for capturing an image of a subject under the control of the CPU 301 to obtain image data. The image sensor I / F 313 is a circuit for controlling the driving of the camera 312. The microphone 314 is a type of built-in sound collection means for inputting sound. The sound input / output I / F 316 is a circuit for processing input / output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I / F 317 is a circuit for transmitting image data to an external display 320 under the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) or Bluetooth (registered trademark).

[0048] 10. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 301 shown in FIG.

[0049] The display 4 is a type of display means configured with liquid crystal or organic electroluminescence (EL) for displaying an image of a subject, operation icons, etc. The display 4 is also connected to the display I / F 317 by a cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0050] The camera 312 includes a lens and a solid-state imaging element that converts light into electric charges to digitize an image (video) of a subject, and a CMOS sensor, CCD sensor, or the like is used as the solid-state imaging element. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 via a USB (Universal Serial Bus) cable, or the like. When an external camera is connected, the external camera is driven in preference to the built-in camera 312 under the control of the CPU 301. Similarly, when an external microphone or an external speaker is connected, the external microphone and the external speaker are driven in preference to the built-in microphone 314 and the built-in speaker 315, respectively, under the control of the CPU 301.

[0051] Moreover, the recording medium 306 is configured to be detachable from the video conference terminal 3. Moreover, as long as it is a non-volatile memory that reads or writes data under the control of the CPU 301, it is not limited to the flash memory 304, and an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

[0052] <Communication management system, PC hardware configuration> Next, the hardware configuration of the communication management system 5 and the PC 7 will be described with reference to Fig. 11. Fig. 11 is a hardware configuration diagram of the communication management system and the PC. Note that since the communication management system 5 and the PC 7 are both computers and have the same configuration, the following describes the configuration of the communication management system 5, and the description of the configuration of the PC 7 will be omitted.

[0053] The communication management system 5 includes a CPU 501 for controlling the overall operation of the communication management system 5, a ROM 502 storing a program used to drive the CPU 501, such as an IPL, a RAM 503 used as a work area for the CPU 501, a HD 504 for storing various data such as programs for the communication management system 5, a HDD (Hard Disk Drive) 505 for controlling the reading or writing of various data from the HD 504 under the control of the CPU 501, a media drive 507 for controlling the reading or writing (storage) of data from a recording medium 506, such as a flash memory, a display 508 for displaying various information such as a cursor, a menu, a window, characters, or an image, a network I / F 509 for data communication using the communication network 100, a keyboard 511 having multiple keys for inputting characters, numbers, various instructions, etc., a mouse 512 for selecting and executing various instructions, selecting a processing target, moving the cursor, etc., a CD-RW (Compact The device is equipped with a CD-RW drive 514 that controls the reading and writing of various data from and to a CD-ReWritable Disc 513, and bus lines 510 such as an address bus and a data bus for electrically connecting the above components as shown in FIG. 11.

[0054] <Hardware configuration of smartphone> Next, the hardware of the smartphone will be described with reference to Fig. 12. Fig. 12 is a hardware configuration diagram of the smartphone. As shown in Fig. 12, the smartphone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration / direction sensor 906, a media I / F 908, and a GPS receiver 909.

[0055] Among these, the CPU 901 controls the overall operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901, such as an IPL. The RAM 903 is used as a work area for the CPU 901. The EEPROM 904 reads or writes various data, such as smartphone programs, under the control of the CPU 901. The CMOS (Complementary Metal Oxide Semiconductor) sensor 905 captures an object (mainly a self-portrait) and obtains image data under the control of the CPU 901. The acceleration / direction sensor 906 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I / F 908 controls the reading or writing (storage) of data from or to a recording medium 907, such as a flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0056] The smartphone 9 also has a long-distance communication circuit 911, a camera 912, an image sensor I / F 913, a microphone 914, a speaker 915, an audio input / output I / F 916, a display 917, an external device connection I / F 918, a short-distance communication circuit 919, an antenna 919a of the short-distance communication circuit 919, and a touch panel 921.

[0057] Among these, the long-distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 901. The image sensor I / F 913 is a circuit that controls the driving of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 916 is a circuit that processes input and output of sound signals between the microphone 914 and the speaker 915 under the control of the CPU 901. The display 917 is a type of display means such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 918 is an interface for connecting various external devices. The short-distance communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means that a user operates the smartphone 9 by pressing the display 917.

[0058] The smartphone 9 also includes a bus line 910. The bus line 910 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 901.

[0059] Incidentally, the recording media such as CD-ROMs on which the above-mentioned programs are stored, and the HD on which these programs are stored, can be provided domestically or internationally as program products.

[0060] <<Functional configuration of the embodiment>> Next, the functional configuration of this embodiment will be described with reference to Fig. 13 to Fig. 17. Fig. 13 is a functional block diagram of the photographing devices 1a and 1b, the video conference terminal 3, the communication management system 5, the PC 7, and the smartphone 9, which constitute a part of the image communication system of this embodiment.

[0061] <Functional configuration of the imaging device 1a> 13, the photographing device 1a has a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a memory / readout unit 19a. Each of these units is a function or means realized by operating any of the components shown in FIG. 9 in response to an instruction from the CPU 111 in accordance with a program for the photographing device loaded from the SRAM 113 onto the DRAM 114.

[0062] The photographing device 1 also has a storage unit 1000a that is configured by the ROM 112, the SRAM 113, and the DRAM 114 shown in Fig. 9. The storage unit 1000a stores a GUID (Globally Unique Identifier) ​​of the own device.

[0063] (Functional configuration of the imaging device 1a) Next, each functional configuration of the image capturing device 1a will be described in more detail with reference to FIG. 9 and FIG.

[0064] The reception unit 12a of the photographing device 1a is realized mainly by the processing of the operation unit 115 and the CPU 111 shown in FIG. 9, and receives operation input from the user.

[0065] The imaging section 13a is mainly realized by the imaging unit 101, image processing unit 104, imaging control unit 105, and processing by the CPU 111 shown in FIG. 9, and captures images of scenery and the like to obtain captured image data.

[0066] The sound collection section 14a is realized by the microphone 108 and sound processing unit 109 shown in FIG.

[0067] The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38 of the video conference terminal 3 by short-range wireless communication technology such as the NFC standard, BlueTooth, or WiFi.

[0068] The storage / reading unit 19a is realized mainly by the processing of the CPU 111 shown in FIG. 9, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.

[0069] <Functional configuration of the imaging device 1b> The photographing device 1b has a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, and a memory / readout unit 19b. These have the same functions as the reception unit 12a, the imaging unit 13a, the sound collection unit 14a, the communication unit 18a, and the memory / readout unit 19a in the photographing device 1a, respectively, and therefore the description thereof will be omitted. The photographing device 1 also has a memory unit 1000b constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG. 9. The memory unit 1000b stores the GUID of the own device.

[0070] <Functional configuration of videoconferencing terminal 3> 13, the video conference terminal 3 has a transmission / reception unit 31, a reception unit 32, an image / sound processing unit 33, a display control unit 34, a determination unit 35, a creation unit 36, a communication unit 38, and a storage / readout unit 39. Each of these units is a function or means realized by any of the components shown in FIG. 10 operating in response to an instruction from the CPU 301 in accordance with the program for the video conference terminal 3 expanded from the flash memory 304 onto the RAM 303.

[0071] The video conference terminal 3 also has a storage unit 3000 constructed by a ROM 302, a RAM 303, and a flash memory 304 shown in Fig. 10. An image type management DB 3001 and a photographing device management DB 3002 are constructed in the storage unit 3000. Of these, the image type management DB 3001 is constructed by an image type management table shown in Fig. 14. The photographing device management DB 3002 is constructed by a photographing device management table shown in Fig. 15.

[0072] (Image type management table) FIG. 14 is a conceptual diagram showing an image type management table. In this image type management table, image data ID, an IP address which is an example of the destination of the transmission source terminal, and a source name are stored and managed in association with each other. Among these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is added to image data transmitted from the same transmission source terminal. This allows the transmission destination terminal (receiving communication terminal) to identify the transmission source terminal of the received image data. The IP address of the transmission source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for identifying the photographing device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal such as the video conference terminal 3 according to a predetermined naming rule.

[0073] For example, it is shown that three communication terminals with IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1.3" are respectively transmitting image data indicated by image data IDs "RS001", "RS002", and "RS003". Furthermore, the types of images indicated by the source names of the communication terminals are "Video_Theta", "Video", and "Video_Theta", which indicate that the image types are "special image", "normal image", and "special image", respectively. Note that the special image here is a spherical panoramic image.

[0074] Data other than image data may also be managed in association with the image data ID. Data other than image data may be, for example, audio data or document data for screen sharing.

[0075] (Radiation device management table) 15 is a conceptual diagram showing an image capturing device management table. In this image capturing device management table, the vendor ID and product ID of the GUID of an image capturing device capable of obtaining two hemispherical images that are the basis of a celestial sphere panoramic image are stored and managed. As the GUID, for example, a vendor ID (VID) and product ID (PID) used in a USB device can be used. The vendor ID and product ID are stored in a communication terminal such as a video conference terminal when the terminal is shipped from the factory, but may be additionally stored after the shipment from the factory.

[0076] (Functional configuration of videoconferencing terminal 3) Next, each functional configuration of the video conference terminal 3 will be described in more detail with reference to FIG. 10 and FIG.

[0077] The transmission / reception unit 31 of the video conference terminal 3 is mainly realized by the processing of the network I / F 311 and CPU 301 shown in FIG. 10, and transmits and receives various data (or information) to and from the communication management system 5 via the communication network 100.

[0078] The reception unit 32 is realized mainly by the operation buttons 308 and processing by the CPU 301, and receives various selections or inputs from the user. In addition to the operation buttons 308, a touch panel or the like may be used as another input means.

[0079] 10, and performs image processing on image data obtained by capturing an image of a subject with camera 312. After the user's voice is converted into an audio signal by microphone 314, image / audio processing unit 33 performs audio processing on audio data related to this audio signal.

[0080] Furthermore, the image / sound processing unit 33 performs image processing on image data received from another communication terminal based on image type information such as a source name, so that the display control unit 34 displays an image on the display 4. Specifically, when the image type information indicates that the image is a special image, the image / sound processing unit 33 creates omnidirectional panoramic image data by converting the image data (for example, data of each hemispherical image as shown in Figs. 3(a) and (b)) into omnidirectional panoramic image data as shown in Fig. 4(b), and further creates a predetermined area image as shown in Fig. 6(b). In addition, the image / sound processing unit 33 outputs an audio signal related to audio data received from another communication terminal via the communication management system 5 to the speaker 315, and causes the speaker 315 to output audio.

[0081] The display control unit 34 is realized mainly by the processing of the display I / F 317 and the CPU 301, and performs control for displaying various images, characters, and the like on the display 4.

[0082] The determination unit 35 is realized mainly by the processing of the CPU 301, and determines, for example, the image type related to the image data received from the photographing device 1a.

[0083] The creation unit 36 ​​is mainly realized by the processing of the CPU 301, and creates a source name, which is an example of image type information, in accordance with the naming rules described above based on the result of the determination by the determination unit 35 of whether the image is a general image or a special image (here, a spherical panoramic image). For example, if the determination unit 35 determines that the image is a general image, the creation unit 36 ​​creates a source name "Video" indicating that it is a general image. On the other hand, if the determination unit 35 determines that the image is a special image, the creation unit 36 ​​creates a source name "Video_Theta" indicating that it is a special image.

[0084] The communication unit 38 is mainly realized by the processing of the short-range communication circuit 319, the antenna 318a, and the CPU 301, and can communicate with the communication unit 18a of the photographing device 1a by short-range wireless technology such as NFC, BlueTooth, WiFi, etc. Although the communication unit 38 and the transceiver unit 31 have been described as having separate communication units, they may be configured to share a common communication unit.

[0085] The storage / reading unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 10, and stores various data (or information) in the storage unit 3000 and reads out various data (or information) from the storage unit 3000.

[0086] <Functional configuration of the communication management system> Next, each functional component of the communication management system 5 will be described in detail with reference to Fig. 11 and Fig. 13. The communication management system 5 has a transmission / reception unit 51, a determination unit 55, a generation unit 56, and a storage / readout unit 59. Each of these units is a function or means realized by any of the components shown in Fig. 11 operating in response to an instruction from the CPU 501 in accordance with the program for the communication management system 5 loaded from the HD 504 onto the RAM 503.

[0087] The communication management system 5 also has a storage unit 5000 constructed by the RAM 503 and HD 504 shown in Fig. 11. A session management DB 5001 and an image type management DB 5002 are constructed in this storage unit 5000. Of these, the session management DB 5001 is constructed by the session management table shown in Fig. 16. The image type management DB 5002 is constructed by the image type management table shown in Fig. 17.

[0088] (Session management table) 16 is a conceptual diagram showing a session management table. In this session management table, the session ID and the IP addresses of the participating communication terminals are stored and managed in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session that realizes a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal such as the video conference terminal 3, and is used when selecting a communication session in each communication terminal. The IP addresses of the participating communication terminals indicate the IP addresses of the communication terminals that have participated in the virtual conference room indicated by the associated session ID.

[0089] (Image type management table) FIG. 17 is a conceptual diagram showing an image type management table. In the image type management table shown in FIG. 17, in addition to each piece of information managed in the image type management table shown in FIG. 14, a session ID that is the same as the session ID managed in the session management table is associated and managed. Here, it is shown that three communication terminals with IP addresses of "1.2.1.3", "1.2.2.3", and "1.3.1.3" are participating in the virtual conference room indicated by the same session ID "se101". In addition, the communication management system 5 manages the same image data ID, the IP address of the sender terminal, and the image type information managed by the communication terminal such as the video conference terminal 3 in order to transmit image type information, etc. to the communication terminal already in a video call and the newly participating communication terminal when a new communication terminal enters the virtual conference room. This eliminates the need to transmit and receive image type information, etc. between the communication terminal already in a video call and the newly participating communication terminal.

[0090] (Functional configuration of communication management system) Next, each functional configuration of the communication management system 5 will be described in detail with reference to FIG. 11 and FIG.

[0091] The transmission / reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 11, and transmits and receives various data (or information) to / from the video conference terminal 3 or the PC 7 via the communication network 100.

[0092] The determination unit 55 is mainly realized by the processing of the CPU 501 and makes various determinations.

[0093] The generation unit 56 is mainly realized by the processing of the CPU 501 and generates an image data ID.

[0094] The storage / reading unit 59 is mainly realized by the HDD 505 shown in FIG. 11 and the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 or reads out various data (or information) from the storage unit 5000.

[0095] <Functional Configuration of PC> Next, with reference to FIGS. 11 and 13, the functional configuration of the PC 7 will be described in detail. The PC 7 basically has the same functions as the video conference terminal 3. That is, as shown in FIG. 13, the PC 7 has a transmission / reception unit 71, a reception unit 72, an image / audio processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a communication unit 78, and a storage / reading unit 79. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating according to an instruction from the CPU 501 in accordance with the program for the PC 7 expanded from the HD 504 onto the RAM 503.

[0096] In addition, the PC 7 has a storage unit 7000 constructed by the ROM 502, the RAM 503, and the HD 504 shown in FIG. 11. In this storage unit 7000, an image type management DB 7001 and a photographing device management DB 7002 are constructed. Note that since the image type management DB 7001 and the photographing device management DB 7002 have the same configuration as the image type management DB 3001 and the photographing device management DB 3002, respectively, the description thereof will be omitted.

[0097] (PC functional configuration) The transmission / reception unit 71 of the PC 7 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG.

[0098] The reception unit 72 is mainly realized by the processing of the keyboard 511, the mouse 512 and the CPU 501, and realizes the same function as the reception unit 32. The image / sound processing unit 73 is mainly realized by an instruction from the CPU 501, and realizes the same function as the image / sound processing unit 33. The display control unit 74 is mainly realized by the processing of the CPU 501, and realizes the same function as the display control unit 34. The determination unit 75 is mainly realized by the processing of the CPU 501, and realizes the same function as the determination unit 35. The creation unit 76 is mainly realized by the processing of the CPU 501, and realizes the same function as the creation unit 36. The communication unit 78 is mainly realized by the processing of the CPU 501, and realizes the same function as the communication unit 38. The storage / readout unit 79 is realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 7000 and reads out various data (or information) from the storage unit 7000.

[0099] <Smartphone functional configuration> Next, the functional configuration of the smartphone 9 will be described in detail with reference to Fig. 12 and Fig. 13. The smartphone 9 basically has the same functions as the video conference terminal 3. That is, as shown in Fig. 13, the smartphone 9 has a transmission / reception unit 91, a reception unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a communication unit 98, and a storage / readout unit 99. Each of these units is a function or means realized by operating any of the components shown in Fig. 12 by an instruction from the CPU 901 according to the program for the smartphone 9 expanded from the EEPROM 904 onto the RAM 903.

[0100] 12. An image type management DB 9001 and an imaging device management DB 9002 are configured in the storage unit 9000. Note that the image type management DB 9001 and the imaging device management DB 9002 have the same configurations as the image type management DB 3001 and the imaging device management DB 3002, respectively, and therefore a description thereof will be omitted.

[0101] (Smartphone functional configuration) The transmitter / receiver 91 of the smartphone 9 is mainly realized by the processing of the long-distance communication circuit 911 and the CPU 901 shown in FIG.

[0102] The reception unit 92 is realized mainly by the processing of the touch panel 921 and the CPU 901 , and realizes the same functions as the reception unit 32 .

[0103] The image / sound processor 93 is implemented mainly by instructions from the CPU 901 , and realizes the same functions as the image / sound processor 33 .

[0104] The display control unit 94 is realized mainly by the processing of the CPU 901, and realizes the same functions as the display control unit .

[0105] The determination unit 95 is realized mainly by the processing of the CPU 901, and realizes the same functions as the determination unit .

[0106] The creation unit 96 is realized mainly by the processing of the CPU 901, and realizes the same functions as the creation unit .

[0107] The communication unit 98 is mainly realized by the processing of the CPU 901, and realizes the same functions as the communication unit .

[0108] The storage / reading unit 99 is realized by the processing of the CPU 901 , and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000 .

[0109] <<Processing or Operation of the Embodiment>> <Participation process> Next, the processing or operation of this embodiment will be described with reference to Fig. 18 to Fig. 22. First, the process of joining a specific communication session will be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a sequence diagram showing the process of joining a specific communication session. Fig. 19 is a diagram showing a selection screen for a communication session (virtual conference room).

[0110] First, when a user at site A (e.g., user A1) operates the videoconference terminal 3 to display a selection screen for a communication session (virtual conference room), the reception unit 32 receives the operation to display the selection screen, and the display control unit 34 displays the selection screen as shown in Fig. 19 on the display 4 (step S21). This selection screen displays selection buttons b1, b2, b3, etc., which indicate the virtual conference rooms R1, R2, R3, etc., to be selected. Each selection button b1, etc. is associated with a session ID.

[0111] When user A1 selects a desired selection button (selection button b1 in this case) of the virtual conference room, the reception unit 32 accepts the selection of the communication session (step S22). Then, the transmission / reception unit 31 transmits a participation request to the virtual conference room to the communication management system 5 (step S23). This participation request includes a session ID indicating the communication session whose selection was accepted in step S22, and the IP address of the video conference terminal 3 that is the request source terminal. As a result, the transmission / reception unit 51 of the communication management system 5 receives the participation request.

[0112] Next, the storage / reading unit 99 performs a process of participating in the communication session by adding the IP address received in step S23 to the participating terminal IP address field in the record of the session management DB 5001 (see FIG. 16) that has the same session ID as the session ID received in step S23 (step S24). Then, the transmitting / receiving unit 51 transmits a participation request response to the video conference terminal 3 (step S25). This participation request response includes the session ID received in step S23 and the result of the participation process. As a result, the transmitting / receiving unit 31 of the video conference terminal 3 receives the participation request response. Hereinafter, a case where the participation process is successful will be described.

[0113] <Image type information management process> Next, the management process of image type information will be described with reference to Fig. 20. Fig. 20 is a sequence diagram showing the management process of image type information.

[0114] First, when a user at site A (for example, user A1) connects the USB cable of the cradle 2a with the photographing device 1a attached thereto to the video conference terminal 3, the memory / read unit 19a of the photographing device 1a reads out the GUID of its own device (the photographing device 1a) stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of its own device to the communication unit 38 of the video conference terminal 3 (step S51). As a result, the communication unit 38 of the video conference terminal 3 receives the GUID of the photographing device 1a.

[0115] Next, the determination unit 35 of the video conference terminal 3 determines whether the same vendor ID and product ID as the vendor ID and product ID in the GUID received in step S51 are managed in the photographing device management DB 3002 (see FIG. 15), thereby determining the image type (step S52). Specifically, if the same vendor ID and product ID are managed in the photographing device management DB 3002, the determination unit 35 determines that the photographing device 1a is a photographing device that captures special images (here, a celestial sphere panoramic image). On the other hand, if the same vendor ID and product ID are not managed in the photographing device management DB 3002, the determination unit 35 determines that the photographing device 1a is a photographing device that captures general images.

[0116] Next, the storage / reading unit 39 stores in the image type management DB 3001 (see FIG. 14) the IP address of the terminal itself (video conference terminal 3) which is the source terminal, in association with the image type information which is the result of the judgment made in step S52 (step S53). In this state, the image data ID is not associated. The image type information is, for example, a source name determined according to a predetermined naming rule, and an image type (general image, special image).

[0117] Next, the transmitting / receiving unit 31 transmits a request to add image type information to the communication management system 5 (step S54). This request to add image type information includes the IP address of the own terminal, which is the transmission source terminal stored in step S53, and the image type information. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request to add image type information.

[0118] Next, the storage / read unit 59 of the communication management system 5 searches the session management DB 5001 (see FIG. 16) using the IP address of the source terminal received in step S54 as a search key, and reads out the corresponding session ID (step S55).

[0119] Next, the generating unit 56 generates a unique image data ID (step S56). Then, the storing / reading unit 59 stores the session ID read out in step S55, the image data ID generated in step S56, and the IP address of the transmission source terminal and the image type information received in step S54 in association with each other as a new record in the image type management DB 5002 (see FIG. 17) (step S57). Then, the transmitting / receiving unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3. As a result, the transmitting / receiving unit 31 of the video conference terminal 3 receives the image data ID (step S58).

[0120] Next, the storage / reading unit 39 of the videoconference terminal 3 stores the image data ID received in step S58 in the image type management DB 3001 (see FIG. 14) in association with the IP address and image type information of the terminal itself (videoconference terminal 3), which is the sending terminal, stored in step S53 above (step S59).

[0121] Meanwhile, the transmitting / receiving unit 51 of the communication management system 5 transmits an addition notification of the image type information to the smartphone 9, which is another communication terminal (step S60). This addition notification of the image type information includes the image data ID generated in step S56, and the IP address and image type information of the terminal itself (the video conference terminal 3), which is the transmission source terminal, stored in step S53. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the addition notification of the image type information. Note that the transmission destination of the transmitting / receiving unit 51 is another IP address associated with the same session ID as the IP address of the video conference terminal 3 in the session management DB 5001 (see FIG. 16). In other words, the transmission destination is another communication terminal in the same virtual conference room as the video conference terminal 3.

[0122] Next, the storage / readout unit 99 of the smartphone 9 associates the image data ID, the IP address of the sending terminal, and the image type information received in step S60 and stores them as a new record in the image type management DB 9001 (see FIG. 14) (step S61). Similarly, an additional communication of the image type information is sent to another communication terminal, PC 7, which also stores the information in the image type management DB 7001 (see FIG. 14). As a result, the communication terminals can share the same information in the image type management DBs 3001, 7001, 9001.

[0123] <Image data communication processing> The communication process of image data in a video call will be described with reference to Fig. 21 to Fig. 23. Fig. 21 is a sequence diagram showing the communication process of image data in a video call.

[0124] First, image data obtained by photographing an object, scenery, or the like is transmitted from the communication unit 18a of the photographing device 1a to the communication unit 38 of the video conference terminal 3 (step S101). In this case, since the photographing device 1a is a device capable of obtaining two hemispherical images that are the basis of a celestial sphere panoramic image, the image data is composed of data of two hemispherical images as shown in Figs. 3(a) and (b). As a result, the communication unit 38 of the video conference terminal 3 receives the image data.

[0125] Next, the transmitting / receiving unit 31 of the video conference terminal 3 transmits the image data sent from the image capturing device 1a to the communication management system 5 (step S102). This transmission includes an image data ID for identifying the image data to be transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the image data and the image data ID.

[0126] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits the image data sent from the video conference terminal 3 to the smartphone 9 (step S103). This transmission includes an image data ID for identifying the image data to be transmitted. As a result, the transmitting / receiving unit 51 of the smartphone 9 receives the image data and the image data ID.

[0127] Next, in the smartphone 9, the storage / readout unit 99 searches the image type management DB 9001 (see FIG. 14) using the image data ID received in step S103 as a search key to read out corresponding image type information (source name) (step S104). Then, if the image type information indicates that the image is a special image (here, a celestial sphere panoramic image) (Video_Theta), the image / sound processing unit 93 creates a celestial sphere panoramic image from the image data received in step S103, and further creates a predetermined area image (step S105). In this case, the image / sound processing unit 93 combines an icon 191 indicating that the image is a celestial sphere panoramic image (described later) with the predetermined area image based on the image type information indicating that the image is a special image (Video_Theta).

[0128] Next, the display control unit 94 displays the predetermined area image including the icon 191 on the display 917 of the smartphone 9. When the image type information indicates that it is a general image (Video), the image / sound processing unit 93 does not create a celestial sphere panoramic image from the image data received in step S103, and the display control unit 94 displays a general image not including the icon 191.

[0129] Next, the state of a video call will be described with reference to Fig. 22. Fig. 22 is an image diagram showing the state of a video call. Of these, Fig. 22(a) shows a case where the photographing device 1a is not used, and Fig. 22(b) shows a case where the photographing device 1a is used.

[0130] First, as shown in Fig. 22(a), when using the camera 312 (see Fig. 10) provided in advance in the video conference terminal 3 without using the image capturing device 1a, the angle of view is 125 degrees horizontally and 70 degrees vertically, so the video conference terminal 3 must be placed at the edge of the desk where each user A1, etc. can be seen. Therefore, each user A1, etc. must face the video conference terminal 3 when speaking. Also, since each user A1, etc. faces the video conference terminal 3, the display 4 must also be placed on the side of the video conference terminal 3. As a result, users A2 and A4 who are far from the video conference terminal 3 must speak relatively loudly because they are far from the microphone 314 (see Fig. 10), and the contents displayed on the display 4 are difficult to see.

[0131] 22(b), when the image capturing device 1a is used, two hemispherical images that are the basis of a celestial sphere panoramic image can be obtained, so the video conference terminal 3 and the display 4 can be placed relatively close to the center of the desk. This allows each user A1, etc., to speak relatively quietly since they are close to the microphone 314, and the contents displayed on the display 4 can be easily seen.

[0132] Next, a display example of the display 917 at the site B will be described with reference to FIG. 23. FIG. 23 is a display example of the display at the site B. Among these, FIG. 23(a) shows a case where a celestial sphere panorama image and a predetermined area image are not created from the image data sent from the video conference terminal 3 (camera 1a) and the camera 1b, and the image data sent from the PC 7 (camera 8) is displayed as is. FIG. 23(b) shows a case where a celestial sphere panorama image and a predetermined area image are created from the image data sent from the video conference terminal 3 (camera 1a) and the camera 1b, and the image data sent from the PC 7 (camera 8) is displayed as is. Note that an image of the site A is displayed in the upper left display area of ​​the display 917, an image of the site B (own site) is displayed in the upper right display area, and an image of the site C is displayed in the lower left display area. In this example, since simultaneous video calls are being made at three sites, nothing is displayed in the lower right display area.

[0133] When image data sent from the image capturing devices 1a and 1b capable of capturing spherical panoramic images is displayed as is, it is displayed as shown in Fig. 23(a), where location A on the upper left side and location B (own location) on the upper right side are displayed as the front hemispherical image and the rear hemispherical image, respectively, as shown in Fig. 3(a) and (b).

[0134] In response to this, when the image / sound processor 93 creates a celestial sphere panoramic image from image data sent from the photographing devices 1a and 1b capable of obtaining two hemispherical images that are the basis of the celestial sphere panoramic image, and further creates a predetermined area image, the predetermined area image, which is a planar image, is displayed as shown in Fig. 23(b). Note that the site C is equipped with the photographing device 8 that obtains a general image, so that a general image is displayed in both Figs. 23(a) and (b).

[0135] Furthermore, an icon 191 indicating that the image is a spherical panoramic image is displayed at the top left of each of the images from site A and site B (own site). The display position of the icon 191 need not be the top left, but may be anywhere, such as the top right, bottom left, or bottom right. The type of the icon 191 is not limited to that shown in FIG. 23(b). Instead of the icon 191, text such as "spherical image" may be displayed, or a combination of an icon and text may be displayed.

[0136] Furthermore, the user can change the predetermined area related to the predetermined area image in the same omnidirectional panoramic image. That is, the users B1 and B2 place their fingers on the touch panel 921 of the smartphone 9 and move them, so that the reception unit 92 receives the movement of the fingers, and the display control unit 94 can shift, rotate, reduce, or enlarge the predetermined area image. As a result, even if only users A1 and A2, who are part of the site A, are displayed in the initial setting (default), as shown in FIG. 23(b), the predetermined area image can be shifted to display users A4 and A3.

[0137] <<Main Effects of This Embodiment>> As described above, according to this embodiment, a communication terminal such as the video conference terminal 3 can create a celestial sphere panoramic image and further create a predetermined area image based on the image data ID transmitted together with the image data and corresponding image type information. This has the effect of preventing the front hemispherical image and the rear hemispherical image from being displayed as shown in Fig. 23(a).

[0138] [Other examples] Another example of the management process of image type information will be described with reference to Fig. 24. Fig. 24 is a sequence diagram showing another example of the management process of image type information.

[0139] First, when a user at site A (for example, user A1) connects the USB cable of the cradle 2a with the photographing device 1a attached thereto to the video conference terminal 3, the memory / read unit 19a of the photographing device 1a reads out the GUID of its own device (the photographing device 1a) stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of its own device to the communication unit 38 of the video conference terminal 3 (step S151). As a result, the communication unit 38 of the video conference terminal 3 receives the GUID of the photographing device 1a.

[0140] Next, the transmitting / receiving unit 31 of the video conference terminal 3 transmits a request for the image data ID to the communication management system 5 (step S152). This request includes the IP address of the video conference terminal 3, which is the transmission source terminal (own terminal). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request for the image data ID.

[0141] Next, the generating unit 56 generates a unique image data ID (step S153). Then, the communication management system 5 transmits a notification of the image data ID to the smartphone 9 (step S154). This notification includes the image data ID generated in step S153 and the IP address of the video conference terminal 3 as the transmission source terminal received in step S152. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the notification of the image data ID. Then, in the smartphone 9, the storing / reading unit 99 associates the image data ID and the IP address of the transmission source terminal received in step S154 with each other and stores them as a new record in the image type management DB 9001 (see FIG. 14) (step S155). At this point, the image type information is not associated and stored.

[0142] Meanwhile, in the communication management system 5, in order to respond to the request for the image data ID in step S152, the transmission / reception unit 51 transmits the image data ID to the video conference terminal 3 (step S156). As a result, the transmission / reception unit 31 of the video conference terminal 3 receives the image data ID. Then, in the video conference terminal 3, the storage / reading unit 39 associates the image data ID received in step S156 with the IP address of the sender terminal (own terminal) managed by the video conference terminal 3 as a new record in the image type management DB 3001 (see FIG. 14) (step S157). Note that at this point, the image type information has not been associated and stored.

[0143] Next, the determination unit 35 of the video conference terminal 3 determines whether the same vendor ID and product ID as the GUID received in step S151 are managed in the imaging device management DB 3002 (see FIG. 15), thereby determining the image type (step S158). This determination is the same as the process in step S52 above.

[0144] Next, the storage / reading unit 39 stores in the image type management DB 3001 (see FIG. 14) the IP address of the terminal (video conference terminal 3) which is the source terminal, and the image type information which is the result of the determination made in step S158, in association with the image data ID already stored in step S157 (step S159). Then, the transmitting / receiving unit 31 transmits a notification of the image type information to the communication management system 5 (step S160). This notification of the image type information includes the image data ID stored in step S157, and the IP address and image type information of the terminal (video conference terminal 3) which is the source terminal, which are stored in step S159. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the notification of the image type information.

[0145] Next, the storage / read unit 59 of the communication management system 5 searches the session management DB 5001 (see FIG. 16) using the IP address of the source terminal received in step S160 as a search key, and reads out the corresponding session ID (step S161).

[0146] Next, the storage / reading unit 59 stores the session ID read out in step S161, and the IP address of the source terminal, the image data ID, and the image type information received in step S160 in association with each other as a new record in the image type management DB 5002 (see FIG. 17) (step S162). Then, the communication management system 5 transmits a notification of the image type information to the smartphone 9, which is another communication terminal (step S163). This notification of the image type information includes the image data ID and image type information stored in step S162. As a result, the transmission / reception unit 91 of the smartphone 9 receives the notification of the image type information.

[0147] Next, the storage / readout unit 99 of the smartphone 9 associates the image type information received in step S163 with the same image data ID as the image data ID received in step S163 and additionally stores the image type information in the image type management DB 9001 (see FIG. 14) (step S164), thereby completing the management process of the image type information. [Explanation of symbols]

[0148] 1a Imaging device 1b Imaging device 3. Video conference terminal (an example of a communication terminal, an example of a first communication terminal) 5. Communications Management System 7 PC (an example of a communication terminal) 8. Imaging Equipment 9 Smartphone (an example of a communication terminal, an example of a second communication terminal) 31 Transmitter / receiver 51 Transmitting / receiving unit (an example of a transmitting means, an example of a receiving means) 56 Generation unit (an example of a generation means) [Prior art documents] [Patent documents]

[0149] [Patent Document 1] JP 2012-178135 A [Patent Document 2] JP 2011-223076 A

Claims

1. An image communication system for transmitting and receiving image data acquired by an image capturing device at each of the locations, comprising: a display control means for displaying, on a display, a planar image of a part of a special image generated from the image data and a general image photographed by the photographing device, based on the type of the image data; An image communication system having the above configuration.

2. The display control means, when the type of the image data indicates a special image, displays a planar image of a part of the special image generated from the image data on a display, and when the type of the image data indicates a general image, displays a general image photographed by the photographing device on a display.

2. The image communication system according to claim 1.

3. The photographing devices at each of the bases include at least a photographing device for photographing special images and a photographing device for photographing general images, 3. An image communication system according to claim 1 or 2.

4. The general image is a planar image.

4. The image communication system according to claim 3.

5. the special image is a spherical panoramic image, The planar image is a predetermined area image of the celestial sphere panoramic image.

4. The image communication system according to claim 3.

6. The display control means displaying an icon indicating a celestial sphere panoramic image on the predetermined region image based on the type of the image data; 6. An image communication system according to claim 5.

7. A communication terminal that displays image data acquired by an image capturing device at each of the locations on a display, a display control means for displaying, on the display, a planar image of a part of a special image generated from the image data and a general image photographed by the photographing device, based on the type of the image data; A communication terminal having the above configuration.

8. A communication method for displaying image data acquired by an image capturing device at each of the locations on a display, the method comprising: displaying, on the display, a planar image of a part of a special image generated from the image data and a general image photographed by the photographing device based on the type of the image data; A communication method comprising:

9. A method for providing a function of displaying image data acquired by an image capturing device at each of the locations on a display, the method comprising: A method for providing a communication terminal performing a video call with a function for displaying on the display a planar image of a portion of a special image generated from the image data and a general image captured by the photographing device based on the type of image data.

10. A program executed on a communication terminal that displays image data acquired by an image capturing device at each of the locations and received from the respective locations, the program comprising: a program code for displaying, on the display, a planar image of a part of a special image generated from the image data and a general image photographed by the photographing device, based on a type of the image data; A program having:

Images