Program, display providing method, communication terminal and image communication system

Abstract

To solve the problem in which: when a communication terminal which received image data obtained from a special imaging device displays the image data in accordance with a plane image, two hemispherical images that will be a source of a panoramic omnidirectional sphere image are displayed, resulting in an image which is very hard to see; and when a communication device which received image data obtained from a general imaging device performs the processing of creating a panoramic omnidirectional sphere image, the image is corrupted and becomes very hard to see.SOLUTION: A system receives image type information indicating the image type from the first communication device (S54), generates the image data identification information for identifying the image data to be transmitted from the first communication terminal to the second communication terminal on the basis of the receiving of the image type information (S56), transmits the image data identification information generated as above to the first communication terminal (S58) and transmits the generated image data identification information and the image type information received as above to the second communication terminal (S60).SELECTED DRAWING: Figure 20

Description

[Technical Field]

[0001] The present invention relates to a communication management system, a communication system, a communication management method, and a program. [Background technology]

[0002] Conference systems that hold remote conferences between remote locations via communication networks such as the Internet are becoming widespread. In these conference systems, in a conference room where one of the parties, such as attendees, is present, a communication terminal of the remote conference system is used to capture and collect images of the conference room and audio, such as speech, of the parties, and converts these into digital data and transmits them to the communication terminal of the other party. This allows for a video call to be conducted by displaying images on a display and outputting audio from a speaker in the other party's conference room, thereby enabling a conference between remote locations to be held in an environment similar to a real conference (see Patent Document 1).

[0003] Meanwhile, there is known a technology in which a communication terminal is connected to an imaging device capable of acquiring a celestial sphere panoramic image in real time, the celestial sphere panoramic image acquired from the imaging device is distributed to each communication terminal of the other party, and each communication terminal of the other party sequentially performs image conversion processing from the celestial sphere panoramic image to a rectangular planar image and displays it on a display or the like (see Patent Document 2). Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the types of camera devices used at the respective locations are different, such as a general camera device that obtains rectangular planar images and a special camera device that obtains two hemispherical images that are the basis for a celestial sphere panoramic image, the following problems arise on the communication terminal side that receives the image data.

[0005] That is, when a communication terminal that receives image data acquired from a special imaging device displays the image data in combination with a planar image, two hemispherical images that are the basis of the celestial sphere panoramic image are displayed, and since user A1 etc. are displayed facing downward in the spherical image as shown in Fig. 23(a), the image becomes very difficult to see. On the other hand, when a communication terminal that receives image data acquired from a general imaging device performs processing to create a celestial sphere panoramic image, the image becomes corrupted and very difficult to see. [Means for solving the problem]

[0006] The invention of claim 1 is a communication management system characterized by having a receiving means for receiving image type information indicating the type of image from a first communication terminal, a generating means for generating image data identification information for identifying image data to be transmitted from the first communication terminal to a second communication terminal based on the received image type information, and a transmitting means for transmitting the generated image data identification information to the first communication terminal and transmitting the generated image data identification information and the received image type information to the second communication terminal. [Effects of the Invention]

[0007] According to the present invention, the second communication terminal that receives the image data can perform image processing in accordance with the image type information, thereby providing the effect of being able to display an image that is easy to see. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1(a) is a left side view of the imaging device, FIG. 1(b) is a front view of the imaging device, and FIG. 1(c) is a plan view of the imaging device. [Figure 2] FIG. 2 is an image diagram of the imaging device in use. [Figure 3] Figure 3(a) shows a hemispherical image (before) taken with the imaging device, Figure 3(b) shows a hemispherical image (after) taken with the imaging device, and Figure 3(c) shows an image represented by the Mercator projection. [Figure 4]FIG. 4(a) is a conceptual diagram showing how a sphere is covered with a Mercator image, and FIG. 4(b) is a diagram showing a spherical panoramic image. [Figure 5] FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the celestial sphere panoramic image is a three-dimensional sphere. [Figure 6] FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 7] FIG. 7 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area. [Figure 8] FIG. 8 is a schematic diagram of an image communication system according to an embodiment of the present invention. [Figure 9] FIG. 9 is a hardware configuration diagram of the imaging device. [Figure 10] FIG. 10 is a diagram showing the hardware configuration of a video conference terminal. [Figure 11] FIG. 11 is a hardware configuration diagram of the communication management system and the PC. [Figure 12] FIG. 12 is a hardware configuration diagram of a smartphone. [Figure 13] FIG. 13 is a functional block diagram of the image communication system. [Figure 14] FIG. 14 is a conceptual diagram showing an image type management table. [Figure 15] FIG. 15 is a conceptual diagram showing the imaging device management table. [Figure 16] FIG. 16 is a conceptual diagram showing a session management table. [Figure 17] FIG. 17 is a conceptual diagram showing an image type management table. [Figure 18] FIG. 18 is a sequence diagram showing the process of joining a specific communication session. [Figure 19] FIG. 19 shows a selection screen for a communication session (virtual conference room). [Figure 20] FIG. 20 is a sequence diagram showing the management process of image type information. [Figure 21]FIG. 21 is a sequence diagram showing a communication process of image data in a video call. [Figure 22] FIG. 22(a) is a diagram showing a case where a photographing device is not used, and FIG. 22(b) is a diagram showing a case where a photographing device is used. [Figure 23] 23A and 23B are diagrams showing display examples of the display at site B, in which FIG. 23A shows a case where the image data transmitted from the image capturing devices 1a and 1b is displayed as is without creating a celestial sphere panoramic image and a predetermined area image, and FIG. 23B shows a case where the celestial sphere panoramic image and a predetermined area image are created from the image data transmitted from the image capturing devices 1a and 1b. [Figure 24] FIG. 24 is a sequence diagram showing another example of the image type information management process. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] <<Outline of the embodiment>> <How to generate a spherical panoramic image> A method for generating a spherical panoramic image will be described with reference to FIGS.

[0011] First, the appearance of the imaging device 1 will be described using Fig. 1. The imaging device 1 is a digital camera for obtaining captured images that serve as the basis for a full-sphere (360°) panoramic image. Fig. 1(a) is a left side view of the imaging device, Fig. 1(b) is a front view of the imaging device, and Fig. 1(c) is a plan view of the imaging device.

[0012] As shown in FIG. 1(a), the photographing device 1 is small enough to be held in one hand. As shown in FIGS. 1(a), 1(b), and 1(c), an image sensor 103a is provided on the front side (front side) of the upper portion of the photographing device 1, and an image sensor 103b is provided on the rear side (rear side). These image sensors (image sensors) 103a and 103b are used in conjunction with optical components (e.g., fisheye lenses 102a and 102b, described below) capable of capturing hemispherical images (with a field angle of 180° or more). As shown in FIG. 1(b), an operation unit 115, such as a shutter button, is provided on the surface of the photographing device 1 opposite the front side.

[0013] Next, the usage of the image capturing device 1 will be described with reference to Fig. 2. Fig. 2 is an image diagram of the image capturing device in use. As shown in Fig. 2, the image capturing device 1 is used, for example, by being held by a user and capturing images of subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image capturing element 103a and the image capturing element 103b shown in Fig. 1, respectively.

[0014] Next, an outline of the process of creating a celestial sphere panoramic image from images captured by the image capturing device 1 will be described with reference to Fig. 3 and Fig. 4. Fig. 3(a) is a diagram showing a hemispherical image (front side) captured by the image capturing device, Fig. 3(b) is a diagram showing a hemispherical image (rear side) captured by the image capturing device, and Fig. 3(c) is a diagram showing an image expressed by Mercator projection (hereinafter referred to as "Mercator image"). Fig. 4(a) is a conceptual diagram showing a state in which a sphere is covered with a Mercator image, and Fig. 4(b) is a diagram showing a celestial sphere panoramic image.

[0015] As shown in Fig. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a fisheye lens 102a (described later). Also, as shown in Fig. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a fisheye lens 102b (described later). Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are combined by the image capturing device 1 to create a Mercator image as shown in Fig. 3(c).

[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted to cover the spherical surface as shown in FIG. 4(a), and a spherical panoramic image as shown in FIG. 4(b) is created. In this way, the spherical panoramic image is expressed as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the spherical panoramic image may be a still image or a video.

[0017] As described above, a spherical panoramic image is an image pasted to cover the spherical surface, which gives a sense of incongruity to people. Therefore, by displaying a predetermined region of the spherical panoramic image (hereinafter referred to as a "predetermined region image") as a flat image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to people. This will be described with reference to FIGS. 5 and 6.

[0018] FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the celestial sphere panoramic image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user viewing the celestial sphere panoramic image displayed as a three-dimensional sphere. FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing the predetermined area image when displayed on a display. In FIG. 6(a), the celestial sphere panoramic image shown in FIG. 4 is represented by a three-dimensional sphere CS. If the celestial sphere panoramic image generated in this way is a 3D sphere CS, as shown in FIG. 5, the virtual camera IC is located outside the celestial sphere panoramic image. The predetermined area T in the celestial sphere panoramic image is the shooting area of ​​the virtual camera IC, and is specified by predetermined area information indicating position coordinates (x(rH), y(rV), angle of view α(angle)) including the angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere panoramic image. Zooming of the predetermined area T can be expressed by widening or narrowing the range (arc) of the angle of view α. Zooming of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the omnidirectional panoramic image.

[0019] Then, the predetermined area image, which is an image of the predetermined area T shown in FIG. 6(a), is displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined area image represented by the initially set (default) predetermined area information. Note that the predetermined area information may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the position coordinates of the virtual camera IC. In the following, the position coordinates (x(rH), y(rV) and angle of view α(angle)) of the virtual camera IC will be used for explanation.

[0020] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described using FIG. 7. FIG. 7 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 7, when the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC is 2L, the center point CP becomes the (x, y) parameter of the predetermined area information. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). In FIG. 7, the trigonometric function shown in the following formula (1) generally holds.

[0021] Lf=tan(α / 2) (Equation 1)

[0022] <Outline of the image communication system> Next, an outline of the configuration of the image communication system of this embodiment will be described with reference to Fig. 8. Fig. 8 is a schematic diagram of the configuration of the image communication system of this embodiment.

[0023] 8, the image communication system of this embodiment is configured by photographing devices 1a and 1b, a video conference terminal 3, a communication management system 5, a PC (Personal Computer) 7, a photographing device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

[0024] Of these, the image capturing devices 1a and 1b are special digital cameras for capturing images of subjects, scenery, etc. to obtain two hemispherical images that are the basis for a celestial sphere panoramic image, as described above, while the image capturing device 8 is a general digital camera for capturing images of subjects, scenery, etc. to obtain a general planar image.

[0025] The video conference terminal 3 is a terminal dedicated to video conferences and displays images of a video call on a display 4 via a wired cable such as a USB (Universal Serial Bus) cable. The video conference terminal 3 normally captures images using a camera 312 (described later). However, when the video conference terminal 3 is connected via a wired cable to a cradle 2a that holds the image capture device 1a, the image capture device 1a is given priority, and two hemispherical images that form the basis of a celestial sphere panoramic image can be obtained. When a wired cable is used, the cradle 2a not only performs communication between the image capture device 1a and the video conference terminal 3, but also serves to supply power to and support the image capture device 1a. Note that the image capture device 1a, the cradle 2a, the video conference terminal 3, and the display 4 are all located at the same site, site A. Four users A1, A2, A3, and A4 are participating in the video call at site A.

[0026] The communication management system 5 manages communications between the video conference terminals 3, PCs 7, and smartphones 9, and manages the types of image data (general images and special images) that are sent and received. In this example, the special images are spherical panoramic images. The communication management system 5 is installed at a service company that provides video communication services. The communication management system 5 may be constructed by a single computer, or may be constructed by multiple computers in which each unit (function, means, or storage unit) is divided and arbitrarily assigned.

[0027] The PC 7 is capable of video communication by attaching the camera device 8. In this example, the PC 7 and the camera device 8 are located at the same location, site C. Also, at site C, one user C1 is participating in the video communication.

[0028] The smartphone 9 displays an image of the video call on a display 917 (described later) provided on the smartphone 9. The smartphone 9 normally captures images using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 (described later) provided on the smartphone 9, but can also acquire two hemispherical image data that form the basis of the omnidirectional panoramic image acquired by the image capture device 1b using wireless communication technology such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark). When using wireless communication technology, the cradle 2b only serves to supply power to the image capture device 1b and support the image capture device 1b. Here, the image capture device 1b, the cradle 2b, and the smartphone 9 are all located at the same site, site B. At site B, two users B1 and B2 are participating in the video call.

[0029] The video conference terminal 3, the PC 7, and the smartphone 9 are examples of communication terminals. OpenGL ES is installed in each communication terminal, and the communication terminals can create predetermined area information indicating a partial area of ​​a celestial sphere panoramic image and create a predetermined area image from a celestial sphere panoramic image transmitted from another communication terminal.

[0030] The arrangement of the terminals, devices, and users shown in FIG. 8 is one example, and other examples may be used. For example, at site C, instead of camera device 8, a camera device capable of capturing a celestial sphere panoramic image may be used. The communication terminal also includes a digital television, a smartwatch, a car navigation device, and the like. Hereinafter, any one of camera devices 1a and 1b will be referred to as "camera device 1."

[0031] <<Hardware configuration of the embodiment>> 9 to 12, the hardware configurations of the image capturing device 1, the video conference terminal 3, the communication management system 5, the PC 7, and the smartphone 9 of this embodiment will be described in detail. Note that the image capturing device 8 is a general camera, and therefore a detailed description thereof will be omitted.

[0032] <Hardware configuration of the imaging device 1> First, the hardware configuration of the image capturing device 1 will be described using Fig. 9. Fig. 9 is a hardware configuration diagram of the image capturing device 1. In the following, the image capturing device 1 is assumed to be an omnidirectional (omnidirectional) image capturing device using two image capturing elements, but the number of image capturing elements may be any number greater than or equal to two. Furthermore, the image capturing device does not necessarily have to be a device dedicated to omnidirectional image capturing; an ordinary digital camera, smartphone, or the like may be equipped with an omnidirectional image capturing unit as an add-on, so that it has substantially the same functions as the image capturing device 1.

[0033] As shown in FIG. 9, the photographing device 1 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and an antenna 117a.

[0034] Of these, imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b, each with a field angle of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each wide-angle lens. Imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts optical images captured by fisheye lenses 102a and 102b into image data in the form of electrical signals and outputs the converted image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

[0035] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are connected via a serial I / F bus (such as an I2C bus) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to a CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like.

[0036] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then synthesizes the image data to create Mercator image data such as that shown in Figure 3(c).

[0037] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0038] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 1 have a preview display function or a function for displaying moving images on a display (for example, the display of the video conference terminal 3). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0039] As will be described later, the imaging control unit 105 also functions as a synchronization control means that synchronizes the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the photographing device 1 is not provided with a display unit, but may be provided with a display unit.

[0040] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I / F bus and performs predetermined processing on the sound data.

[0041] The CPU 111 controls the overall operation of the imaging device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and processed Mercator image data.

[0042] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. The user operates the operation buttons to input various shooting modes, shooting conditions, etc.

[0043] The network I / F 116 is a general term for an interface circuit (such as a USB I / F) with external media such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. The Mercator image data stored in the DRAM 114 is recorded on external media via the network I / F 116, or transmitted to an external device such as the video conference terminal 3 via the network I / F 116 as needed.

[0044] The communication unit 117 communicates with an external device such as the video conference terminal 3 by short-range wireless technology such as WiFi or NFC (Near Field Communication) via an antenna 117a provided in the image capturing device 1. The communication unit 117 can also transmit Mercator image data to the external device of the video conference terminal 3.

[0045] The electronic compass 118 calculates the orientation and tilt (roll rotation angle) of the image capturing device 1 from the Earth's magnetism and outputs the orientation and tilt information. This orientation and tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of captured images. The related information also includes data such as the date and time the image was captured and the data size of the image data.

[0046] <Videoconferencing terminal hardware configuration> Next, the hardware configuration of the video conference terminal 3 will be described with reference to Fig. 10. Fig. 10 is a diagram showing the hardware configuration of the video conference terminal. As shown in Fig. 10, the video conference terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, an operation button 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an image sensor I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319.

[0047] Of these, the CPU 301 controls the overall operation of the video conference terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as an IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and audio data. The SSD (Solid State Drive) 305 controls the reading and writing of various data from and to the flash memory 304 under the control of the CPU 301. Note that an HDD may be used instead of an SSD. The media I / F 307 controls the reading and writing (storage) of data from and to a recording medium 306, such as a flash memory. The operation button 308 is a button that is operated when selecting a destination for the video conference terminal 3, etc. The power switch 309 is a switch for turning the power of the video conference terminal 3 ON / OFF.

[0048] The network I / F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 301. The image sensor I / F 313 is a circuit that controls the driving of the camera 312. The microphone 314 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 316 is a circuit that processes input and output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I / F 317 is a circuit that transmits image data to the external display 4 under the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) or Bluetooth (registered trademark).

[0049] 10. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 301 shown in FIG.

[0050] The display 4 is a type of display means configured with a liquid crystal or organic electroluminescence (EL) display that displays an image of a subject, operation icons, etc. The display 4 is connected to the display I / F 317 via a cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signals. HDMI is a registered trademark.

[0051] The camera 312 includes a lens and a solid-state imaging element that converts light into an electric charge to digitize an image (video) of a subject, and a CMOS sensor, CCD sensor, or the like is used as the solid-state imaging element. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 via a USB (Universal Serial Bus) cable, or the like. When an external camera is connected, the external camera is driven under the control of the CPU 301 in preference to the built-in camera 312. Similarly, when an external microphone or an external speaker is connected, the external microphone or external speaker is driven under the control of the CPU 301 in preference to the built-in microphone 314 or built-in speaker 315, respectively.

[0052] The recording medium 306 is detachable from the video conference terminal 3. As long as it is a non-volatile memory that reads or writes data under the control of the CPU 301, it is not limited to the flash memory 304, and an EEPROM (Electrically Erasable and Programmable ROM) or the like may also be used.

[0053] <Communication management system, PC hardware configuration> Next, the hardware configuration of the communication management system 5 and the PC 7 will be described with reference to Fig. 11. Fig. 11 is a diagram showing the hardware configuration of the communication management system and the PC. Note that since the communication management system 5 and the PC 7 are both computers and have the same configuration, the following will describe the configuration of the communication management system 5, and will omit a description of the configuration of the PC 7.

[0054] The communication management system 5 includes a CPU 501 for controlling the overall operation of the communication management system 5, a ROM 502 storing a program used to drive the CPU 501, such as an IPL, a RAM 503 used as a work area for the CPU 501, an HD 504 for storing various data such as programs for the communication management system 5, an HDD (Hard Disk Drive) 505 for controlling reading or writing of various data from or to the HD 504 under the control of the CPU 501, a media drive 507 for controlling reading or writing (storing) of data from or to a recording medium 506, such as a flash memory, a display 508 for displaying various information such as a cursor, menu, window, character, or image, a network I / F 509 for data communication using the communication network 100, a keyboard 511 having multiple keys for inputting characters, numbers, various instructions, etc., a mouse 512 for selecting and executing various instructions, selecting a processing target, moving the cursor, etc., a CD-RW (Compact It is equipped with a CD-RW drive 514 that controls the reading and writing of various data from and to a CD-RW (Disc-Rewritable) 513, and bus lines 510 such as an address bus and a data bus for electrically connecting the above components as shown in FIG. 11.

[0055] <Hardware configuration of smartphone> Next, the hardware of the smartphone will be described with reference to Fig. 12. Fig. 12 is a hardware configuration diagram of the smartphone. As shown in Fig. 12, the smartphone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration / direction sensor 906, a media I / F 908, and a GPS receiver 909.

[0056] Of these, the CPU 901 controls the overall operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901, such as an IPL. The RAM 903 is used as a work area for the CPU 901. The EEPROM 904 reads and writes various data, such as smartphone programs, under the control of the CPU 901. The CMOS (Complementary Metal Oxide Semiconductor) sensor 905 captures images of subjects (mainly self-portraits) and obtains image data under the control of the CPU 901. The acceleration / azimuth sensor 906 is a variety of sensors, such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor. The media I / F 908 controls the reading and writing (storage) of data from and to a recording medium 907, such as a flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0057] The smartphone 9 also includes a long-distance communication circuit 911, a camera 912, an image sensor I / F 913, a microphone 914, a speaker 915, an audio input / output I / F 916, a display 917, an external device connection I / F 918, a short-distance communication circuit 919, an antenna 919a of the short-distance communication circuit 919, and a touch panel 921.

[0058] Of these, the long-distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 901. The image sensor I / F 913 is a circuit that controls the operation of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs audio. The sound input / output I / F 916 is a circuit that processes the input and output of audio signals between the microphone 914 and the speaker 915 under the control of the CPU 901. The display 917 is a type of display means, such as an LCD or organic EL, that displays an image of a subject, various icons, etc. The external device connection I / F 918 is an interface for connecting various external devices. The short-distance communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means that allows a user to operate the smartphone 9 by pressing the display 917.

[0059] The smartphone 9 also includes a bus line 910. The bus line 910 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 901.

[0060] It should be noted that the recording media such as CD-ROMs on which the above programs are stored, and the HDDs on which these programs are stored, can be provided domestically or internationally as program products.

[0061] <<Functional configuration of the embodiment>> Next, the functional configuration of this embodiment will be described with reference to Fig. 13 to Fig. 17. Fig. 13 is a functional block diagram of the image capture devices 1a and 1b, the video conference terminal 3, the communication management system 5, the PC 7, and the smartphone 9, which constitute part of the image communication system of this embodiment.

[0062] <Functional configuration of the imaging device 1a> 13, the image capturing device 1a has a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a storage / readout unit 19a. Each of these units is a function or means realized by operating any of the components shown in FIG. 9 in response to an instruction from the CPU 111 in accordance with a program for the image capturing device that has been loaded from the SRAM 113 onto the DRAM 114.

[0063] The image capturing device 1a also has a storage unit 1000a configured by the ROM 112, SRAM 113, and DRAM 114 shown in Fig. 9. The storage unit 1000a stores a GUID (Globally Unique Identifier) ​​of the image capturing device 1a.

[0064] (Functional configuration of the imaging device 1a) Next, each functional configuration of the image capturing device 1a will be described in more detail with reference to FIGS.

[0065] The reception unit 12a of the photographing device 1a is mainly realized by the processing of the operation unit 115 and CPU 111 shown in FIG. 9, and receives operation inputs from the user.

[0066] The imaging section 13a is mainly realized by the imaging unit 101, image processing unit 104, imaging control unit 105, and processing by the CPU 111 shown in FIG. 9, and captures images of scenery and the like to obtain captured image data.

[0067] The sound collection unit 14a is realized by the microphone 108 and sound processing unit 109 shown in FIG. 9, and processing by the CPU 111, and collects sounds around the image capture device 1a.

[0068] The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38 of the video conference terminal 3 by short-range wireless communication technology such as the NFC standard, BlueTooth, or WiFi.

[0069] The storage / reading unit 19a is mainly realized by the processing of the CPU 111 shown in FIG. 9, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.

[0070] <Functional configuration of the imaging device 1b> The photographing device 1b has a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, and a storage / readout unit 19b. These have the same functions as the reception unit 12a, the imaging unit 13a, the sound collection unit 14a, the communication unit 18a, and the storage / readout unit 19a in the photographing device 1a, respectively, and therefore description thereof will be omitted. The photographing device 1b also has a storage unit 1000b constructed by a ROM 112, an SRAM 113, and a DRAM 114 shown in FIG. 9. The storage unit 1000b stores the GUID of the own device.

[0071] <Functional configuration of video conference terminal 3> 13, the video conference terminal 3 includes a transmitting / receiving unit 31, a receiving unit 32, an image / sound processing unit 33, a display control unit 34, a determining unit 35, a creating unit 36, a communication unit 38, and a storage / reading unit 39. Each of these units is a function or means realized by operating any of the components shown in FIG. 10 in response to an instruction from the CPU 301 in accordance with the program for the video conference terminal 3 loaded from the flash memory 304 onto the RAM 303.

[0072] The video conference terminal 3 also has a storage unit 3000 constructed with a ROM 302, a RAM 303, and a flash memory 304 shown in Fig. 10. An image type management DB 3001 and a photographing device management DB 3002 are constructed in this storage unit 3000. Of these, the image type management DB 3001 is constructed with an image type management table shown in Fig. 14. The photographing device management DB 3002 is constructed with a photographing device management table shown in Fig. 15.

[0073] (Image type management table) FIG. 14 is a conceptual diagram showing an image type management table. In this image type management table, image data IDs, IP addresses (which are an example of destinations of source terminals), and source names are stored and managed in association with each other. Of these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is attached to image data transmitted from the same source terminal. This allows the destination terminal (receiving communication terminal) to identify the source terminal of the received image data. The IP address of the source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for identifying the imaging device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal, such as the videoconferencing terminal 3, in accordance with a predetermined naming convention.

[0074] For example, it is shown that three communication terminals with IP addresses "1.2.1.3," "1.2.2.3," and "1.3.1.3" are respectively transmitting image data indicated by image data IDs "RS001," "RS002," and "RS003." Furthermore, the image types indicated by the source names of each communication terminal are "Video_Theta," "Video," and "Video_Theta," which indicate the image types as "special image," "general image," and "special image," respectively. Note that the special image here is a spherical panoramic image.

[0075] Note that data other than image data may also be managed in association with the image data ID. Data other than image data may be, for example, sound data or document data for screen sharing.

[0076] (Radiation equipment management table) 15 is a conceptual diagram showing an image capturing device management table. In this image capturing device management table, the vendor ID and product ID of the GUID of an image capturing device that can obtain two hemispherical images that are the basis of a celestial sphere panoramic image are stored and managed. As the GUID, for example, a vendor ID (VID) and product ID (PID) used in a USB device can be used. The vendor ID and product ID are stored in a communication terminal such as a video conferencing terminal when it is shipped from the factory, but may also be stored after shipping.

[0077] (Functional configuration of video conferencing terminal 3) Next, each functional configuration of the video conference terminal 3 will be described in more detail with reference to FIGS.

[0078] The transmission / reception unit 31 of the video conference terminal 3 is mainly realized by the processing of the network I / F 311 and CPU 301 shown in Figure 10, and transmits and receives various data (or information) with the communication management system 5 via the communication network 100.

[0079] The reception unit 32 is realized mainly by the operation buttons 308 and processing by the CPU 301, and receives various selections or inputs from the user. In addition to the operation buttons 308, a touch panel or the like may be used as another input means.

[0080] 10, and performs image processing on image data obtained by capturing an image of a subject with camera 312. After the user's voice is converted into an audio signal by microphone 314, image / audio processing unit 33 performs audio processing on the audio data related to this audio signal.

[0081] Furthermore, the image / sound processing unit 33 performs image processing on image data received from another communication terminal based on image type information such as a source name, so that the display control unit 34 can display an image on the display 4. Specifically, when the image type information indicates that the image is a special image, the image / sound processing unit 33 creates spherical panoramic image data by converting the image data (for example, data of each hemispherical image such as shown in FIGS. 3(a) and 3(b)) into spherical panoramic image data such as shown in FIG. 4(b), and further creates a predetermined area image such as shown in FIG. 6(b). In addition, the image / sound processing unit 33 outputs an audio signal related to audio data received from another communication terminal via the communication management system 5 to the speaker 315, and causes the speaker 315 to output audio.

[0082] The display control unit 34 is realized mainly by the processing of the display I / F 317 and the CPU 301, and performs control for displaying various images, characters, and the like on the display 4.

[0083] The determination unit 35 is realized mainly by the processing of the CPU 301, and determines the image type associated with the image data received from the photographing device 1a, for example.

[0084] Creation unit 36 ​​is mainly realized by the processing of CPU 301, and creates a source name, which is an example of image type information, in accordance with the naming rules described above based on the result of determination by determination unit 35 as a general image or a special image (here, a spherical panoramic image). For example, if determination unit 35 determines that the image is a general image, creation unit 36 ​​creates a source name "Video" indicating that it is a general image. On the other hand, if determination unit 35 determines that the image is a special image, creation unit 36 ​​creates a source name "Video_Theta" indicating that it is a special image.

[0085] The communication unit 38 is mainly realized by the short-range communication circuit 319, the antenna 318a, and processing by the CPU 301, and can communicate with the communication unit 18a of the photographing device 1a using short-range wireless technology such as NFC, Bluetooth, WiFi, etc. Although the communication unit 38 and the transceiver unit 31 have been described as having separate communication units, they may also be configured to share a common communication unit.

[0086] The storage / reading unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 10, and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000.

[0087] <Functional configuration of communication management system> Next, each functional configuration of the communication management system 5 will be described in detail with reference to Figures 11 and 13. The communication management system 5 has a transmitting / receiving unit 51, a determining unit 55, a generating unit 56, and a storing / reading unit 59. Each of these units is a function or means realized when any of the components shown in Figure 11 operates in response to an instruction from the CPU 501 in accordance with the program for the communication management system 5 loaded from the HD 504 onto the RAM 503.

[0088] The communication management system 5 also has a storage unit 5000 constructed by the RAM 503 and HD 504 shown in Fig. 11. A session management DB 5001 and an image type management DB 5002 are constructed in this storage unit 5000. Of these, the session management DB 5001 is constructed by the session management table shown in Fig. 16. The image type management DB 5002 is constructed by the image type management table shown in Fig. 17.

[0089] (Session management table) FIG. 16 is a conceptual diagram showing a session management table. In this session management table, session IDs and IP addresses of participating communication terminals are stored and managed in association with each other. Of these, the session ID is an example of session identification information for identifying a communication session that realizes a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal, such as the video conference terminal 3, and is used when selecting a communication session in each communication terminal. The IP addresses of participating communication terminals indicate the IP addresses of communication terminals that have participated in the virtual conference room identified by the associated session ID.

[0090] (Image type management table) FIG. 17 is a conceptual diagram illustrating an image type management table. In addition to the various pieces of information managed in the image type management table shown in FIG. 14, the image type management table shown in FIG. 17 associates and manages the same session IDs as those managed in the session management table. Here, three communication terminals with IP addresses "1.2.1.3," "1.2.2.3," and "1.3.1.3" are participating in the virtual conference room identified by the same session ID "se101." The communication management system 5 manages the same image data ID, source terminal IP address, and image type information managed by communication terminals such as the video conference terminal 3. This is because, when a new communication terminal joins the virtual conference room, image type information and the like are transmitted to communication terminals already in a video call and the newly joining communication terminal. This eliminates the need to transmit image type information and the like between a communication terminal already in a video call and the newly joining communication terminal.

[0091] (Functional configuration of communication management system) Next, each functional configuration of the communication management system 5 will be described in detail with reference to FIGS.

[0092] The transmission / reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 11, and transmits and receives various data (or information) to / from the video conference terminal 3 or the PC 7 via the communication network 100.

[0093] The determination unit 55 is mainly realized by the processing of the CPU 501 and makes various determinations.

[0094] The generation unit 56 is mainly realized by the processing of the CPU 501 and generates an image data ID.

[0095] The storage / reading unit 59 is mainly realized by the HDD 505 shown in FIG. 11 and the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 or reads out various data (or information) from the storage unit 5000.

[0096] <Functional configuration of the PC> Next, the functional configuration of the PC 7 will be described in detail using FIGS. 11 and 13. The PC 7 basically has the same functions as the video conference terminal 3. That is, as shown in FIG. 13, the PC 7 has a transmission / reception unit 71, a reception unit 72, an image / audio processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a communication unit 78, and a storage / reading unit 79. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating according to an instruction from the CPU 501 in accordance with the program for the PC 7 developed from the HD 504 onto the RAM 503.

[0097] In addition, the PC 7 has a storage unit 7000 constructed by the ROM 502, the RAM 503, and the HD 504 shown in FIG. 11. In this storage unit 7000, an image type management DB 7001 and a photographing device management DB 7002 are constructed. Note that since the image type management DB 7001 and the photographing device management DB 7002 have the same configuration as the image type management DB 3001 and the photographing device management DB 3002 respectively, the description thereof is omitted.

[0098] (PC functional configuration) The transmitting / receiving unit 71 of the PC 7 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG.

[0099] The reception unit 72 is realized mainly by processing of the keyboard 511, the mouse 512, and the CPU 501, and realizes the same function as the reception unit 32. The image / sound processing unit 73 is realized mainly by instructions from the CPU 501, and realizes the same function as the image / sound processing unit 33. The display control unit 74 is realized mainly by processing of the CPU 501, and realizes the same function as the display control unit 34. The determination unit 75 is realized mainly by processing of the CPU 501, and realizes the same function as the determination unit 35. The creation unit 76 is realized mainly by processing of the CPU 501, and realizes the same function as the creation unit 36. The communication unit 78 is realized mainly by processing of the CPU 501, and realizes the same function as the communication unit 38. The storage / readout unit 79 is realized by processing of the CPU 501, and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000.

[0100] <Smartphone functional configuration> Next, the functional configuration of the smartphone 9 will be described in detail with reference to Figures 12 and 13. The smartphone 9 basically has the same functions as the video conference terminal 3. That is, as shown in Figure 13, the smartphone 9 has a transmission / reception unit 91, a reception unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a communication unit 98, and a storage / readout unit 99. Each of these units is a function or means realized when any of the components shown in Figure 12 operates in response to an instruction from the CPU 901 in accordance with a program for the smartphone 9 loaded from the EEPROM 904 onto the RAM 903.

[0101] 12. An image type management DB 9001 and an imaging device management DB 9002 are configured in this storage unit 9000. Note that the image type management DB 9001 and the imaging device management DB 9002 have the same configurations as the image type management DB 3001 and the imaging device management DB 3002, respectively, and therefore a description thereof will be omitted.

[0102] (Smartphone functional configuration) The transmitter / receiver 91 of the smartphone 9 is mainly realized by the processing of the long-distance communication circuit 911 and the CPU 901 shown in FIG.

[0103] The reception unit 92 is realized mainly by the processing of the touch panel 921 and the CPU 901 , and realizes the same functions as the reception unit 32 .

[0104] The image / sound processing unit 93 is implemented mainly by instructions from the CPU 901 , and realizes the same functions as the image / sound processing unit 33 .

[0105] The display control unit 94 is realized mainly by the processing of the CPU 901, and realizes the same functions as the display control unit .

[0106] The determination unit 95 is implemented mainly by the processing of the CPU 901, and realizes the same functions as the determination unit .

[0107] The creation unit 96 is realized mainly by the processing of the CPU 901, and realizes the same functions as the creation unit .

[0108] The communication unit 98 is mainly realized by the processing of the CPU 901, and realizes the same functions as the communication unit .

[0109] The storage / readout unit 99 is realized by the processing of the CPU 901 , and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000 .

[0110] <<Processing or Operation of the Embodiment>> <Participation process> Next, the processing or operation of this embodiment will be described with reference to Fig. 18 to Fig. 22. First, the process of joining a specific communication session will be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a sequence diagram showing the process of joining a specific communication session. Fig. 19 is a diagram showing a selection screen for a communication session (virtual conference room).

[0111] First, when a user at site A (e.g., user A1) operates the videoconference terminal 3 to display a selection screen for a communication session (virtual conference room), the reception unit 32 receives the operation to display the selection screen, and the display control unit 34 displays the selection screen as shown in Fig. 19 on the display 4 (step S21). This selection screen displays selection buttons b1, b2, b3, etc., which indicate the virtual conference rooms R1, R2, R3, etc., to be selected. Each selection button b1, etc., is associated with a session ID.

[0112] When user A1 selects a desired selection button (here, selection button b1) in the virtual conference room, the reception unit 32 receives the selection of the communication session (step S22). Then, the transmission / reception unit 31 transmits a request to join the virtual conference room to the communication management system 5 (step S23). This participation request includes a session ID indicating the communication session whose selection was received in step S22 and the IP address of the video conference terminal 3 that is the requesting terminal. As a result, the transmission / reception unit 51 of the communication management system 5 receives the participation request.

[0113] Next, the storage / readout unit 99 performs a process of joining the communication session by adding the IP address received in step S23 to the joining terminal IP address field of the record in the session management DB 5001 (see FIG. 16) that has the same session ID as the session ID received in step S23 (step S24). Then, the transmission / reception unit 51 transmits a join request response to the video conference terminal 3 (step S25). This join request response includes the session ID received in step S23 and the result of the join processing. As a result, the transmission / reception unit 31 of the video conference terminal 3 receives the join request response. The following describes the case where the join processing is successful.

[0114] <Image type information management process> Next, the management process of image type information will be described with reference to Fig. 20. Fig. 20 is a sequence diagram showing the management process of image type information.

[0115] First, when a user at site A (for example, user A1) connects the USB cable of the cradle 2a to which the image capturing device 1a is attached to the video conference terminal 3, the memory / readout unit 19a of the image capturing device 1a reads out the GUID of the image capturing device 1a stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of the image capturing device 1a to the communication unit 38 of the video conference terminal 3 (step S51). As a result, the communication unit 38 of the video conference terminal 3 receives the GUID of the image capturing device 1a.

[0116] Next, the determination unit 35 of the video conference terminal 3 determines the image type by determining whether the same vendor ID and product ID as those in the GUID received in step S51 are managed in the camera device management DB 3002 (see FIG. 15 ) (step S52). Specifically, if the same vendor ID and product ID are managed in the camera device management DB 3002, the determination unit 35 determines that the camera device 1a is a camera device that captures special images (here, omnidirectional panoramic images). On the other hand, if the same vendor ID and product ID are not managed in the camera device management DB 3002, the determination unit 35 determines that the camera device 1a is a camera device that captures general images.

[0117] Next, the storage / readout unit 39 stores the IP address of the terminal (video conference terminal 3) that is the source terminal and the image type information that is the result of the determination made in step S52 in the image type management DB 3001 (see FIG. 14) in association with each other (step S53). In this state, no image data ID is associated. The image type information is, for example, a source name determined according to a predetermined naming rule and an image type (general image, special image).

[0118] Next, the transmitting / receiving unit 31 transmits a request to add image type information to the communication management system 5 (step S54). This request to add image type information includes the IP address of the terminal itself, which is the transmission source terminal stored in step S53, and the image type information. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request to add image type information.

[0119] Next, the storage / reading unit 59 of the communication management system 5 searches the session management DB 5001 (see FIG. 16) using the IP address of the source terminal received in step S54 as a search key, and reads out the corresponding session ID (step S55).

[0120] Next, the generation unit 56 generates a unique image data ID (step S56). Then, the storage / reading unit 59 associates the session ID read out in step S55, the image data ID generated in step S56, and the IP address of the transmission source terminal and image type information received in step S54 and stores them as a new record in the image type management DB 5002 (see FIG. 17) (step S57). Then, the transmission / reception unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3. As a result, the transmission / reception unit 31 of the video conference terminal 3 receives the image data ID (step S58).

[0121] Next, the storage / readout unit 39 of the video conference terminal 3 stores the image data ID received in step S58 in the image type management DB 3001 (see FIG. 14) in association with the IP address and image type information of the terminal (video conference terminal 3) that is the source terminal stored in step S53 (step S59).

[0122] Meanwhile, the transmitter / receiver 51 of the communication management system 5 transmits an addition notification of image type information to the smartphone 9, which is another communication terminal (step S60). This addition notification of image type information includes the image data ID generated in step S56, as well as the IP address and image type information of the terminal (video conference terminal 3), which is the sender terminal, stored in step S53. As a result, the transmitter / receiver 91 of the smartphone 9 receives the addition notification of image type information. Note that the destination of the transmission / reception unit 51 is another IP address that is associated in the session management DB 5001 (see FIG. 16) with the same session ID as the IP address of the video conference terminal 3. In other words, the destination is another communication terminal that is in the same virtual conference room as the video conference terminal 3.

[0123] Next, the storage / readout unit 99 of the smartphone 9 associates the image data ID, the IP address of the sending terminal, and the image type information received in step S60 and stores them as a new record in the image type management DB 9001 (see FIG. 14) (step S61). Similarly, an additional communication of the image type information is sent to another communication terminal, PC 7, which also stores the information in the image type management DB 7001 (see FIG. 14). As a result, the communication terminals can share the same information in their image type management DBs 3001, 7001, and 9001.

[0124] <Image data communication processing> The communication process of image data in a video call will be described with reference to Fig. 21 to Fig. 23. Fig. 21 is a sequence diagram showing the communication process of image data in a video call.

[0125] First, image data obtained by capturing an image of a subject, scenery, or the like is transmitted from the communication unit 18a of the image capturing device 1a to the communication unit 38 of the video conference terminal 3 (step S101). In this case, since the image capturing device 1a is a device capable of obtaining two hemispherical images that are the basis for a celestial sphere panoramic image, the image data is composed of data of the two hemispherical images, as shown in Figures 3(a) and 3(b). As a result, the communication unit 38 of the video conference terminal 3 receives the image data.

[0126] Next, the transmitting / receiving unit 31 of the video conference terminal 3 transmits the image data sent from the image capturing device 1a to the communication management system 5 (step S102). This transmission includes an image data ID for identifying the image data to be transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the image data and the image data ID.

[0127] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits the image data sent from the video conference terminal 3 to the smartphone 9 (step S103). This transmission includes an image data ID for identifying the image data to be transmitted. As a result, the transmitting / receiving unit 51 of the smartphone 9 receives the image data and the image data ID.

[0128] Next, in the smartphone 9, the storage / readout unit 99 searches the image type management DB 9001 (see FIG. 14 ) using the image data ID received in step S103 as a search key, thereby reading out corresponding image type information (source name) (step S104). Then, if the image type information indicates that the image is a special image (here, a celestial sphere panoramic image) (Video_Theta), the image / sound processing unit 93 creates a celestial sphere panoramic image from the image data received in step S103, and further creates a predetermined area image (step S105). In this case, the image / sound processing unit 93 combines an icon 191 indicating that the image is a celestial sphere panoramic image (described later) with the predetermined area image, based on the image type information indicating that the image is a special image (Video_Theta).

[0129] Next, the display control unit 94 displays the predetermined area image including the icon 191 on the display 917 of the smartphone 9 (step S106). Note that if the image type information indicates a general image (Video), the image / sound processing unit 93 does not create a celestial sphere panoramic image from the image data received in step S103, and the display control unit 94 displays a general image that does not include the icon 191.

[0130] Next, the state of a video call will be explained using Fig. 22. Fig. 22 is an image diagram showing the state of a video call. Of these, Fig. 22(a) shows a case where the image capturing device 1a is not used, and Fig. 22(b) shows a case where the image capturing device 1a is used.

[0131] First, as shown in FIG. 22(a), when using the camera 312 (see FIG. 10) pre-installed in the video conference terminal 3 without using the image capture device 1a, the angle of view is 125 degrees horizontally and 70 degrees vertically, so the video conference terminal 3 must be placed at the edge of the desk where each user A1, etc. can be seen. This means that each user A1, etc. must face the video conference terminal 3 when speaking. Furthermore, because each user A1, etc. faces the video conference terminal 3, the display 4 must also be placed on the side of the video conference terminal 3. As a result, users A2 and A4 who are far from the video conference terminal 3 are far from the microphone 314 (see FIG. 10), so they must speak relatively loudly and the content displayed on the display 4 is difficult to see.

[0132] 22(b), when the image capturing device 1a is used, two hemispherical images that are the basis for the omnidirectional panoramic image can be obtained, and therefore the video conference terminal 3 and the display 4 can be placed relatively close to the center of the desk. This allows each user A1, etc., to speak in a relatively quiet voice because they are close to the microphone 314, and the contents displayed on the display 4 can be easily seen.

[0133] Next, a display example of the display 917 at site B will be described with reference to FIG. 23. FIG. 23 shows a display example of the display at site B. Of these, FIG. 23(a) shows a case where a celestial sphere panoramic image and a predetermined area image are not created from the image data transmitted from the video conference terminal 3 (camera 1a) and the camera 1b, and the image data transmitted from the PC 7 (camera 8) is displayed as is. FIG. 23(b) shows a case where a celestial sphere panoramic image and a predetermined area image are created from the image data transmitted from the video conference terminal 3 (camera 1a) and the camera 1b, and the image data transmitted from the PC 7 (camera 8) is displayed as is. Note that an image of site A is displayed in the upper left display area of ​​the display 917, an image of site B (the user's site) is displayed in the upper right display area, and an image of site C is displayed in the lower left display area. In this example, since a simultaneous video call is being conducted between three sites, nothing is displayed in the lower right display area.

[0134] When image data sent from image capturing devices 1a and 1b capable of capturing spherical panoramic images is displayed as is, it is displayed as shown in Fig. 23(a), where location A on the upper left and location B (own location) on the upper right are displayed as the front and rear hemispherical images, respectively, as shown in Figs.

[0135] In contrast, when the image / sound processing unit 93 creates a celestial sphere panoramic image from image data sent from the photographing devices 1a and 1b that can obtain two hemispherical images that are the basis of the celestial sphere panoramic image, and further creates a predetermined area image, the predetermined area image, which is a planar image, is displayed as shown in Fig. 23(b). Note that, since the site C is equipped with the photographing device 8 that obtains a general image, a general image is displayed in both Figs. 23(a) and (b).

[0136] Furthermore, an icon 191 indicating that the image is a spherical panoramic image is displayed at the top left of each of the images from site A and site B (own site). The display position of icon 191 need not be the top left, but may be anywhere, such as the top right, bottom left, or bottom right. The type of icon 191 is not limited to that shown in FIG. 23(b). Instead of icon 191, text such as "spherical image" may be displayed, or a combination of an icon and text may be displayed.

[0137] Furthermore, users can change the predetermined area of ​​the predetermined area image in the same omnidirectional panoramic image. That is, users B1 and B2 place their fingers on the touch panel 921 of the smartphone 9 and move them, causing the reception unit 92 to receive the finger movement, and the display control unit 94 to shift, rotate, reduce, or enlarge the predetermined area image. As a result, even if only users A1 and A2 at site A are displayed in the initial setting (default), as shown in FIG. 23(b), the predetermined area image can be shifted to display users A4 and A3.

[0138] <<Main Effects of This Embodiment>> As described above, according to this embodiment, a communication terminal such as the video conference terminal 3 can create a celestial sphere panoramic image and further create a predetermined area image using corresponding image type information based on an image data ID transmitted together with image data. This has the effect of preventing the front hemispherical image and the rear hemispherical image from being displayed, as shown in Fig. 23(a).

[0139] [Other examples] Another example of the management process of image type information will be described with reference to Fig. 24. Fig. 24 is a sequence diagram showing another example of the management process of image type information.

[0140] First, when a user at site A (for example, user A1) connects the USB cable of the cradle 2a to which the image capturing device 1a is attached to the video conference terminal 3, the storage / readout unit 19a of the image capturing device 1a reads out the GUID of the image capturing device 1a stored in the storage unit 1000a, and the communication unit 18a transmits the GUID of the image capturing device 1a to the communication unit 38 of the video conference terminal 3 (step S151). As a result, the communication unit 38 of the video conference terminal 3 receives the GUID of the image capturing device 1a.

[0141] Next, the transmitting / receiving unit 31 of the video conference terminal 3 transmits a request for the image data ID to the communication management system 5 (step S152). This request includes the IP address of the video conference terminal 3, which is the transmission source terminal (own terminal). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request for the image data ID.

[0142] Next, the generation unit 56 generates a unique image data ID (step S153). Then, the communication management system 5 transmits a notification of the image data ID to the smartphone 9 (step S154). This notification includes the image data ID created in step S153 and the IP address of the video conference terminal 3 as the source terminal received in step S152. As a result, the transmission / reception unit 91 of the smartphone 9 receives the notification of the image data ID. Then, in the smartphone 9, the storage / readout unit 99 associates the image data ID and the IP address of the source terminal received in step S154 with each other and stores them as a new record in the image type management DB 9001 (see FIG. 14) (step S155). At this point, image type information has not been associated and stored.

[0143] Meanwhile, in response to the request for the image data ID in step S152, the communication management system 5 causes the transmitting / receiving unit 51 to transmit the image data ID to the video conference terminal 3 (step S156). As a result, the transmitting / receiving unit 31 of the video conference terminal 3 receives the image data ID. Then, in the video conference terminal 3, the storage / reading unit 39 associates the image data ID received in step S156 with the IP address of the sender terminal (own terminal) managed by the video conference terminal 3 as a new record in the image type management DB 3001 (see FIG. 14) (step S157). At this point, the image type information has not been associated and stored.

[0144] Next, the determination unit 35 of the video conference terminal 3 determines the image type by determining whether the same vendor ID and product ID as those of the GUID received in step S151 are managed in the imaging device management DB 3002 (see FIG. 15) (step S158). This determination is the same as the process in step S52.

[0145] Next, the storage / readout unit 39 stores in the image type management DB 3001 (see FIG. 14) the IP address of the terminal (video conference terminal 3) that is the source terminal and the image type information that is the result of the determination made in step S158, in association with the image data ID already stored in step S157 (step S159). Then, the transmission / reception unit 31 transmits a notification of the image type information to the communication management system 5 (step S160). This notification of the image type information includes the image data ID stored in step S157, and the IP address and image type information of the terminal (video conference terminal 3) that is the source terminal that are stored in step S159. As a result, the transmission / reception unit 51 of the communication management system 5 receives the notification of the image type information.

[0146] Next, the storage / reading unit 59 of the communication management system 5 searches the session management DB 5001 (see FIG. 16) using the IP address of the source terminal received in step S160 as a search key, and reads out the corresponding session ID (step S161).

[0147] Next, the storage / readout unit 59 associates the session ID read out in step S161 with the IP address of the source terminal, the image data ID, and the image type information received in step S160 and stores them as a new record in the image type management DB 5002 (see FIG. 17) (step S162). Then, the communication management system 5 transmits a notification of the image type information to the smartphone 9, which is another communication terminal (step S163). This notification of the image type information includes the image data ID and image type information stored in step S162. As a result, the transmission / reception unit 91 of the smartphone 9 receives the notification of the image type information.

[0148] Next, the storage / readout unit 99 of the smartphone 9 associates the image type information received in step S163 with the same image data ID as the image data ID received in step S163 and additionally stores the information in the image type management DB 9001 (see FIG. 14) (step S164), thereby completing the image type information management process. [Explanation of symbols]

[0149] 1a Imaging device 1b Imaging device 3. Video conference terminal (an example of a communication terminal, an example of a first communication terminal) 5. Communications Management System 7 PC (an example of a communication terminal) 8. Imaging Device 9. Smartphone (an example of a communication terminal, an example of a second communication terminal) 31 Transmitter / Receiver 51 Transmitting / receiving unit (an example of a transmitting means, an example of a receiving means) 56 Generation unit (an example of generation means) [Prior art documents] [Patent documents]

[0150] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-178135 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-223076

Claims

1. Computer, receiving means for receiving image data; an image processing means for creating a celestial sphere panoramic image from the received image data when the received image data indicates that the celestial sphere panoramic image is a celestial sphere panoramic image; functioning as a display control means for displaying an image of a partial area of ​​the celestial sphere panoramic image created by the image processing means; The display control means If the received image data does not indicate that the image is a spherical panoramic image, an image indicated by the received image data is displayed. program.

2. the display control means changes and displays a part of the celestial sphere panoramic image in response to an operation by a user to change the part of the celestial sphere panoramic image. The program according to claim 1.

3. 3. The program according to claim 1, wherein the display control means changes and displays the partial area of ​​the celestial sphere panoramic image in response to receiving any one of an operation of shifting, rotating, reducing, or enlarging the partial area of ​​the celestial sphere panoramic image from a user.

4. The computer further functioning as a determination means for determining whether the received image data indicates a celestial sphere panoramic image; The display control means When it is determined that the received image data indicates a celestial sphere panoramic image, an image showing a partial area of ​​the celestial sphere panoramic image is displayed; If it is determined that the received image data does not indicate a spherical panoramic image, an image indicated by the image data is displayed. The program according to any one of claims 1 to 3.

5. The program according to claim 1 , wherein the spherical panoramic image is an image captured by an imaging device that captures the spherical panoramic image.

6. The program according to claim 1 , wherein the display control means displays an icon indicating a spherical panoramic image on an image indicating a partial area of ​​the spherical panoramic image.

7. A method for providing a display according to image data to a communication terminal connected to a communication network by a computer connected to the communication network, comprising: creating a celestial sphere panoramic image from the image data when the image data indicates that the celestial sphere panoramic image is a celestial sphere panoramic image; displaying an image of a partial area of ​​the created celestial sphere panoramic image; and if the image data does not indicate that the image is a spherical panoramic image, displaying the image indicated by the image data. How to provide the display.

8. A receiving means for receiving image data; an image processing means for creating a celestial sphere panoramic image from the received image data when the received image data indicates that the celestial sphere panoramic image is a celestial sphere panoramic image; a display control unit that displays an image of a partial area of ​​the celestial sphere panoramic image created by the image processing unit, The display control means If the received image data does not indicate that the image is a spherical panoramic image, an image indicated by the received image data is displayed. Communications terminal.

9. receiving means for receiving image data; an image processing means for creating a celestial sphere panoramic image from the received image data when the received image data indicates that the celestial sphere panoramic image is a celestial sphere panoramic image; a display control unit that displays an image of a partial area of ​​the celestial sphere panoramic image created by the image processing unit, The display control means If the received image data does not indicate that the image is a spherical panoramic image, an image indicated by the received image data is displayed. Image communication system.

Images