Program, method for providing a display, communication terminal, and image communication system

Abstract

To solve the problem of generation of extremely hard-to-see images caused by a communication terminal receiving image data acquired from a special imaging apparatus and displaying the image data together with a plane image, which results in displaying two hemisphere images as origins of an entire-celestial-sphere panoramic image, or the communication terminal receiving image data acquired from a general imaging apparatus and performing processing of creating an entire-celestial-sphere panoramic image, which results in destroying the image.SOLUTION: A method includes the steps of: receiving image type information indicating a type of an image from a first communication terminal (S54); on the basis of reception of the image type information, generating image data identification information for identifying image data to be transmitted from the first communication terminal to a second communication terminal (S56); transmitting the generated image data identification information to the first communication terminal (S58); and transmitting the generated image data identification information and the received image type information to the second communication terminal (S60).SELECTED DRAWING: Figure 20

Description

Technical Field

[0001] The present invention relates to Relating to a program, a method for providing a display, a communication terminal, and an image communication system .

Background Art

[0002] Conference systems for holding remote conferences with remote locations via communication networks such as the Internet have become widespread. In this conference system, in a conference room where one of the parties such as the attendees of the remote conference is located, an image of the conference room such as the parties of the conference and voices such as speeches are photographed and collected using a communication terminal of the remote conference system, and these are converted into digital data and transmitted to the communication terminal of the other party. As a result, an image can be displayed on the display of the other party's conference room and voice can be output by a speaker, enabling a video call, and thus a technique for holding a conference between remote locations in a state close to an actual conference is known (see Patent Document 1).

[0003] On the other hand, a technique is known in which a photographing device capable of acquiring an omnidirectional panoramic image in real time is connected to a communication terminal, and the omnidirectional panoramic image acquired from this photographing device is distributed to each communication terminal of the other party, and in each communication terminal of the other party, sequential image conversion processing is performed on the omnidirectional panoramic image into a rectangular planar image and displayed on a display or the like (see Patent Document 2).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the types of photographing devices used at each site are different, such as a general photographing device that obtains a rectangular planar image and a special photographing device that obtains two hemispherical images that are the basis of an omnidirectional panoramic image, the following problems occur on the communication terminal side that has received the image data.

[0005] That is, when a communication terminal that has received image data obtained from a special imaging device displays it according to a planar image, two hemispherical images that are the basis of the omnidirectional panoramic image are displayed. As shown in Fig. 23(a), since user A1 etc. are displayed downward in the spherical image, it becomes a very difficult-to-see image. On the other hand, if a communication terminal that has received image data obtained from a general imaging device performs a process of creating an omnidirectional panoramic image, the image will be damaged and become a very difficult-to-see image.

Means for Solving the Problem

[0006] In order to solve the above-described problems and achieve the object, a program according to the present invention causes a computer to function as a receiving unit that receives image data from a communication management system, and a display control unit that causes a display unit to display a first display area and a second display area. When the type of the received image data is a planar image, the planar image related to the received image data is displayed in the first display area. When the type of the received image data is an omnidirectional panorama image, an image of a first area of a part of the omnidirectional panorama image related to the received image data is displayed in the second display area. In response to an operation by a user to change the first area, the image of the first area displayed in the second display area is changed to and displayed as an image of a second area of a part of the omnidirectional panorama image, which is the second area changed based on the operation.

Effect of the Invention

[0007] According to the present invention, on the second communication terminal side that has received the image data, image processing can be performed according to the image type information, so that the effect of being able to display an easy-to-see image is achieved.

Brief Explanation of the Drawings

[0008]

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig. 14

Fig. 15

Fig. 16

Fig. 17

Fig. 18

Fig. 19

Fig. 20

Fig. 21

Fig. 22

Fig. 23

Fig. 24

DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0010] <<OVERVIEW OF THE EMBODIMENT>> <METHOD FOR GENERATING AN OMNIDIRECTIONAL PANORAMIC IMAGE> The method for generating an omnidirectional panoramic image will be described with reference to FIGS. 1 to 7.

[0011] First, the appearance of the imaging device 1 will be described with reference to FIG. 1. The imaging device 1 is a digital camera for obtaining a captured image that is the basis of an omnidirectional (360°) panoramic image. Note that FIG. 1(a) is a left side view of the imaging device, FIG. 1(b) is a front view of the imaging device, and FIG. 1(c) is a plan view of the imaging device.

[0012] As shown in FIG. 1(a), the imaging device 1 is sized such that a person can hold it with one hand. Also, as shown in FIGS. 1(a), 1(b), and 1(c), on the upper part of the imaging device 1, an imaging element 103a is provided on the front side (front) and an imaging element 103b is provided on the back side (rear). These imaging elements (image sensors) 103a and 103b are used in combination with an optical member (for example, the fish-eye lenses 102a and 102b described later) capable of capturing a hemispherical image (viewing angle of 180° or more). Also, as shown in FIG. 1(b), an operation unit 115 such as a shutter button is provided on the surface of the imaging device 1 opposite to the front side.

[0013] Next, the usage status of the imaging device 1 will be described with reference to FIG. 2. Note that FIG. 2 is an image diagram of the usage of the imaging device. As shown in FIG. 2, the imaging device 1 is used, for example, by a user holding it in their hand to capture a subject around the user. In this case, the imaging elements 103a and 103b shown in FIG. 1 capture the subjects around the user, respectively, to obtain two hemispherical images.

[0014] Next, with reference to FIGS. 3 and 4, an outline of the process from the image captured by the imaging device 1 to the creation of the omnidirectional panoramic image will be described. Note that FIG. 3(a) is a hemispherical image (front side) captured by the imaging device, FIG. 3(b) is a hemispherical image (rear side) captured by the imaging device, and FIG. 3(c) is a diagram showing an image represented by the Mercator projection method (hereinafter referred to as the "Mercator image"). FIG. 4(a) is a conceptual diagram showing a state where the sphere is covered with the Mercator image, and FIG. 4(b) is a diagram showing the omnidirectional panoramic image.

[0015] As shown in FIG. 3(a), the image obtained by the imaging element 103a becomes a hemispherical image (front side) curved by the fish-eye lens 102a described later. Also, as shown in FIG. 3(b), the image obtained by the imaging element 103b becomes a hemispherical image (rear side) curved by the fish-eye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image (rear side) inverted by 180 degrees are synthesized by the imaging device 1, and as shown in FIG. 3(c), a Mercator image is created.

[0016] By using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 4(a), the Mercator image is pasted so as to cover the spherical surface, and a panoramic image of the entire sphere as shown in FIG. 4(b) is created. In this way, the panoramic image of the entire sphere is represented as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the panoramic image of the entire sphere may be a still image or a moving image.

[0017] As described above, since the panoramic image of the entire sphere is an image pasted so as to cover the spherical surface, it will feel uncomfortable to humans. Therefore, by displaying a partial predetermined area of the panoramic image of the entire sphere (hereinafter referred to as a "predetermined area image") as a planar image with less curvature, it is possible to perform a display that does not give an uncomfortable feeling to humans. This will be described with reference to FIGS. 5 and 6.

[0018] Note that FIG. 5 is a diagram showing the position of a virtual camera and a predetermined region when the omnidirectional panorama image is a three-dimensional spherical object. The virtual camera IC corresponds to the position of the viewpoint of a user who views the image with respect to the omnidirectional panorama image displayed as a three-dimensional spherical object. Further, FIG. 6(a) is a perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a predetermined region image when displayed on a display. Further, in FIG. 6(a), the omnidirectional panorama image shown in FIG. 4 is represented by a three-dimensional spherical object CS. If the omnidirectional panorama image generated in this way is the spherical object CS, as shown in FIG. 5, the virtual camera IC is located outside the omnidirectional panorama image. The predetermined region T in the omnidirectional panorama image is the shooting region of the virtual camera IC, and is specified by predetermined region information indicating position coordinates (x(rH), y(rV), viewing angle α(angle)) including the viewing angle of the virtual camera IC in a three-dimensional virtual space including the omnidirectional panorama image. The zoom of the predetermined region T can be expressed by expanding or contracting the range (arc) of the viewing angle α. Further, the zoom of the predetermined region T can also be expressed by moving the virtual camera IC closer to or farther from the omnidirectional panorama image.

[0019] Then, the predetermined region image, which is an image of the predetermined region T shown in FIG. 6(a), is displayed on a predetermined display as an image of the shooting region of the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined region image represented by predetermined region information set by default. Note that the predetermined region information may be indicated by the shooting region (X, Y, Z) of the virtual camera IC, which is the predetermined region T, instead of the position coordinates of the virtual camera IC. Hereinafter, the description will be made using the position coordinates (x(rH), y(rV), and viewing angle α(angle)) of the virtual camera IC.

[0020] Next, with reference to FIG. 7, the relationship between the predetermined region information and the image of the predetermined region T will be described. Note that FIG. 7 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in FIG. 7, when the diagonal angle of view of the predetermined region T represented by the angle of view α of the virtual camera IC is 2L, the center point CP becomes the (x, y) parameters of the predetermined region information. f is the distance from the virtual camera IC to the center point CP. L is the distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is the diagonal). And in FIG. 7, generally, the trigonometric function represented by the following formula (1) holds.

[0021] Lf = tan(α / 2) ··· (Formula 1)

[0022] <Outline of the Image Communication System> Subsequently, with reference to FIG. 8, an outline of the configuration of the image communication system of the present embodiment will be described. FIG. 8 is a schematic diagram of the configuration of the image communication system of the present embodiment.

[0023] As shown in FIG. 8, the image communication system of the present embodiment is composed of photographing devices 1a, 1b, video conference terminals 3, a communication management system 5, a PC (Personal Computer) 7, a photographing device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

[0024] Among these, the photographing devices 1a, 1b are special digital cameras for photographing a subject, scenery, etc. to obtain two hemispherical images that are the basis of the omnidirectional panoramic image. On the other hand, the photographing device 8 is a general digital camera for photographing a subject, scenery, etc. to obtain a general planar image.

[0025] The video conferencing terminal 3 is a dedicated terminal for video conferencing, and it displays the images of video calls (video conferences) on the display 4 via a wired cable such as a USB (Universal Serial Bus) cable. The video conferencing terminal 3 usually shoots with a camera 312 described later, but when it is connected by a wired cable to the cradle 2a to which the imaging device 1a is attached, the imaging device 1a is prioritized, and two hemispherical images that are the basis of the omnidirectional panoramic image can be obtained. When using a wired cable, the cradle 2a not only communicates between the imaging device 1a and the video conferencing terminal 3, but also plays a role in supplying power to the imaging device 1a and supporting the imaging device 1a. Here, the imaging device 1a, the cradle 2a, the video conferencing terminal 3, and the display 4 are placed at the same location, location A. Also, at location A, four users A1, A2, A3, and A4 are participating in the video call.

[0026] The communication management system 5 manages the communication of the video conferencing terminal 3, the PC 7, and the smartphone 9, and manages the types of image data (general image and special image types) transmitted and received. Here, the special image is an omnidirectional panoramic image. Note that the communication management system 5 is installed in a service company or the like that provides video communication services. Also, the communication management system 5 may be constructed by a single computer, or may be constructed by a plurality of computers arbitrarily assigned by dividing each part (function, means, or storage part).

[0027] The PC 7 can make a video call by attaching the imaging device 8. Here, the PC 7 and the imaging device 8 are placed at the same location, location C. Also, at location C, one user C 1 is participating in the video call.

[0028] The smartphone 9 displays the image of the video call on the display 917 provided in the device, which will be described later. The smartphone 9 usually takes pictures with a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 etc. provided in the device itself, but it can also acquire two hemispherical image data that are the basis of the omnidirectional panoramic image obtained by the imaging device 1b by using wireless communication technologies such as WiFi (Wireless Fidelity) and Bluetooth (registered trademark). When using wireless communication technologies, the cradle 2b only serves to supply power to the imaging device 1b and support the imaging device 1b. Here, the imaging device 1b, the cradle 2b, and the smartphone 9 are placed at the same base, base B. Also, two users B1 and B2 are participating in the video call at base B.

[0029] Also, the video conferencing terminal 3, the PC 7, and the smartphone 9 are examples of communication terminals. OpenGL ES is installed in each communication terminal, and it can create predetermined area information indicating a partial area of the omnidirectional panoramic image, or create a predetermined area image from the omnidirectional image panorama sent from another communication terminal.

[0030] Note that the arrangement of each terminal, device, and user shown in FIG. 8 is an example, and other examples may also be possible. For example, at base C, instead of the imaging device 8, an imaging device capable of taking pictures related to the omnidirectional panoramic image may be used by the user. Also, communication terminals include digital TVs, smartwatches, car navigation devices, etc. Also, hereinafter, when representing any one of the imaging devices 1a and 1b, it is represented as "imaging device 1".

[0031] <<Hardware Configuration of the Embodiment>> Next, with reference to FIGS. 9 to 12, the hardware configurations of the imaging device 1, the video conferencing terminal 3, the communication management system 5, the PC 7, and the smartphone 9 of this embodiment will be described in detail. Since the imaging device 8 is a general camera, detailed description thereof is omitted.

[0032] <Hardware Configuration of Imaging Device 1> First, the hardware configuration of the imaging device 1 will be described with reference to FIG. 9. FIG. 9 is a hardware configuration diagram of the imaging device 1. Hereinafter, the imaging device 1 is assumed to be an omnidirectional imaging device using two imaging elements, but the number of imaging elements may be two or more. Also, it does not necessarily have to be a device dedicated to omnidirectional imaging. By attaching an after-market omnidirectional imaging unit to an ordinary digital camera, smartphone, etc., it may be made to have substantially the same functions as the imaging device 1.

[0033] As shown in FIG. 9, the imaging device 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, an audio processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I / F 116, a communication unit 117, Electronic compass 118 and an antenna 117a.

[0034] Among these, the imaging unit 101 includes wide-angle lenses 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image (so-called fisheye lenses), and two imaging elements 103a and 103b provided corresponding to the respective wide-angle lenses. The imaging elements 103a and 103b include image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors that convert the optical image formed by the fisheye lenses 102a and 102b into image data of an electrical signal and output it, a timing generation circuit that generates a horizontal or vertical synchronization signal and a pixel clock of this image sensor, and a register group to which various commands and parameters necessary for the operation of this imaging element are set.

[0035] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus) separately. The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via the bus 110. Further, the ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are also connected to the bus 110.

[0036] The image processing unit 104 captures the image data output from the imaging elements 103a and 103b through the parallel I / F bus, performs predetermined processing on each image data, and then synthesizes these image data to create the data of the Mercator image as shown in Fig. 3(c).

[0037] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using the I2C bus, with the imaging control unit 105 as the master device and the imaging elements 103a and 103b as the slave devices. The necessary commands and the like are received from the CPU 111. Also, the imaging control unit 105 fetches the status data and the like of the register groups of the imaging elements 103a and 103b using the I2C bus and sends them to the CPU 111.

[0038] In addition, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some imaging devices 1 may have a preview display function using a display (for example, the display of the video conferencing terminal 3) or a function corresponding to video display. In this case, the output of the image data from the imaging elements 103a and 103b is continuously performed at a predetermined frame rate (frames / minute).

[0039] In addition, as will be described later, the imaging control unit 105 also functions as synchronization control means for synchronizing the output timing of the image data of the imaging elements 103a and 103b in cooperation with the CPU 111. Note that in this embodiment, the imaging device 1 is not provided with a display unit (display), but a display unit may be provided.

[0040] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 captures the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0041] The CPU 111 controls the overall operation of the imaging device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories that store programs executed by the CPU 111, data during processing, and the like. In particular, the DRAM 114 stores image data during processing in the image processing unit 104 and data of processed Mercator images.

[0042] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. The user inputs various shooting modes, shooting conditions, and the like by operating the operation buttons.

[0043] The network I / F 116 is a general term for interface circuits (such as USB I / F) with external media such as SD cards and personal computers. Also, the network I / F 116 may be wireless or wired. The data of the Mercator image stored in the DRAM 114 is recorded on an external medium via this network I / F 116, or transmitted to an external device such as the video conferencing terminal 3 via the network I / F 116 as necessary.

[0044] The communication unit 117 communicates with an external device such as the video conferencing terminal 3 via the antenna 117a provided in the imaging device 1 by using short-range wireless technologies such as WiFi and NFC (Near Field Communication). Also, the communication unit 117 can transmit the data of the Mercator image to the external device of the video conferencing terminal 3.

[0045] The electronic compass 118 calculates the orientation and tilt (Roll rotation angle) of the imaging device 1 from the earth's magnetism and outputs the orientation and tilt information. This orientation and tilt information is an example of related information (metadata) conforming to Exif and is used for image processing such as image correction of the captured image. Note that the related information also includes each data such as the shooting date and time of the image and the data capacity of the image data.

[0046] <Hardware Configuration of Video Conferencing Terminal> Next, the hardware configuration of the video conferencing terminal 3 will be described with reference to FIG. 10. FIG. 10 is a hardware configuration diagram of the video conferencing terminal. As shown in FIG. 10, the video conferencing terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, operation buttons 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an imaging element I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319.

[0047] Among these, the CPU 301 controls the operation of the entire video conferencing terminal 3. The ROM 302 stores programs used for driving the CPU 301 such as the IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data such as communication programs, image data, and audio data. The SSD (Solid State Drive) 305 controls the reading or writing of various data to and from the flash memory 304 according to the control of the CPU 301. Note that an HDD may be used instead of the SSD. The media I / F 307 controls the reading or writing (storage) of data to and from a recording medium 306 such as a flash memory. The operation button 308 is a button that is operated when selecting a destination of the video conferencing terminal 3, etc. The power switch 309 is a switch for switching the ON / OFF of the power of the video conferencing terminal 3.

[0048] Also, the network I / F 311 is an interface for performing data communication using a communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that captures a subject according to the control of the CPU 301 to obtain image data. The image sensor I / F 313 is a circuit that controls the driving of the camera 312. The microphone 314 is a type of built-in sound collection means for inputting sound. The audio input / output I / F 316 is a circuit that processes the input and output of audio signals between the microphone 314 and the speaker 315 according to the control of the CPU 301. The display I / F 317 is a circuit that transmits image data to an external display 4 according to the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark), Bluetooth (registered trademark), etc.

[0049] Also, the bus line 310 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 301 shown in FIG. 10.

[0050] The display 4 is a type of display means composed of a liquid crystal or an organic EL that displays an image of a subject, an operation icon, etc. Further, the display 4 is connected to the display I / F 317 by a cable 4c. This cable 4c may be a cable for an analog RGB (VGA) signal, a cable for component video, or a cable for an HDMI (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signal. HDMI is a registered trademark.

[0051] Note that the camera 312 includes a lens and a solid-state imaging device that converts light into electric charges to digitize an image (video) of a subject. As the solid-state imaging device, a CMOS sensor, a CCD sensor, etc. are used. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 by a USB (Universal Serial Bus) cable or the like. When an external camera is connected, the external camera is driven in preference to the built-in camera 312 under the control of the CPU 301. Similarly, when an external microphone or an external speaker is connected, the external microphone or the external speaker is driven in preference to the built-in microphone 314 or the built-in speaker 315 under the control of the CPU 301.

[0052] Also, the recording medium 306 is configured to be detachable from the video conferencing terminal 3. Further, as long as it is a non-volatile memory that reads or writes data according to the control of the CPU 301, not limited to the flash memory 304, an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

[0053] <Communication management system, hardware configuration of PC> Next, the hardware configurations of the communication management system 5 and the PC 7 will be described with reference to FIG. 11. FIG. 11 is a hardware configuration diagram of the communication management system and the PC. Since both the communication management system 5 and the PC 7 are computers with the same configuration, the configuration of the communication management system 5 will be described below, and the description of the configuration of the PC 7 will be omitted.

[0054] The communication management system 5 includes a CPU 501 that controls the overall operation of the communication management system 5, a ROM 502 that stores programs used for driving the CPU 501 such as IPL, a RAM 503 used as a work area for the CPU 501, an HD 504 that stores various data such as programs for the communication management system 5, an HDD (Hard Disk Drive) 505 that controls the reading or writing of various data to and from the HD 504 according to the control of the CPU 501, a media drive 507 that controls the reading or writing (storage) of data to and from a recording medium 506 such as a flash memory, a display 508 that displays various information such as a cursor, menu, window, characters, or images, a network I / F 509 for data communication using the communication network 100, a keyboard 511 having a plurality of keys for inputting characters, numerical values, various instructions, etc., a mouse 512 for selecting and executing various instructions, selecting a processing target, moving a cursor, etc., a CD-RW drive 514 that controls the reading or writing of various data to and from a CD-RW (Compact Disc-ReWritable) 513 as an example of a removable recording medium, and a bus line 510 such as an address bus and a data bus for electrically connecting the above-described components as shown in FIG. 11.

[0055] <Hardware Configuration of Smart Phone> Next, the hardware of the smart phone will be described with reference to FIG. 12. FIG. 12 is a hardware configuration diagram of the smart phone. As shown in FIG. 12, the smart phone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration / azimuth sensor 906, a media I / F 908, and a GPS receiver 909.

[0056] Among these, the CPU 901 controls the operation of the entire smartphone 9. The ROM 902 stores programs used for driving the CPU 901 such as the IPL. The RAM 903 is used as a work area for the CPU 901. The EEPROM 904 reads or writes various data such as smartphone programs according to the control of the CPU 901. The CMOS (Complementary Metal Oxide Semiconductor) sensor 905 captures a subject (mainly a self-image) according to the control of the CPU 901 to obtain image data. The acceleration / azimuth sensor 906 is various sensors such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor that detect geomagnetism. The media I / F 908 controls the reading or writing (storage) of data to / from a recording medium 907 such as a flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0057] Also, the smartphone 9 includes a long-distance communication circuit 911, a camera 912, an image sensor I / F 913, a microphone 914, a speaker 915, an audio input / output I / F 916, a display 917, an external device connection I / F 918, a short-distance communication circuit 919, an antenna 919a for the short-distance communication circuit 919, and a touch panel 921.

[0058] Among these, the long-distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures a subject according to the control of the CPU 901 to obtain image data. The imaging device I / F 913 is a circuit that controls the driving of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs sound. The audio input / output I / F 916 is a circuit that processes the input / output of sound signals between the microphone 914 and the speaker 915 according to the control of the CPU 901. The display 917 is a type of display means such as a liquid crystal or an organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 918 is an interface for connecting various external devices. The short-distance communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means for operating the smartphone 9 when the user presses the display 917.

[0059] Also, the smartphone 9 includes a bus line 910. The bus line 910 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 901.

[0060] Note that a recording medium such as a CD-ROM in which each of the above programs is stored, and an HD in which these programs are stored can both be provided as a program product to domestic or foreign regions.

[0061] <<Functional Configuration of Embodiment>> Next, with reference to FIGS. 13 to 17, the functional configuration of this embodiment will be described. FIG. 13 is a functional block diagram of the photographing devices 1a and 1b, the video conference terminal 3, the communication management system 5, the PC 7, and the smartphone 9 that constitute a part of the image communication system of this embodiment.

[0062] <Functional Configuration of Photographing Device 1a> As shown in FIG. 13, the imaging device 1a includes a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a storage / reading unit 19a. Each of these units is a function or means realized by any of the components shown in FIG. 9 operating according to instructions from the CPU 111 according to a program for the imaging device developed from the SRAM 113 onto the DRAM 114.

[0063] Also, the imaging device 1 a has a storage unit 1000a constructed by the ROM 112, SRAM 113, and DRAM 114 shown in FIG. 9. The GUID (Globally Unique Identifier) of the device itself is stored in the storage unit 1000a.

[0064] (Functional configurations of the imaging device 1a) Next, with reference to FIGS. 9 and 13, each functional configuration of the imaging device 1a will be described in more detail.

[0065] The reception unit 12a of the imaging device 1a is mainly realized by the operation unit 115 and the processing of the CPU 111 shown in FIG. 9, and receives operation inputs from the user.

[0066] The imaging unit 13a is mainly realized by the imaging unit 101, the image processing unit 104, and the imaging control unit 105 shown in FIG. 9, and the processing of the CPU 111, and captures a scene or the like to obtain captured image data.

[0067] The sound collection unit 14a is realized by the microphone 108 and the sound processing unit 109 shown in FIG. 9, and the processing of the CPU 111, and collects the sounds around the imaging device 1 a .

[0068] The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38 of the video conferencing terminal 3 by short-range wireless communication technologies such as the NFC standard, Bluetooth, and WiFi.

[0069] The memory / reading unit 19a is mainly realized by the processing of the CPU 111 shown in FIG. 9, and stores various data (or information) in the storage unit 1000a or reads out various data (or information) from the storage unit 1000a.

[0070] <Functional configurations of the imaging device 1b> The imaging device 1b includes a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, and a memory / reading unit 19b. Since these respectively have the same functions as the reception unit 12a, imaging unit 13a, sound collection unit 14a, communication unit 18a, and memory / reading unit 19a in the imaging device 1a, the description thereof is omitted. Further, the imaging device 1 b has a storage unit 1000b constructed by the ROM 112, SRAM 113, and DRAM 114 shown in FIG. 9. The GUID of the device itself is stored in the storage unit 1000b.

[0071] <Functional configurations of the video conferencing terminal 3> As shown in FIG. 13, the video conferencing terminal 3 includes a transmission / reception unit 31, a reception unit 32, an image / sound processing unit 33, a display control unit 34, a determination unit 35, a creation unit 36, a communication unit 38, and a memory / reading unit 39. Each of these units is a function or means realized by any of the components shown in FIG. 10 operating according to an instruction from the CPU 301 in accordance with the program for the video conferencing terminal 3 expanded from the flash memory 304 onto the RAM 303.

[0072] Further, the video conferencing terminal 3 has a storage unit 3000 constructed by the ROM 302, RAM 303, and flash memory 304 shown in FIG. 10. An image type management DB 3001 and a photographing device management DB 3002 are constructed in this storage unit 3000. Among these, the image type management DB 3001 is configured by the image type management table shown in FIG. 14. The photographing device management DB 3002 is configured by the photographing device management table shown in FIG. 15.

[0073] (Image type management table) FIG. 14 is a conceptual diagram showing an image type management table. In this image type management table, an image data ID, an IP address which is an example of the destination of the source terminal, and a source name are stored and managed in association with each other. Among these, the image data ID is an example of image data identification information for identifying image data during video communication. The same image data ID is added to the image data transmitted from the same source terminal. Thereby, the destination terminal (the communication terminal on the receiving side) can specify the source terminal of the received image data. The IP address of the source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for specifying the imaging device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal such as the video conferencing terminal 3 according to a naming rule of a predetermined name.

[0074] For example, it is shown that three communication terminals with IP addresses of "1.2.1.3", "1.2.2.3", and "1.3.1.3" respectively transmit image data indicated by image data IDs "RS001", "RS002", and "RS003". Furthermore, the types of images indicated by the source names of each communication terminal are "Video_Theta", "Video", and "Video_Theta", which indicate in order that the image types are "special image", "general image", and "special image". Here, the special image is an omnidirectional panoramic image.

[0075] Note that data other than image data may also be managed in association with the image data ID. Data other than image data is, for example, audio data and material data during screen sharing.

[0076] (Imaging Device Management Table) FIG. 15 is a conceptual diagram showing a photographing device management table. In this photographing device management table, the vendor ID and product ID among the GUIDs of the photographing devices from which two hemispherical images that are the basis of the omnidirectional panoramic image can be obtained are stored and managed. As the GUID, for example, the vendor ID (VID) and product ID (PID) used for USB devices can be used. This vendor ID and product ID are stored since the factory shipment of communication terminals such as video conference terminals, but may also be additionally stored after factory shipment.

[0077] (Each functional configuration of the video conference terminal 3) Next, with reference to FIGS. 10 and 13, each functional configuration of the video conference terminal 3 will be described in more detail.

[0078] The transmission / reception unit 31 of the video conference terminal 3 is mainly realized by the processing of the network I / F 311 and the CPU 301 shown in FIG. 10, and transmits and receives various data (or information) to and from the communication management system 5 via the communication network 100.

[0079] The reception unit 32 is mainly realized by the processing of the operation button 308 and the CPU 301, and receives various selections or inputs from the user. Also, not only the operation button 308 but also other input means such as a touch panel may be used.

[0080] The image / audio processing unit 33 is realized by an instruction from the CPU 301 shown in FIG. 10, and performs image processing on the image data obtained by the camera 312 imaging the subject. Also, after the user's voice is converted into an audio signal by the microphone 314, the image / audio processing unit 33 performs audio processing on the audio data related to this audio signal.

[0081] Furthermore, for the display control unit 34 to display an image on the display 4, the image / sound processing unit 33 performs image processing on the image data received from another communication terminal based on the image type information such as the source name. Specifically, when the image type information indicates that it is a special image, the image / sound processing unit 33 creates omnidirectional panorama image data by converting the image data (for example, the data of each hemisphere image as shown in FIGS. 3(a) and (b)) into omnidirectional panorama image data as shown in FIG. 4(b), and further creates a predetermined area image as shown in FIG. 6(b). Also, the image / sound processing unit 33 outputs the audio signal related to the sound data received from another communication terminal via the communication management system 5 to the speaker 315 to output sound from the speaker 315.

[0082] The display control unit 34 is mainly realized by the processing of the display I / F 317 and the CPU 301, and performs control to display various images, characters, etc. on the display 4.

[0083] The determination unit 35 is mainly realized by the processing of the CPU 301, and determines, for example, the image type related to the image data received from the imaging device 1a.

[0084] The creation unit 36 is mainly realized by the processing of the CPU 301, and creates a source name, which is an example of the image type information, according to the above naming rule based on the result determined by the determination unit 35 as a general image or a special image (here, an omnidirectional panorama image). For example, when the determination unit 35 determines that it is a general image, the creation unit 36 creates a source name "Video" indicating that it is a general image. On the other hand, when the determination unit 35 determines that it is a special image, the creation unit 36 creates a source name "Video_Theta" indicating that it is a special image.

[0085] The communication unit 38 is mainly realized by the processing of the short - range communication circuit 319, the antenna 318a, and the CPU 301, and can communicate with the communication unit 18a of the imaging device 1a by short - range wireless technologies such as NFC, Bluetooth, and WiFi. Although the communication unit 38 and the transmission / reception unit 31 are described as having separate communication units, they may also have a shared configuration.

[0086] The storage / reading unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 10, and stores various data (or information) in the storage unit 3000 or reads out various data (or information) from the storage unit 3000.

[0087] <Functional Configuration of Communication Management System> Next, with reference to FIGS. 11 and 13, each functional configuration of the communication management system 5 will be described in detail. The communication management system 5 includes a transmission / reception unit 51, a determination unit 55, a generation unit 56, and a storage / reading unit 59. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating according to instructions from the CPU 501 according to the communication management system 5 program deployed on the RAM 503 from the HD 504.

[0088] In addition, the communication management system 5 has a storage unit 5000 constructed by the RAM 503 and the HD 504 shown in FIG. 11. In this storage unit 5000, a session management DB 5001 and an image type management DB 5002 are constructed. Among these, the session management DB 5001 is composed of the session management table shown in FIG. 16. The image type management DB 5002 is composed of the image type management table shown in FIG. 17.

[0089] (Session Management Table) FIG. 16 is a conceptual diagram showing a session management table. In this session management table, a session ID and the IP addresses of the participating communication terminals are stored and managed in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session that realizes a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal such as the video conference terminal 3, and is used when selecting a communication session at each communication terminal. The IP address of the participating communication terminal indicates the IP address of the communication terminal that has participated in the virtual conference room indicated by the associated session ID.

[0090] (Image type management table) FIG. 17 is a conceptual diagram showing an image type management table. The image type management table shown in FIG. 17 manages, in addition to each piece of information managed by the image type management table shown in FIG. 14, the same session ID as the session ID managed by the session management table in association with it. Here, it is shown that three communication terminals with IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1.3" have participated in the virtual conference room indicated by the same session ID "se101". In the communication management system 5, the reason for managing the same image data ID, the IP address of the source terminal, and the image type information by communication terminals such as the video conference terminal 3 is to transmit the image type information and the like to the communication terminal that is already in a video call and the newly participating communication terminal when a new communication terminal enters a virtual conference room or the like. Thereby, it is not necessary to perform transmission and reception of image type information and the like between the communication terminal that is already in a video call and the newly participating communication terminal.

[0091] (Each functional configuration of the communication management system) Next, each functional configuration of the communication management system 5 will be described in detail with reference to FIGS. 11 and 13.

[0092] The transmission / reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 11, and transmits and receives various data (or information) to / from the video conference terminal 3 or the PC 7 via the communication network 100.

[0093] The determination unit 55 is mainly realized by the processing of the CPU 501, and makes various determinations.

[0094] The generation unit 56 is mainly realized by the processing of the CPU 501, and generates an image data ID.

[0095] The storage / reading unit 59 is mainly realized by the processing of the HDD 505 shown in FIG. 11 and the CPU 501, and stores various data (or information) in the storage unit 5000 or reads out various data (or information) from the storage unit 5000.

[0096] <Functional configuration of the PC> Next, with reference to FIGS. 11 and 13, the functional configuration of the PC 7 will be described in detail. The PC 7 basically has the same functions as the video conference terminal 3. That is, as shown in FIG. 13, the PC 7 has a transmission / reception unit 71, a reception unit 72, an image / audio processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a communication unit 78, and a storage / reading unit 79. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating according to an instruction from the CPU 501 according to the program for the PC 7 expanded from the HD 504 onto the RAM 503.

[0097] In addition, the PC 7 has a storage unit 7000 constructed by the ROM 502, the RAM 503, and the HD 504 shown in FIG. 11. An image type management DB 7001 and a photographing device management DB 7002 are constructed in this storage unit 7000. Note that since the image type management DB 7001 and the photographing device management DB 7002 have the same configurations as the image type management DB 3001 and the photographing device management DB 3002, respectively, the description thereof is omitted.

[0098] (Functional components of the PC) The transmitting / receiving unit 71 of the PC 7 is mainly realized by the network I / F 509 and the processing of the CPU 501 shown in FIG. 11, and realizes the same functions as the transmitting / receiving unit 31.

[0099] The reception unit 72 is mainly realized by the keyboard 511, the mouse 512, and the processing of the CPU 501, and realizes the same functions as the reception unit 32. The image / audio processing unit 73 is mainly realized by commands from the CPU 501, and realizes the same functions as the image / audio processing unit 33. The display control unit 74 is mainly realized by the processing of the CPU 501, and realizes the same functions as the display control unit 34. The determination unit 75 is mainly realized by the processing of the CPU 501, and realizes the same functions as the determination unit 35. The creation unit 76 is mainly realized by the processing of the CPU 501, and realizes the same functions as the creation unit 36. The communication unit 78 is mainly realized by the processing of the CPU 501, and realizes the same functions as the communication unit 38. The storage / reading unit 79 is realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 7000 or reads out various data (or information) from the storage unit 7000.

[0100] <Functional components of the smartphone> Next, with reference to FIGS. 12 and 13, the functional components of the smartphone 9 will be described in detail. The smartphone 9 basically has the same functions as the video conferencing terminal 3. That is, as shown in FIG. 13, the smartphone 9 has a transmitting / receiving unit 91, a reception unit 92, an image / audio processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a communication unit 98, and a storage / reading unit 99. Each of these units is a function or means realized by any of the components shown in FIG. 12 operating according to commands from the CPU 901 according to the smartphone 9 program expanded from the EEPROM 904 onto the RAM 903.

[0101] In addition, the smartphone 9 has a storage unit 9000 constructed by a ROM 902, a RAM 903, and an EEPROM 904 shown in FIG. 12. An image type management DB 9001 and a photographing device management DB 9002 are constructed in this storage unit 9000. Note that since the image type management DB 9001 and the photographing device management DB 9002 have the same configurations as the image type management DB 3001 and the photographing device management DB 3002, respectively, their descriptions are omitted.

[0102] (Functional configurations of the smartphone) The transmission / reception unit 91 of the smartphone 9 is mainly realized by the processing of a long-distance communication circuit 911 and a CPU 901 shown in FIG. 12, and realizes the same functions as the transmission / reception unit 31.

[0103] The reception unit 92 is mainly realized by the processing of a touch panel 921 and a CPU 901, and realizes the same functions as the reception unit 32.

[0104] The image / audio processing unit 93 is mainly realized by commands from the CPU 901, and realizes the same functions as the image / audio processing unit 33.

[0105] The display control unit 94 is mainly realized by the processing of the CPU 901, and realizes the same functions as the display control unit 34.

[0106] The determination unit 95 is mainly realized by the processing of the CPU 901, and realizes the same functions as the determination unit 35.

[0107] The creation unit 96 is mainly realized by the processing of the CPU 901, and realizes the same functions as the creation unit 36.

[0108] The communication unit 98 is mainly realized by the processing of the CPU 901, and realizes the same functions as the communication unit 38.

[0109] The storage / reading unit 99 is realized by the processing of the CPU 901, and stores various data (or information) in the storage unit 9000 or reads out various data (or information) from the storage unit 9000.

[0110] <<Processing or operation of the embodiment>> <Processing of participation> Subsequently, with reference to FIGS. 18 to 22, the processing or operation of this embodiment will be described. First, with reference to FIGS. 18 and 19, the participation processing for a specific communication session will be described. FIG. 18 is a sequence diagram showing the participation processing for a specific communication session. FIG. 19 is a diagram showing a selection screen for a communication session (virtual conference room).

[0111] First, when a user at Site A (for example, User A1) performs an operation to display a selection screen for a communication session (virtual conference room) on the video conferencing terminal 3, the reception unit 32 receives the operation to display the selection screen, and the display control unit 34 displays a selection screen as shown in FIG. 19 on the display 4 (step S21). On this selection screen, selection buttons b1, b2, b3, etc. indicating each virtual conference room R1, R2, R3, etc. to be selected are displayed. Also, each session ID is associated with each selection button b1, etc.

[0112] Here, when User A1 selects a desired selection button for the virtual conference room (here, selection button b1), the reception unit 32 receives the selection of the communication session (step S22). Then, the transceiver unit 31 transmits a participation request for the virtual conference room to the communication management system 5 (step S23). This participation request includes the session ID indicating the communication session selected in step S22 and the IP address of the video conferencing terminal 3 which is the source terminal of the request. Thereby, the transceiver unit 51 of the communication management system 5 receives the participation request.

[0113] Next, the memory / reading unit 99 performs the participation process for the communication session by adding the IP address received in step S23 to the field of the participating terminal IP address in the record with the same session ID as the session ID received in step S23 in the session management DB 5001 (see FIG. 16) (step S24). Then, the transmission / reception unit 51 transmits a participation request response to the video conference terminal 3 (step S25). This participation request response includes the session ID received in step S23 and the participation process result. As a result, the transmission / reception unit 31 of the video conference terminal 3 receives the participation request response. Hereinafter, the case where the participation process is successful will be described.

[0114] <Management Process of Image Type Information> Subsequently, the management process of image type information will be described with reference to FIG. 20. FIG. 20 is a sequence diagram showing the management process of image type information.

[0115] First, when a user at Site A (for example, user A1) connects the USB cable of the cradle 2a with the imaging device 1a attached to the video conference terminal 3, the memory / reading unit 19a of the imaging device 1a reads the GUID of its own device (imaging device 1a) stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of its own device to the communication unit 38 of the video conference terminal 3 (step S51). As a result, the communication unit 38 of the video conference terminal 3 receives the GUID of the imaging device 1a.

[0116] Next, the determination unit 35 of the video conferencing terminal 3 determines the image type by determining whether the vendor ID and product ID in the GUID received in step S51 are managed in the imaging device management DB3002 (see FIG. 15) (step S52). Specifically, in the imaging device management DB3002, if the same vendor ID and product ID are managed, the determination unit 35 determines that the imaging device 1a is an imaging device that captures special images (here, an omnidirectional panoramic image). On the other hand, if the same vendor ID and product ID are not managed in the imaging device management DB3002, the determination unit 35 determines that the imaging device 1a is an imaging device that captures general images.

[0117] Next, the storage / reading unit 39 stores, in the image type management DB3001 (see FIG. 14), the IP address of the own terminal (video conferencing terminal 3) which is the transmission source terminal, in association with the image type information which is the determination result determined in step S52 (step S53). In this state, the image data ID is not associated. The image type information is, for example, a source name defined according to a predetermined naming rule, or an image type (general image, special image).

[0118] Next, the transmission / reception unit 31 transmits a request for adding image type information to the communication management system 5 (step S54). This request for adding image type information includes the IP address of the own terminal which is the transmission source terminal stored in step S53, and the image type information. As a result, the transmission / reception unit 51 of the communication management system 5 receives the request for adding image type information.

[0119] Next, the storage / reading unit 59 of the communication management system 5 reads out the corresponding session ID by searching the session management DB5001 (see FIG. 16) using the IP address of the transmission source terminal received in step S54 as a search key (step S55).

[0120] Next, the generation unit 56 generates a unique image data ID (step S56). Then, the storage / reading unit 59 stores, as a new record in the image type management DB5002 (see FIG. 17), the session ID read in step S55, the image data ID generated in step S56, and the IP address and image type information of the transmitting source terminal received in step S54 in an associated manner (step S57). Then, the transmission / reception unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3. As a result, the transmission / reception unit 31 of the video conference terminal 3 receives the image data ID (step S58).

[0121] Next, the storage / reading unit 39 of the video conference terminal 3 stores the image data ID received in step S58 in association with the IP address and image type information of its own terminal (video conference terminal 3), which is the transmitting source terminal, stored in step S53, in the image type management DB3001 (see FIG. 14) (step S59).

[0122] On the other hand, the transmission / reception unit 51 of the communication management system 5 transmits an additional notification of image type information to the smartphone 9, which is another communication terminal (step S60). This additional notification of image type information includes the image data ID generated in step S56, and the IP address and image type information of its own terminal (video conference terminal 3), which is the transmitting source terminal, stored in step S53. As a result, the transmission / reception unit 91 of the smartphone 9 receives the additional notification of image type information. Note that the transmission destination of the transmission / reception unit 51 is another IP address associated with the same session ID as the IP address of the video conference terminal 3 in the session management DB5001 (see FIG. 16). That is, the transmission destination is another communication terminal that has entered the same virtual conference room as the video conference terminal 3.

[0123] Next, the memory / readout unit 99 of the smartphone 9 stores, as a new record in the image type management DB 9001 (see FIG. 14), the image data ID, the IP address of the source terminal, and the image type information received in step S60 in an associated manner (step S61). Similarly, additional communication of the image type information is also transmitted to the PC 7, which is another communication terminal, and the PC 7 also stores it in the image type management DB 7001 (see FIG. 14). As described above, each communication terminal can share the same information in each of the image type management DBs 3001, 7001, and 9001.

[0124] <Communication Processing of Image Data> The communication processing of image data in a video call will be described with reference to FIGS. 21 to 23. FIG. 21 is a sequence diagram showing the communication processing of image data in a video call.

[0125] First, the communication unit 18a of the imaging device 1a transmits image data obtained by photographing a subject, a landscape, etc. to the communication unit 38 of the video conferencing terminal 3 (step S101). In this case, since the imaging device 1a is a device that can obtain two hemispherical images that are the basis of the omnidirectional panoramic image, as shown in FIGS. 3(a) and (b), the image data is composed of the data of the two hemispherical images. As a result, the communication unit 38 of the video conferencing terminal 3 receives the image data.

[0126] Next, the transceiver unit 31 of the video conferencing terminal 3 transmits the image data sent from the imaging device 1a to the communication management system 5 (step S102). This transmission includes an image data ID for identifying the image data to be transmitted. As a result, the transceiver unit 51 of the communication management system 5 receives the image data and the image data ID.

[0127] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits the image data sent from the video conferencing terminal 3 to the smartphone 9 (step S103). This transmission includes an image data ID for identifying the image data to be transmitted. As a result, the transmitting / receiving unit 51 of the smartphone 9 receives the image data and the image data ID.

[0128] Next, in the smartphone 9, the storage / reading unit 99 searches the image type management DB9001 (see FIG. 14) using the image data ID received in step S103 as a search key, and reads out the corresponding image type information (source name) (step S104). Then, when the image type information indicates that the image is a special image (here, an omnidirectional panoramic image) (Video_Theta), the image / audio processing unit 93 creates an omnidirectional panoramic image from the image data received in step S103, and further creates a predetermined region image (step S105). In this case, the image / audio processing unit 93 synthesizes an icon 191 indicating that it is an omnidirectional panoramic image, which will be described later, into the predetermined region image based on the image type information indicating that it is a special image (Video_Theta).

[0129] Next, the display control unit 94 displays the predetermined region image including the icon 191 on the display 917 of the smartphone 9 (Step S106) . When the image type information indicates that the image is a general image (Video), the image / audio processing unit 93 does not create an omnidirectional panoramic image from the image data received in step S103, and the display control unit 94 displays a general image that does not include the icon 191.

[0130] Subsequently, the state of the video call will be described with reference to FIG. 22. FIG. 22 is an image diagram showing the state of the video call. Among them, FIG. 22(a) shows the case where the imaging device 1a is not used, and FIG. 22(b) shows the case where the imaging device 1a is used.

[0131] First, as shown in Fig. 22(a), when using the camera 312 (see Fig. 10) provided in the video conference terminal 3 in advance without using the imaging device 1a, since the viewing angle is 125 degrees horizontally and 70 degrees vertically, the video conference terminal 3 has to be placed at the edge of the table where each user A1, etc. is reflected. For this reason, each user A1, etc. has to talk facing the video conference terminal 3 side. Also, since each user A1, etc. faces the video conference terminal 3, the display 4 will also be placed on the side of the video conference terminal 3. As a result, users A2 and A4 who are far from the video conference terminal 3 are far from the microphone 314 (see Fig. 10), so they have to talk in a relatively loud voice, and it is difficult to see the display content of the display 4.

[0132] On the other hand, as shown in Fig. 22(b), when using the imaging device 1a, since two hemispherical images that are the basis of the omnidirectional panoramic image can be obtained, the video conference terminal 3 and the display 4 can be placed relatively in the center of the table. As a result, each user A1, etc. is close to the microphone 314, so they can talk in a relatively quiet voice, and the display content of the display 4 is also easier to see.

[0133] Next, with reference to FIG. 23, a display example of the display 917 at the base B will be described. FIG. 23 is a display example of the display at the base B. Among these, FIG. 23(a) shows a case where the image data sent from the video conferencing terminal 3 (imaging device 1a) and the imaging device 1b is displayed as it is without creating an omnidirectional panoramic image and a predetermined area image, and a case where the image data sent from the PC 7 (imaging device 8) is displayed as it is. FIG. 23(b) shows a case where an omnidirectional panoramic image and a predetermined area image are created from the image data sent from the video conferencing terminal 3 (imaging device 1a) and the imaging device 1b, and a case where the image data sent from the PC 7 (imaging device 8) is displayed as it is. Note that an image of the base A is displayed in the upper left display area of the display 917, an image of the base B (own base) is displayed in the upper right display area, and an image of the base C is displayed in the lower left display area. Also, in this example, since a three-site simultaneous video call is being made, nothing is displayed in the lower right display area.

[0134] When the image data sent from the imaging devices 1a and 1b capable of capturing an omnidirectional panoramic image is displayed as it is, it is displayed as shown in FIG. 23(a). This is because the base A on the upper left side and the base B (own base) on the upper right side are each displayed as a front hemisphere image and a rear hemisphere image as shown in FIGS. 3(a) and (b).

[0135] On the other hand, when the image and sound processing unit 93 creates an omnidirectional panoramic image and further creates a predetermined area image from the image data sent from the imaging devices 1a and 1b that can obtain two hemisphere images that are the basis of the omnidirectional panoramic image, a predetermined area image that is a planar image is displayed as shown in FIG. 23(b). Note that since the imaging device 8 for obtaining a general image is provided at the base C, a general image is displayed in both FIGS. 23(a) and (b).

[0136] Furthermore, an icon 191 indicating that it is an omnidirectional panorama image is displayed at the upper left of each of the images of base point A and base point B (own base point). Note that the display position of the icon 191 may be anywhere, such as the upper right, lower left, and lower right, instead of the upper left. Also, the type of the icon 191 is not limited to that shown in Fig. 23(b). Instead of the icon 191, characters such as "omnidirectional image" may be used, or a combination of an icon and characters may be used.

[0137] In addition, the user can change a predetermined area related to a predetermined area image in the same omnidirectional panorama image. That is, users B1 and B2 touch and move a finger on the touch panel 921 of the smartphone 9, the reception unit 92 receives the movement of the finger, and the display control unit 94 can shift, rotate, reduce, or enlarge the predetermined area image. As a result, as shown in Fig. 23(b), even when only some of the users A1 and A2 of base point A are displayed in the initial setting (default), the users A4 and A3 can be displayed by shifting the predetermined area image.

[0138] <<Main effects of this embodiment>> As described above, according to this embodiment, a communication terminal such as the video conferencing terminal 3 can create an omnidirectional panorama image and further create a predetermined area image based on the corresponding image type information according to the image data ID sent together with the image data. As a result, as shown in Fig. 23(a), it is possible to prevent the front hemisphere image and the rear hemisphere image from being displayed.

[0139] 〔Other examples〕 Another example of the management process of the image type information will be described with reference to Fig. 24. Fig. 24 is a sequence diagram showing another example of the management process of the image type information.

[0140] First, when a user at base A (for example, user A1) connects the USB cable of the cradle 2a with the imaging device 1a attached to the video conferencing terminal 3, the storage / readout unit 19a of the imaging device 1a reads out the GUID of its own device (imaging device 1a) stored in the storage unit 1000a, and the communication unit 18a transmits the GUID of its own device to the communication unit 38 of the video conferencing terminal 3 (step S151). As a result, the communication unit 38 of the video conferencing terminal 3 receives the GUID of the imaging device 1a.

[0141] Next, the transceiver unit 31 of the video conferencing terminal 3 transmits a request for an image data ID to the communication management system 5 (step S152). This request includes the IP address of the video conferencing terminal 3, which is the source terminal (own terminal). As a result, the transceiver unit 51 of the communication management system 5 receives the request for the image data ID.

[0142] Next, the generation unit 56 generates a unique image data ID (step S153). Then, the communication management system 5 transmits a notification of the image data ID to the smartphone 9 (step S154). This notification includes the image data ID created in step S153 and the IP address of the video conferencing terminal 3 as the source terminal received in step S152. As a result, the transceiver unit 91 of the smartphone 9 receives the notification of the image data ID. Then, in the smartphone 9, the storage / readout unit 99 stores the image data ID received in step S154 and the IP address of the source terminal in association with each other as a new record in the image type management DB9001 (see FIG. 14) (step S155). At this point, the image type information is not stored in association.

[0143] On the other hand, in order to respond to the request for the image data ID in step S152, the transmission / reception unit 51 of the communication management system 5 transmits the image data ID to the video conference terminal 3 (step S156). As a result, the transmission / reception unit 31 of the video conference terminal 3 receives the image data ID. Then, in the video conference terminal 3, the storage / readout unit 39 stores, as a new record, the image data ID received in step S156 and the IP address of the transmission source terminal (own terminal) managed by the video conference terminal 3 in the image type management DB3001 (see FIG. 14) in an associated manner (step S157). At this point, the image type information is not stored in an associated manner.

[0144] Next, the determination unit 35 of the video conference terminal 3 determines the image type by determining whether the vendor ID and product ID of the GUID received in step S151 are managed in the imaging device management DB3002 (see FIG. 15) (step S158). This determination is the same as the process in step S52 described above.

[0145] Next, the storage / readout unit 39 stores, in the image type management DB3001 (see FIG. 14), the IP address of the own terminal (video conference terminal 3) which is the transmission source terminal and the image type information which is the determination result determined in step S158 in an associated manner with respect to the image data ID already stored in step S157 (step S159). Then, the transmission / reception unit 31 transmits a notification of the image type information to the communication management system 5 (step S160). This notification of the image type information includes the image data ID stored in step S157, and the IP address of the own terminal which is the transmission source terminal and the image type information stored in step S159. As a result, the transmission / reception unit 51 of the communication management system 5 receives the notification of the image type information.

[0146] Next, the storage / reading unit 59 of the communication management system 5 searches the session management DB5001 (see FIG. 16) using the IP address of the source terminal received in step S160 as a search key, and reads out the corresponding session ID (step S161).

[0147] Next, the storage / reading unit 59 stores, as a new record in the image type management DB5002 (see FIG. 17), the session ID read out in step S161, as well as the IP address of the source terminal, the image data ID, and the image type information received in step S160, in an associated manner (step S162). Then, the communication management system 5 transmits a notification of the image type information to the smartphone 9, which is another communication terminal (step S163). This notification of the image type information includes the image data ID and the image type information stored in step S162. As a result, the transmission / reception unit 91 of the smartphone 9 receives the notification of the image type information.

[0148] Next, the storage / reading unit 99 of the smartphone 9 additionally stores, in the image type management DB9001 (see FIG. 14), the image type information received in step S163 in an associated manner with the same image data ID as the image data ID received in step S163 (step S164). Thereby, the management process of the image type information is completed.

Explanation of Signs

[0149] 1a Photographing device 1b Photographing device 3 Video conferencing terminal (an example of a communication terminal, an example of a first communication terminal) 5 Communication management system 7 PC (an example of a communication terminal) 8 Photographing device 9 Smartphone (an example of a communication terminal, an example of a second communication terminal) 31 Transmission / reception unit 51 Transmission / reception unit (an example of transmission means, an example of reception means) 56 Generation unit (an example of generation means)

Prior Art Documents

Patent Document

[0150]

Patent Document 1

Patent Document 2

Claims

1. Computer, a receiving means for receiving image data from the communication management system; a display control unit that displays the first display area and the second display area on the display unit; The display control means If the type of the received image data is a planar image, the planar image according to the received image data is displayed in the first display area; When the type of the received image data is a celestial sphere panoramic image, an image of a first region that is a part of the celestial sphere panoramic image according to the received image data is displayed in the second display region; in response to a user's operation to change the first area, the image of the first area displayed in the second display area is changed to an image of a second area that is a part of the celestial sphere panoramic image and that has been changed based on the operation, and the image is displayed. program.

2. The computer further functioning as a determination unit for determining whether the type of the received image data is a spherical panoramic image or a planar image; the display control means, when it is determined that the type of the received image data is a planar image, causes the planar image according to the received image data to be displayed in the first display area, and, when it is determined that the type of the received image data is a celestial sphere panoramic image, causes an image of a first area that is a part of the celestial sphere panoramic image according to the received image data to be displayed in the second display area. The program according to claim 1.

3. 3. The program according to claim 1, wherein, in response to receiving, from a user, any one of an operation to shift, rotate, reduce, or enlarge the first region that is a part of the celestial sphere panoramic image, the display control means displays an image of the second region that is shifted, rotated, reduced, or enlarged based on the received operation, instead of the image of the first region that is a part of the celestial sphere panoramic image.

4. The program according to claim 1 , wherein the first display area and the second display area are displayed on the same display unit.

5. The program according to claim 1 , wherein the spherical panoramic image is an image captured by an imaging device that captures the spherical panoramic image.

6. The program according to any one of claims 1 to 5, wherein the planar image is an image captured by a photographing device for photographing the planar image.

7. The program according to any one of claims 1 to 6, wherein the display control means displays an icon indicating a full-sphere panorama image on an image of a first partial region in the full-sphere panorama image.

8. A method in which a computer connected to a communication network provides a display corresponding to image data to a communication terminal connected to the communication network, comprising: displaying a first display area and a second display area on a display unit; when the type of the image data is a planar image, displaying the planar image related to the image data in the first display area; when the type of the image data is a full-sphere panorama image, displaying an image of a first partial region in the full-sphere panorama image related to the image data in the second display area; changing and displaying the image of the first region displayed in the second display area to an image of a second partial region in the full-sphere panorama image, which is the second region changed based on the operation, in response to an operation by a user of the communication terminal to change the first region. A method for providing a display.

9. receiving means for receiving image data from a communication management system; display control means for displaying a first display area and a second display area on a display unit, wherein the display control means: when the type of the received image data is a planar image, displays the planar image related to the received image data in the first display area, and when the type of the received image data is a full-sphere panorama image, displays an image of a first partial region in the full-sphere panorama image related to the received image data in the second display area; in response to an operation by a user to change the first region, changes and displays the image of the first region displayed in the second display area to an image of a second partial region in the full-sphere panorama image, which is the second region changed based on the operation. A communication terminal.

10. receiving means for receiving image data; display control means for displaying a first display area and a second display area on a display unit, wherein the display control means: When the type of the received image data is a planar image, display the planar image corresponding to the received image data in the first display area; when the type of the received image data is an omnidirectional panorama image, display an image of a first area of a part of the omnidirectional panorama image corresponding to the received image data in the second display area. In response to an operation by the user to change the first area, change and display the image of the first area displayed in the second display area to an image of a second area of a part of the omnidirectional panorama image, where the second area is changed based on the operation. An image communication system.

Images