Image communication system, communication terminal, display method and program

Abstract

To provide a communication terminal, an image communication system, a display method, and a program that can prevent a delay of image data even when transmitting and receiving data of a full 360-degree spherical image and a superposition image.SOLUTION: In an image communication system, a communication terminal capable of displaying, on display means, a predetermined area image that is an image of a predetermined area in the entire image, has: receiving means that receives data of the entire image, data of a superposition image, and superposition position information indicating a position for superposing the superposition image in the entire image, which are transmitted by another communication terminal; image processing means that superposes the superposition image on the entire image based on the superposition position information; and display control means that causes the display means to display the predetermined area image in the entire image. The display control means causes the display means to display the predetermined area image and the superposition image with image quality changed according to a ratio between the predetermined area image and the superposition image in a display area of the display means.SELECTED DRAWING: Figure 37

Description

[Technical field]

[0001] The present invention relates to a communication terminal, an image communication system, a display method, and a program. [Background technology]

[0002] Conference systems that hold remote conferences between remote locations via a communication network such as the Internet are becoming widespread. In this conference system, in a conference room where one of the parties, such as attendees, is present, images of the conference room of the parties and audio such as remarks are captured and collected using a communication terminal of the remote conference system, which is then converted into digital data and transmitted to the communication terminal of the other party. This allows a video call to be conducted by displaying images on a display and outputting audio from a speaker in the other party's conference room, making it possible to hold a conference between remote locations in a state close to an actual conference (see Patent Document 1).

[0003] Also, a technique is known in which a communication terminal is connected to an imaging device capable of acquiring a celestial sphere panoramic image in real time, the celestial sphere panoramic image acquired from the imaging device is transmitted to each communication terminal of the other party, and each communication terminal of the other party displays a predetermined area image indicating a predetermined area that is a part of the celestial sphere panoramic image on a display or the like (see Patent Document 2). This allows users at each base to independently determine and display a predetermined area image indicating a predetermined area that interests them from the entire image of the celestial sphere panoramic image.

[0004] It has also been disclosed that the sending communication terminal transmits the original image and an image of the object of interest to a server, and the receiving communication terminal detects the coordinates of the object of interest in the current image, thereby superimposing a reference image (superimposed image) such as graphical information on the area in the original image that the sending user wants to draw attention to (see Patent Document 3). Summary of the Invention [Problem to be solved by the invention]

[0005] However, when data of a celestial sphere (panoramic image) and data of a superimposed image are transmitted and received between a plurality of communication terminals via a communication network such as the Internet, a problem arises in that the communication load increases and delays in image data occur. [Means for solving the problem]

[0006] The invention of claim 1 is a communication terminal capable of displaying a specified area image, which is an image of a specified area in an entire image, on a display means, and comprising a receiving means for receiving data of the entire image, data of a superimposed image, and superimposition position information indicating a position at which the superimposed image is to be superimposed on the entire image, transmitted from another communication terminal, an image processing means for superimposing the superimposed image on the entire image based on the superimposition position information, and a display control means for causing the display means to display the specified area image in the entire image, wherein the display control means causes the display means to display the specified area image and the superimposed image with image quality changed according to the proportion of the specified area image and the superimposed image within the display area of ​​the display means. Effect of the Invention

[0007] As described above, according to the present invention, even when data of a celestial sphere image and a superimposed image are transmitted and received, it is possible to suppress delays in image data. [Brief description of the drawings]

[0008] [Figure 1] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. [Diagram 2] FIG. 13 is an image of the imaging device in use. [Diagram 3] (a) is a hemispherical image (front) taken by the photographing device, (b) is a hemispherical image (back) taken by the photographing device, and (c) is an image represented by the Mercator projection. [Figure 4] (a) A conceptual diagram showing how a sphere is covered with a Mercator image, and (b) a diagram showing a spherical panoramic image. [Diagram 5] FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere panoramic image is a three-dimensional solid sphere. [Figure 6] FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 7] 13 is a diagram showing the relationship between predetermined region information and an image of a predetermined region T. FIG. [Figure 8] This is a diagram showing points in three-dimensional Euclidean space in spherical coordinates. [Figure 9] 1 is a schematic diagram of an image communication system according to an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating a hardware configuration of the imaging device. [Figure 11] FIG. 2 is a hardware configuration diagram of a video conference terminal. [Figure 12] FIG. 2 is a hardware configuration diagram of a communication management system and a PC. [Figure 13] FIG. 1 is a hardware configuration diagram of a smartphone. [Figure 14] FIG. 2 is a functional block diagram of a portion of the image communication system. [Figure 15] FIG. 2 is a functional block diagram of a portion of the image communication system. [Figure 16] FIG. 13 is a conceptual diagram illustrating an image type management table. [Figure 17] FIG. 13 is a conceptual diagram illustrating a photographing device management table. [Figure 18] FIG. 13 is a conceptual diagram showing a predetermined area management table. [Figure 19] FIG. 13 is a conceptual diagram illustrating a superimposition position management table. [Figure 20] FIG. [Figure 21] FIG. 13 is a conceptual diagram illustrating a session management table. [Figure 22] FIG. 13 is a conceptual diagram illustrating an image type management table. [Diagram 23] FIG. 13 is a conceptual diagram showing a predetermined area information management table. [Figure 24]FIG. 13 is a conceptual diagram illustrating a ratio management table. [Diagram 25] FIG. 11 is a sequence diagram showing a process of joining a specific communication session. [Figure 26] FIG. 13 is a diagram showing a selection screen for a communication session (virtual conference room). [Figure 27] FIG. 11 is a sequence diagram showing a management process of image type information. [Figure 28] FIG. 13 is an image diagram showing a state of a video call. [Figure 29] 11 is a sequence diagram showing a communication process of captured image data, audio data, and document image data in a video call. FIG. [Diagram 30] 30 shows display examples on the display at site B, where (a) shows a case where the image data transmitted from the photographing devices 1a and 1b is displayed as is without creating a celestial sphere panoramic image and a predetermined area image, (b) shows a case where the celestial sphere panoramic image and a predetermined area image are created from the image data transmitted from the photographing devices 1a and 1b, and (c) shows a case where the predetermined area image in (b) in FIG. 30 is changed. [Diagram 31] FIG. 11 is a sequence diagram showing a process of sharing predetermined area information. [Diagram 32] FIG. 11 is a sequence diagram showing another process for sharing predetermined area information. [Diagram 33] FIG. 11 is a sequence diagram showing a process of sharing superimposition position information. [Diagram 34] 13 is an example of a screen for changing the superimposition position. [Diagram 35] 11 is a flowchart showing a display process of a superimposed image. [Diagram 36] An example of the display of a superimposed image (reference image) at site B is shown. [Figure 37] FIG. 11 is a sequence diagram showing a process for changing image quality. [Figure 38] FIG. 13 is a diagram showing a case where a predetermined area image including a document image is changed. [Figure 39] FIG. 13 is a diagram showing a changed predetermined area. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to FIGS.

[0010] <<Outline of the embodiment>> <How to generate spherical panoramic images> A method for generating a spherical panoramic image will be described with reference to FIGS.

[0011] First, the appearance of the imaging device 1 will be described with reference to Fig. 1. The imaging device 1 is a digital camera for obtaining captured images that are the basis of a three-dimensional spherical (360°) panoramic image. Fig. 1(a) is a left side view of the imaging device, Fig. 1(b) is a front view of the imaging device, and Fig. 1(c) is a plan view of the imaging device.

[0012] As shown in FIG. 1(a), the photographing device 1 is sized so that a person can hold it in one hand. Also, as shown in FIG. 1(a), FIG. 1(b), and FIG. 1(c), an image sensor 103a is provided on the front side (front side) and an image sensor 103b is provided on the back side (rear side) at the top of the photographing device 1. These image sensors (image sensors) 103a and 103b are used in conjunction with optical members (for example, fisheye lenses 102a and 102b described later) capable of photographing hemispherical images (angle of view of 180° or more). Also, as shown in FIG. 1(b), an operation unit 115 such as a shutter button is provided on the surface opposite to the front side of the photographing device 1.

[0013] Next, the usage of the photographing device 1 will be described with reference to Fig. 2. Fig. 2 is an image diagram of the photographing device in use. As shown in Fig. 2, the photographing device 1 is used, for example, by a user holding it in his / her hand to photograph subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image sensor 103a and the image sensor 103b shown in Fig. 1, respectively.

[0014] Next, an outline of the process of creating a celestial sphere panoramic image from an image captured by the image capture device 1 will be described with reference to Fig. 3 and Fig. 4. Fig. 3(a) is a diagram showing a hemispherical image (front side) captured by the image capture device, Fig. 3(b) is a diagram showing a hemispherical image (rear side) captured by the image capture device, and Fig. 3(c) is a diagram showing an image expressed by the Mercator projection (hereinafter referred to as a "Mercator image"). Fig. 4(a) is a conceptual diagram showing a state in which a sphere is covered with a Mercator image, and Fig. 4(b) is a diagram showing a celestial sphere panoramic image.

[0015] As shown in Fig. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a fisheye lens 102a described below. Also, as shown in Fig. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a fisheye lens 102b described below. Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are superimposed by the photographing device 1 to create a Mercator image as shown in Fig. 3(c).

[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted to cover the sphere as shown in FIG. 4(a), and a celestial sphere panoramic image as shown in FIG. 4(b) is created. In this way, the celestial sphere panoramic image is represented as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere panoramic image may be a still image or a video.

[0017] As described above, since a celestial sphere panoramic image is an image pasted so as to cover a spherical surface, it gives a sense of incongruity to humans when viewed. Therefore, by displaying a predetermined region (hereinafter referred to as a "predetermined region image") of a celestial sphere panoramic image as a planar image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to humans. This will be described with reference to Figs. 5 and 6.

[0018] FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user who views the omnidirectional image CE displayed as a three-dimensional sphere. FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a predetermined area image when displayed on a display. In FIG. 6(a), the omnidirectional image CE shown in FIG. 4 is represented by a three-dimensional sphere CS. If the omnidirectional image CE generated in this way is a sphere CS, the virtual camera IC is located inside the omnidirectional image CE as shown in FIG. 5. The predetermined area T in the omnidirectional image CE is a shooting area of ​​the virtual camera IC, and is specified by predetermined area information indicating the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE.

[0019] Then, the predetermined area image Q shown in Fig. 6(a) is displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC, as shown in Fig. 6(b). The image shown in Fig. 6(b) is a predetermined area image represented by the predetermined area information initially set (default). Note that the predetermined area information may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the position coordinates of the virtual camera IC. In the following, the shooting direction (rH, rV) and the angle of view (α) of the virtual camera IC will be used for explanation.

[0020] The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 7. FIG. 7 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 7, rH indicates Horizontal Radian, rV indicates Vertical Radian, and α indicates the angle of view (Angle). That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the shooting direction (rH, rV) becomes the center point CP of the predetermined area T, which is the shooting area of ​​the virtual camera IC. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). In FIG. 7, the trigonometric function generally shown in the following (Equation 1) holds.

[0021] Lf = tan(α / 2) (Eq. 1) FIG. 8 is a diagram showing points in a three-dimensional Euclidean space in spherical coordinates. The position coordinates of the center point CP when expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius, polar angle, and azimuth angle, respectively. The radius r is the distance from the origin of the three-dimensional virtual space including the spherical panoramic image to the center point CP, and is therefore equal to f. FIG. 8 is a diagram showing these relationships. Hereinafter, the explanation will be given using the position coordinates (r, θ, φ) of the virtual camera IC.

[0022] <Outline of the image communication system> Next, an outline of the configuration of the image communication system of this embodiment will be described with reference to Fig. 9. Fig. 9 is a schematic diagram of the configuration of the image communication system of this embodiment.

[0023] 9, the image communication system of this embodiment is composed of photographing devices 1a and 1b, video conference terminals 3a and 3b, displays 4a and 4b, a communication management system 5, a PC (Personal Computer) 6, a PC 7, a photographing device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

[0024] Of these, the photographing devices 1a and 1b are special digital cameras for photographing subjects, scenery, etc. to obtain two hemispherical images that are the basis of a celestial sphere panoramic image, as described above. On the other hand, the photographing device 8 is a general digital camera for photographing subjects, scenery, etc. to obtain a general planar image.

[0025] The video conference terminals 3a and 3b are dedicated terminals for video conferences, and display images of video calls (video calls) on the displays 4a and 4b, respectively, via wired cables such as USB (Universal Serial Bus) cables. The video conference terminal 3a usually takes pictures with a camera 312 in FIG. 11 described later, but when the video conference terminal 3a is connected to a cradle 2a to which the photographing device 1a is attached via a wired cable, the photographing device 1a is given priority, and two hemispherical images that are the basis of a celestial sphere panoramic image can be obtained. Furthermore, the video conference terminal 3a is connected to a PC 6, and the screen of the PC 6 can be obtained. When a wired cable is used, the cradle 2a not only performs communication between the photographing device 1a and the video conference terminal 3a, but also plays a role of supplying power to the photographing device 1a and supporting the photographing device 1a. Here, the photographing device 1a, the cradle 2a, the video conference terminal 3a, the display 4a, and the PC 6 are located at the same base, that is, base A. Furthermore, at base A, four users A1, A2, A3, and A4 are participating in the video call. On the other hand, the video conference terminal 3d and the display 4d are located at the same location, site D. Furthermore, at site D, three users D1, D2, and D3 are participating in a video call.

[0026] The communication management system 5 manages and controls the communication of the video conference terminals 3a and 3b, the PC 7, and the smartphone 9, and manages the types of image data (classification of general images and special images) transmitted and received. Therefore, the communication management system is also a communication control system. Here, the special images are spherical panoramic images. The communication management system 5 is installed in a service company that provides video communication services. The communication management system 5 may be constructed by a single computer, or may be constructed by multiple computers in which each section (function, means, or storage section) is divided and arbitrarily assigned.

[0027] The PC 6 generates data of a material image (material image data) to be displayed in a video conference. The material image may be, but is not limited to, an image displayed, created, or edited by a general-purpose application running on the PC 6, or an image captured by a general digital camera or the like that is reproduced on the PC 6.

[0028] The PC 7 is capable of video calling by having the photographing device 8 attached thereto. In this example, the PC 7 and the photographing device 8 are located at the same location, site C. Also, at site C, one user C is participating in the video call.

[0029] The smartphone 9 displays an image of the video call on a display 917 provided in the smartphone 9, which will be described later. The smartphone 9 usually captures images using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 (described later) provided in the smartphone 9, but can obtain two hemispherical image data that are the basis of the omnidirectional panoramic image obtained by the image capture device 1b by using wireless communication technology such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark). When using wireless communication technology, the cradle 2b only serves to supply power to the image capture device 1b and support the image capture device 1b. Here, the image capture device 1b, the cradle 2b, and the smartphone 9 are located at the same base, base B. At base B, two users B1 and B2 are participating in the video call.

[0030] The video conference terminals 3a and 3d, the PC 7, and the smartphone 9 are examples of communication terminals. OpenGL ES is installed in each communication terminal, and the communication terminals can create predetermined area information indicating a partial area of ​​a celestial sphere panoramic image and create a predetermined area image from a celestial sphere panoramic image transmitted from another communication terminal.

[0031] The arrangement of the terminals (communication terminal, display, photographing device), devices, and users shown in FIG. 9 is one example, and other examples may be used. For example, at site C, a photographing device capable of photographing a celestial sphere panoramic image may be used instead of the photographing device 8. The communication terminal also includes a digital television, a smart watch, a car navigation device, and the like. Hereinafter, any photographing device among the photographing devices 1a and 1b will be referred to as "photographing device 1". Any video conference terminal among the video conference terminals 3a and 3b will be referred to as "video conference terminal 3". Any display among the displays 4a and 4b will be referred to as "display 4".

[0032] <<Hardware configuration of the embodiment>> Next, the hardware configurations of the image capturing device 1, the video conference terminal 3, the communication management system 5, the PC 6, the PC 7, and the smartphone 9 of this embodiment will be described in detail with reference to FIGS. It should be noted that the photographing device 8 is a general camera, and therefore a detailed description thereof will be omitted.

[0033] <Hardware configuration of the imaging device 1> First, the hardware configuration of the photographing device 1 will be described with reference to Fig. 10. Fig. 10 is a hardware configuration diagram of the photographing device 1. In the following, the photographing device 1 is assumed to be an omnidirectional (all-directional) photographing device using two imaging elements, but the number of imaging elements may be two or more. In addition, the photographing device 1 does not necessarily have to be a device dedicated to omnidirectional photographing, and an omnidirectional photographing unit may be attached to a normal digital camera, smartphone, or the like to have substantially the same functions as the photographing device 1.

[0034] As shown in FIG. 10, the photographing device 1 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I / F 116, a communication unit 117, and an antenna 117a.

[0035] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to the wide-angle lenses. The imaging elements 103a and 103b include an image sensor such as a CMOS sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the fisheye lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the imaging elements are set.

[0036] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are connected to a serial I / F bus (such as an I2C bus) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to a CPU 111 via a bus 110. Furthermore, ROM 112, SRAM 113, DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like are also connected to the bus 110.

[0037] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs a predetermined processing on each image data, and then superimposes these image data to create Mercator image data such as that shown in Figure 3(c).

[0038] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0039] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 1 have a preview display function or a function corresponding to video display on a display (for example, the display of the video conference terminal 3a). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / minute).

[0040] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. Note that, in this embodiment, the photographing device 1 is not provided with a display unit, but may be provided with a display unit.

[0041] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I / F bus, and performs a predetermined process on the sound data.

[0042] The CPU 111 controls the overall operation of the photographing device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and processed Mercator image data.

[0043] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. A user operates the operation buttons to input various shooting modes, shooting conditions, and the like.

[0044] The network I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. The Mercator image data stored in the DRAM 114 is recorded in an external medium via the network I / F 116, or transmitted to an external device such as the video conference terminal 3a via the network I / F 116 as necessary.

[0045] The communication unit 117 communicates with an external device such as the video conference terminal 3a by short-range wireless technology such as WiFi or NFC (Near Field Communication) via an antenna 117a provided in the image capturing device 1. The communication unit 117 can also transmit Mercator image data to the external device of the video conference terminal 3a.

[0046] The electronic compass 118 calculates the direction and tilt (Roll rotation angle) of the image capturing device 1 from the earth's magnetism, and outputs the direction and tilt information. This direction and tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data such as the date and time the image was captured, and the data size of the image data.

[0047] <Hardware configuration of video conferencing terminal> Next, the hardware configuration of the video conference terminal 3 will be described with reference to Fig. 11. Fig. 11 is a diagram showing the hardware configuration of the video conference terminal. As shown in Fig. 11, the video conference terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, an operation button 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an image sensor I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319.

[0048] Of these, the CPU 301 controls the overall operation of the video conference terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as an IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and sound data. The SSD (Solid State Drive) 305 controls reading and writing of various data from the flash memory 304 under the control of the CPU 301. Note that a HDD may be used instead of the SSD. The media I / F 307 controls reading and writing (storing) of data from a recording medium 306, such as a flash memory. The operation button 308 is a button that is operated when selecting a destination of the video conference terminal 3, etc. The power switch 309 is a switch for switching the power of the video conference terminal 3 ON / OFF.

[0049] The network I / F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means for capturing an image of a subject under the control of the CPU 301 to obtain image data. The image sensor I / F 313 is a circuit for controlling the driving of the camera 312. The microphone 314 is a type of built-in sound collection means for inputting sound. The sound input / output I / F 316 is a circuit for processing input / output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I / F 317 is a circuit for transmitting image data to the external display 4 under the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) or Bluetooth (registered trademark).

[0050] 11. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 301 shown in FIG.

[0051] The display 4 is a type of display means configured with liquid crystal or organic electroluminescence (EL) for displaying an image of a subject, operation icons, etc. The display 4 is also connected to the display I / F 317 by a cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0052] The camera 312 includes a lens and a solid-state imaging element that converts light into electric charges to digitize an image (video) of a subject, and a CMOS sensor, CCD sensor, or the like is used as the solid-state imaging element. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 via a USB (Universal Serial Bus) cable, or the like. When an external camera is connected, the external camera is driven in preference to the built-in camera 312 under the control of the CPU 301. Similarly, when an external microphone or an external speaker is connected, the external microphone and the external speaker are driven in preference to the built-in microphone 314 and the built-in speaker 315, respectively, under the control of the CPU 301.

[0053] Moreover, the recording medium 306 is configured to be detachable from the video conference terminal 3. Moreover, as long as it is a non-volatile memory that reads or writes data under the control of the CPU 301, it is not limited to the flash memory 304, and an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

[0054] <Communication management system, PC hardware configuration> Next, the hardware configuration of the communication management system 5, PC 6, and PC 7 will be described with reference to Fig. 12. Fig. 12 is a hardware configuration diagram of the communication management system and PC. Note that since the communication management system 5, PC 6, and PC 7 are all computers with the same configuration, the following describes the configuration of the communication management system 5, and omits the description of the configurations of PC 6 and PC 7.

[0055] The communication management system 5 includes a CPU 501 for controlling the overall operation of the communication management system 5, a ROM 502 storing a program used to drive the CPU 501, such as an IPL, a RAM 503 used as a work area for the CPU 501, a HD 504 for storing various data such as programs for the communication management system 5, a HDD (Hard Disk Drive) 505 for controlling the reading or writing of various data from the HD 504 under the control of the CPU 501, a media drive 507 for controlling the reading or writing (storage) of data from a recording medium 506, such as a flash memory, a display 508 for displaying various information such as a cursor, a menu, a window, characters, or an image, a network I / F 509 for data communication using the communication network 100, a keyboard 511 having multiple keys for inputting characters, numbers, various instructions, etc., a mouse 512 for selecting and executing various instructions, selecting a processing target, moving the cursor, etc., a CD-RW (Compact Disc 12. The optical disc drive 510 includes a CD-RW drive 514 that controls reading of various data from a CD-ROM drive 513 (CD-ReWritable), and bus lines 510 such as an address bus and a data bus for electrically connecting the above-mentioned components as shown in FIG.

[0056] <Hardware configuration of smartphone> Next, the hardware of the smartphone will be described with reference to Fig. 13. Fig. 13 is a hardware configuration diagram of the smartphone. As shown in Fig. 13, the smartphone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration / direction sensor 906, a media I / F 908, and a GPS receiver 909.

[0057] Among these, the CPU 901 controls the overall operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901, such as an IPL. The RAM 903 is used as a work area for the CPU 901. The EEPROM 904 reads or writes various data, such as smartphone programs, under the control of the CPU 901. The CMOS sensor 905 captures an image of a subject (mainly a self-portrait) under the control of the CPU 901 to obtain image data. The acceleration / direction sensor 906 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I / F 908 controls the reading or writing (storage) of data from or to a recording medium 907, such as a flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0058] The smartphone 9 also has a long-distance communication circuit 911, a camera 912, an image sensor I / F 913, a microphone 914, a speaker 915, an audio input / output I / F 916, a display 917, an external device connection I / F 918, a short-distance communication circuit 919, an antenna 919a of the short-distance communication circuit 919, and a touch panel 921.

[0059] Among these, the long-distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 901. The image sensor I / F 913 is a circuit that controls the driving of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 916 is a circuit that processes input and output of sound signals between the microphone 914 and the speaker 915 under the control of the CPU 901. The display 917 is a type of display means such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 918 is an interface for connecting various external devices. The short-distance communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means that a user operates the smartphone 9 by pressing the display 917.

[0060] The smartphone 9 also includes a bus line 910. The bus line 910 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 901.

[0061] Incidentally, the recording media such as CD-ROMs on which the above-mentioned programs are stored, and the HD on which these programs are stored, can be provided domestically or internationally as program products.

[0062] <<Functional configuration of the embodiment>> Next, the functional configuration of this embodiment will be described with reference to Figures 14 to 21. Figures 14 and 15 are functional block diagrams of a portion of an image communication system.

[0063] <Functional configuration of the imaging device 1a> 14, the photographing device 1a has a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a memory / readout unit 19a. Each of these units is a function or means realized by operating any of the components shown in FIG. 10 in response to an instruction from a CPU 111 in accordance with a program for the photographing device loaded from the SRAM 113 onto the DRAM 114.

[0064] The photographing device 1 also has a storage unit 1000a that is configured by the ROM 112, the SRAM 113, and the DRAM 114 shown in Fig. 10. The storage unit 1000a stores a GUID (Globally Unique Identifier) ​​of the photographing device 1 itself.

[0065] It should be noted that photographing device 1b has a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, a memory / readout unit 19b, and a memory unit 1000b; however, as these respectively realize the same functions as reception unit 12a, imaging unit 13a, sound collection unit 14a, communication unit 18a, memory / readout unit 19a, and memory unit 1000a in photographing device 1a, their descriptions are omitted.

[0066] (Functional configuration of the imaging device 1a) Next, each functional configuration of the image capturing device 1a will be described in more detail with reference to FIG. 10 and FIG.

[0067] The reception unit 12a of the photographing device 1 is realized mainly by the processing of the operation unit 115 and the CPU 111 shown in FIG. 10, and receives operation input from the user.

[0068] The imaging section 13a is mainly realized by the imaging unit 101, image processing unit 104, and imaging control unit 105 shown in FIG. 10, and processing by the CPU 111, and captures images of scenery or the like and obtains data of the captured image (captured image data).

[0069] The sound collection section 14a is realized by the components 108 and sound processing unit 109 shown in FIG. 10, and processing by the CPU 111, and collects sounds around the image capture device 1a.

[0070] The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38a of the video conference terminal 3a by short-range wireless communication technology such as the NFC standard, BlueTooth, or WiFi.

[0071] The storage / reading unit 19a is realized mainly by the processing of the CPU 111 shown in FIG. 10, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.

[0072] <Functional configuration of the video conference terminal 3a> 14, the video conference terminal 3a includes a transmission / reception unit 31a, a reception unit 32a, an image / sound processing unit 33a, a display control unit 34a, a determination unit 35a, a creation unit 36a, a calculation unit 37a, a communication unit 38a, and a storage / readout unit 39a. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating in response to an instruction from the CPU 301 in accordance with the program for the video conference terminal 3a loaded onto the RAM 303 from the flash memory 304.

[0073] The video conference terminal 3a also has a storage unit 3000a constructed by the ROM 302, RAM 303, and flash memory 304 shown in Fig. 11. In the storage unit 3000a, an image type management DB 3001a, a photographing device management DB 3002a, a predetermined area management DB 3003a, and a superimposition position management DB 3004a are constructed. Of these, the image type management DB 3001a is constructed by the image type management table shown in Fig. 16. The photographing device management DB 3002a is constructed by the photographing device management table shown in Fig. 17. The predetermined area management DB 3003a is constructed by the predetermined area management table shown in Fig. 18. The superimposition position management DB 3004a is constructed by the superimposition position management table shown in Fig. 19.

[0074] The video conference terminal 3d has a transmission / reception unit 31d, a reception unit 32d, an image / sound processing unit 33d, a display control unit 34d, a judgment unit 35d, a creation unit 36d, a calculation unit 37d, a communication unit 38d, a memory / readout unit 39d, and a memory unit 3000d; however, because these respectively realize the same functions as the transmission / reception unit 31a, the reception unit 32a, the image / sound processing unit 33a, the display control unit 34a, the judgment unit 35a, the creation unit 36a, the calculation unit 37a, the communication unit 38a, the memory / readout unit 39a, and the memory unit 3000a in the video conference terminal 3a, description of these will be omitted. In addition, an image type management DB 3001d, an imaging device management DB 3002d, a specified area management DB 3003d, and a superimposition position management DB 3004d are configured in the memory unit 3000d of the video conference terminal 3d. However, since these have the same data structures as the image type management DB 3001a, the imaging device management DB 3002a, the specified area management DB 3003a, and the superimposition position management DB 3004a of the video conference terminal 3a, their descriptions are omitted.

[0075] (Image type management table) FIG. 16 is a conceptual diagram showing an image type management table. In this image type management table, image data ID, an IP address which is an example of the destination of the transmission source terminal, and a source name are stored and managed in association with each other. Among these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is added to image data transmitted from the same transmission source terminal. This allows the transmission destination terminal (receiving communication terminal) to identify the transmission source terminal of the received image data. The IP address of the transmission source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for identifying the photographing device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal such as the video conference terminal 3a according to a predetermined naming rule.

[0076] For example, it is shown that four communication terminals with IP addresses "1.2.1.3", "1.2.2.3", "1.3.1.3", and "1.3.2.3" are transmitting image data indicated by image data IDs "RS001", "RS002", "RS003", and "RS004", respectively. Furthermore, the types of images indicated by the source names of the communication terminals are "Video_Theta", "Video_Theta", "Video", and "Video", which indicate that the image types are "special image", "special image", "general image", and "general image", respectively. Note that the special image here is a spherical panoramic image.

[0077] Note that data other than image data may also be managed in association with an image data ID. Data other than image data is, for example, sound data. (Radiation device management table) 17 is a conceptual diagram showing an image capturing device management table. In this image capturing device management table, the vendor ID and product ID of the GUID of an image capturing device capable of obtaining two hemispherical images that are the basis of a celestial sphere panoramic image are stored and managed. As the GUID, for example, a vendor ID (VID) and product ID (PID) used in a USB device can be used. The vendor ID and product ID are stored in a communication terminal such as a video conference terminal when the terminal is shipped from the factory, but may be additionally stored after the shipment from the factory.

[0078] (Prescribed area management table) FIG. 18 is a conceptual diagram showing a predetermined area management table. In this predetermined area management table, the IP address of the communication terminal that is the transmission source of the photographed image data, the IP address of the communication terminal that is the transmission destination of the photographed image data, and predetermined area information indicating the predetermined area image being displayed on the communication terminal that is the transmission destination of the photographed image data are associated and stored and managed. The communication terminal that is the transmission destination of the photographed image data is also the communication terminal that is the transmission source of the predetermined area information. As shown in FIG. 6 and FIG. 7, the predetermined area information is a conversion parameter for converting the photographed image into an image of the predetermined area T in the photographed image (predetermined area image). The IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for identifying the communication terminal, and the like. Here, the IP address is expressed by simplifying the IPv4 address. The IP address may be IPv6.

[0079] For example, in the first to third rows of the predetermined area management table in FIG. 18, when the IP address of the video conference terminal 3a is "1.2.1.3", it is managed that the captured image data transmitted from the video conference terminal 3a is transmitted to the video conference terminal 3d having the IP address "1.2.2.3", the PC 7 having the IP address "1.3.1.3", and the smartphone 9 having the IP address "1.3.2.3" via the communication management system 5. Furthermore, it is managed that the video conference terminal 3d is the communication terminal that transmitted the predetermined area information (r=10, θ=20, φ=30). Similarly, it is managed that the PC 7 is the communication terminal that transmitted the predetermined area information (r=10, θ=30, φ=40). Moreover, it is managed that the smartphone 9 is the communication terminal that transmitted the predetermined area information (r=30, θ=40, φ=50).

[0080] In addition, when the transmission / reception unit 31a receives new specified area information that includes the same set of IP addresses as the IP address of the communication terminal that sent the captured image data that is already managed and the IP address of the communication terminal that is the destination of the captured image data, the storage / reading unit 39a overwrites the specified area information already managed with the newly received specified area information.

[0081] (Superimposition position management table) 19 is a conceptual diagram showing a superimposition position management table. In this superimposition position management table, the IP address of the communication terminal that is the transmission source of the captured image data and the superimposition position information indicating where the document image, which is the superimposition source image, is to be superimposed on the spherical panoramic image, which is the superimposition destination image, are stored and managed in association with each other.

[0082] The superimposition coordinates as an example of the superimposition position information will be described with reference to FIG. 20. FIG. 20 shows a Mercator image of a celestial sphere image on which a document image, which is an original image, is superimposed. The superimposition coordinates are composed of start point coordinates and end point coordinates. When the top left vertex position of the document image, which is the original image, is the start point, the start point coordinates can be expressed as (0,0) and the end point coordinates can be expressed as (xn,yn). In this case, when the start point coordinates (0,0) and the end point coordinates (xn,yn) are superimposed at positions corresponding to the coordinates (x1,y1) and the coordinates (x2,y2) in the Mercator image, respectively, the start point coordinates are associated with (x1,x2) and the end point coordinates are associated with (x2,y2) in the superimposition position management table. Note that the IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for identifying a communication terminal, and the like. Note that the IP address is expressed by simplifying the IPv4 address. The IP address may be IPv6.

[0083] (Functional configuration of the video conference terminal 3a) Next, the functional components of the video conference terminal 3a will be described in more detail with reference to FIG. 11 and FIG.

[0084] The transmission / reception unit 31a of the video conference terminal 3a is mainly realized by the processing of the network I / F 311 and CPU 301 shown in FIG. 11, and transmits and receives various data (or information) to and from the communication management system 5 via the communication network 100.

[0085] The reception unit 32a is realized mainly by the operation buttons 308 and processing by the CPU 301, and receives various selections or inputs from the user. In addition to the operation buttons 308, a touch panel or the like may be used as another input means.

[0086] 11, and performs image processing on image data obtained by capturing an image of a subject with camera 312. After the user's voice is converted into an audio signal by microphone 314, image / sound processing unit 33a performs audio processing on audio data related to this audio signal.

[0087] Furthermore, the image / sound processing unit 33a performs image processing on image data received from another communication terminal based on image type information such as a source name so that the display control unit 34 displays an image on the display 4. Specifically, when the image type information indicates that the image is a special image, the image / sound processing unit 33a creates a celestial sphere panorama image by converting the image data (for example, data of each hemispherical image as shown in Fig. 3(a) and (b)) into a celestial sphere panorama image as shown in Fig. 4(b). Furthermore, when there is a document image to be superimposed on the celestial sphere panorama image, a superimposed image is generated in which the document image is superimposed on the celestial sphere panorama image based on the document image data and the superimposition position information. Furthermore, the image / sound processing unit 33a creates a predetermined area image as shown in Fig. 6(b). Furthermore, the image / sound processing unit 33a outputs an audio signal related to the audio data received from another communication terminal via the communication management system 5 to the speaker 315, and causes the speaker 315 to output audio.

[0088] The display control unit 34a is realized mainly by the processing of the display I / F 317 and the CPU 301, and performs control for displaying various images, characters, and the like on the display 4.

[0089] The determination unit 35a is realized mainly by the processing of the CPU 301, and for example, determines the image type related to the image data received from the photographing device 1a, but is not limited to this and also performs various other determinations related to the image data.

[0090] The creation unit 36a is mainly realized by the processing of the CPU 301, and creates a source name, which is an example of image type information, in accordance with the naming rules described above based on the result of the determination by the determination unit 35a of whether the image is a general image or a special image (here, a spherical panoramic image). For example, if the determination unit 35a determines that the image is a general image, the creation unit 36a creates a source name "Video" indicating that it is a general image. On the other hand, if the determination unit 35a determines that the image is a special image, the creation unit 36a creates a source name "Video_Theta" indicating that it is a special image.

[0091] The calculation unit 37a is realized mainly by the processing of the CPU 301, and calculates the superimposition position information.

[0092] The communication unit 38a is mainly realized by the processing of the short-range communication circuit 319, the antenna 318a, and the CPU 301, and can communicate with the communication unit 18a of the photographing device 1a by short-range wireless technology such as NFC, BlueTooth, WiFi, etc. Although the communication unit 38a and the transmission / reception unit 31a have been described as having separate communication units, they may be configured to share a common communication unit.

[0093] The storage / reading unit 39a is mainly realized by the processing of the CPU 301 shown in FIG. 11, and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000.

[0094] <Functional configuration of the communication management system> Next, each functional component of the communication management system 5 will be described in detail with reference to Fig. 12 and Fig. 15. The communication management system 5 has a transmission / reception unit 51, a determination unit 55, a generation unit 56, and a storage / readout unit 59. Each of these units is a function or means realized by any of the components shown in Fig. 12 operating in response to an instruction from the CPU 501 in accordance with the program for the communication management system 5 loaded from the HD 504 onto the RAM 503.

[0095] The communication management system 5 also has a storage unit 5000 constructed by the RAM 503 and HD 504 shown in Fig. 12. A session management DB 5001, an image type management DB 5002, a predetermined area management DB 5003, and a ratio management DB 5004 are constructed in this storage unit 5000. Of these, the session management DB 5001 is constructed by the session management table shown in Fig. 21. The image type management DB 5002 is constructed by the image type management table shown in Fig. 22. The predetermined area management DB 5003 is constructed by the predetermined area information management table shown in Fig. 23. The ratio management DB 5004 is constructed by the ratio management table shown in Fig. 24.

[0096] (Session management table) FIG. 21 is a conceptual diagram showing a session management table. In this session management table, the session ID and the IP addresses of the participating communication terminals are stored and managed in association with each other. Among them, the session ID is an example of session identification information for identifying a communication session that realizes a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal such as the video conference terminal 3a, and is used when selecting a communication session in each communication terminal. The IP addresses of the participating communication terminals indicate the IP addresses of the communication terminals that have participated in the virtual conference room identified by the associated session ID.

[0097] (Image type management table) FIG. 22 is a conceptual diagram showing an image type management table. In the image type management table shown in FIG. 22, in addition to each piece of information managed in the image type management table shown in FIG. 16, the same session ID as the session ID managed in the session management table is associated and managed. Here, it is shown that three communication terminals with IP addresses of "1.2.1.3", "1.2.2.3", and "1.3.1.3" are participating in the virtual conference room indicated by the same session ID "se101". In addition, the communication management system 5 manages the same image data ID, the IP address of the sender terminal, and the image type information managed by the communication terminal such as the video conference terminal 3a in order to transmit image type information, etc. to the communication terminal already in a video call and the newly participating communication terminal when a new communication terminal enters the virtual conference room. This eliminates the need to transmit and receive image type information, etc. between the communication terminal already in a video call and the newly participating communication terminal.

[0098] (Prescribed area information management table) Fig. 23 is a conceptual diagram showing a predetermined area information management table. This predetermined area information management table basically has the same data structure as the predetermined area information management table shown in Fig. 18. However, as described later, the transmitting / receiving unit 51 transmits the latest predetermined area information to each communication terminal at regular intervals (e.g., 30 seconds), so that all the predetermined area information received by the transmitting / receiving unit 51 is stored without being deleted until the predetermined area information is transmitted at regular intervals. In Fig. 21, the newer the predetermined area information, the higher the management position.

[0099] (Ratio management table) FIG. 24 is a conceptual diagram showing a ratio management table. In this ratio management table, the IP address of the communication terminal that is the transmission source of the photographed image data, the IP address of the communication terminal that is the transmission destination of the photographed image data, and ratio information are associated, stored, and managed. The ratio information indicates the ratio of the display areas of the predetermined area image in the omnidirectional panorama image data and the material image (an example of a superimposed image) related to the material image data that are sent from the communication terminal of the transmission source to the display means such as a display in the communication terminal of the transmission destination (reception side) of the photographed image data when displaying them.

[0100] (Functional configurations of the communication management system) Next, with reference to FIGS. 12 and 15, the functional configurations of the communication management system 5 will be described in detail.

[0101] The transmission / reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 12, and transmits and receives various data (or information) to and from the video conference terminals 3a, 3d, or the PC 7 via the communication network 100.

[0102] The determination unit 55 is mainly realized by the processing of the CPU 501 and makes various determinations.

[0103] The generation unit 56 is mainly realized by the processing of the CPU 501 and generates an image data ID.

[0104] The storage / reading unit 59 is mainly realized by the processing of the HDD 505 shown in FIG. 12 and the CPU 501, and stores various data (or information) in the storage unit 5000 or reads out various data (or information) from the storage unit 5000.

[0105] <Functional configuration of the PC>[ Next, with reference to FIGS. 12 and 14, the functional configurations of the PC 6 will be described in detail.

[0106] 14, the PC 6 has a reception unit 62, a display control unit 64, a communication unit 68, and a storage / readout unit 69. Each of these units is a function or means realized by any of the components shown in FIG. 12 operating in response to an instruction from the CPU 501 in accordance with the program for the PC 6 loaded onto the RAM 503 from the HD 504.

[0107] The PC 6 also has a storage unit 6000 constructed by the ROM 502, the RAM 503, and the HD 504 shown in FIG.

[0108] (PC functional configuration) The reception unit 62 of the PC 6 is realized mainly by the processing of the keyboard 511, the mouse 512, and the CPU 501, and realizes the same function as the reception unit 32a. The display control unit 64 is realized mainly by the processing of the CPU 501, and performs control for displaying various images, characters, etc. on the display 508. The communication unit 68 is realized mainly by the processing of the CPU 501, and can communicate with the communication unit 38a of the video conference terminal 3a by short-range wireless communication technology such as the NFC standard, BlueTooth, WiFi, etc. The storage / reading unit 69a is realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 6000 and reads various data (or information) from the storage unit 6000.

[0109] <Functional configuration of a PC as a communication terminal> Next, the functional configuration of the PC 7 will be described in detail with reference to Fig. 12 and Fig. 15. The PC 7 basically has the same functions as the video conference terminal 3a. That is, as shown in Fig. 14, the PC 7 has a transmitting / receiving unit 71, a receiving unit 72, an image / sound processing unit 73, a display control unit 74, a determining unit 75, a creating unit 76, a calculating unit 77, a communicating unit 78, and a storing / reading unit 79. Each of these units is a function or means realized by any of the components shown in Fig. 12 operating in response to an instruction from the CPU 501 in accordance with the program for the PC 7 loaded from the HD 504 onto the RAM 503.

[0110] 12. An image type management DB 7001, an imaging device management DB 7002, a predetermined area management DB 7003, and a superimposition position management DB 7004 are constructed in this storage unit 7000. Note that the image type management DB 7001, the imaging device management DB 7002, the predetermined area management DB 7003, and the superimposition position management DB 7004 have the same data structures as the image type management DB 3001a, the imaging device management DB 3002a, the predetermined area management DB 3003a, and the superimposition position management DB 3004a, respectively, and therefore descriptions thereof will be omitted.

[0111] (Functional configuration of a PC as a communication terminal) The transmission / reception unit 71 of the PC 7 is realized mainly by the processing of the network I / F 509 and the CPU 501 shown in FIG. 12, and realizes the same function as the transmission / reception unit 31a.

[0112] The reception unit 72 is mainly realized by the processing of the keyboard 511, the mouse 512 and the CPU 501, and realizes the same function as the reception unit 32a. The image / sound processing unit 73 is mainly realized by instructions from the CPU 501, and realizes the same function as the image / sound processing unit 33a. The display control unit 74 is mainly realized by the processing of the CPU 501, and realizes the same function as the display control unit 34a. The determination unit 75 is mainly realized by the processing of the CPU 501, and realizes the same function as the determination unit 35a. The creation unit 76 is mainly realized by the processing of the CPU 501, and realizes the same function as the creation unit 36a. The calculation unit 77 is mainly realized by the processing of the CPU 501, and realizes the same function as the calculation unit 37a. The communication unit 78 is mainly realized by the processing of the CPU 501, and realizes the same function as the communication unit 38a. The storage / reading unit 79 a is realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 7000 and reads out various data (or information) from the storage unit 7000 .

[0113] <Smartphone functional configuration> Next, the functional configuration of the smartphone 9 will be described in detail with reference to Figs. 13 and 14. The smartphone 9 basically has the same functions as the video conference terminal 3a. That is, as shown in Fig. 14, the smartphone 9 has a transmission / reception unit 91, a reception unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a calculation unit 97, a communication unit 98, and a storage / readout unit 99. Each of these units is a function or means realized by any of the components shown in Fig. 13 operating in response to an instruction from the CPU 901 in accordance with the program for the smartphone 9 expanded from the EEPROM 904 onto the RAM 903.

[0114] 13. An image type management DB 9001, an imaging device management DB 9002, a predetermined area management DB 9003, and a superimposition position management DB 9004 are configured in the memory unit 9000. Note that the image type management DB 9001, the imaging device management DB 9002, the predetermined area management DB 9003, and the superimposition position management DB 9004 have the same data configurations as the image type management DB 3001a, the imaging device management DB 3002a, the predetermined area management DB 3003a, and the superimposition position management DB 3004a, respectively, and therefore descriptions thereof will be omitted.

[0115] (Smartphone functional configuration) The transmitter / receiver 91 of the smartphone 9 is mainly realized by the processing of the long-distance communication circuit 911 and the CPU 901 shown in FIG. 13, and realizes the same functions as the transmitter / receiver 31a.

[0116] The reception unit 92 is realized mainly by the processing of the touch panel 921 and the CPU 901, and realizes the same functions as the reception unit 32a.

[0117] The image / sound processing unit 93 is mainly realized by instructions from the CPU 901, and realizes the same function as the image / sound processing unit 33a. The display control unit 94 is mainly realized by the processing of the CPU 901, and realizes the same function as the display control unit 34a. The determination unit 95 is mainly realized by the processing of the CPU 901, and realizes the same function as the determination unit 35a. The creation unit 96 is mainly realized by the processing of the CPU 901, and realizes the same function as the creation unit 36a. The calculation unit 97 is mainly realized by the processing of the CPU 901, and realizes the same function as the calculation unit 37a. The communication unit 98 is mainly realized by the processing of the CPU 901, and realizes the same function as the communication unit 38a. The storage / readout unit 99 is realized by the processing of the CPU 901, and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000.

[0118] <<Processing or Operation of the Embodiment>> Next, the processing or operation of this embodiment will be described with reference to FIGS.

[0119] <Participation process> First, a process of joining a specific communication session will be described with reference to Fig. 25 and Fig. 26. Fig. 25 is a sequence diagram showing the process of joining a specific communication session. Fig. 26 is a diagram showing a selection screen for a communication session (virtual conference room).

[0120] First, when a user at site A (e.g., user A1) operates the video conference terminal 3a to display a selection screen for a communication session (virtual conference room), the reception unit 32a receives the operation to display the selection screen, and the display control unit 34a displays the selection screen as shown in Fig. 26 on the display 4a (step S21). This selection screen displays selection buttons b1, b2, b3, etc., which indicate the virtual conference rooms R1, R2, R3, etc., to be selected. Each selection button b1, etc. is associated with a session ID.

[0121] When user A1 selects a desired selection button (selection button b1 in this case) of the virtual conference room, the reception unit 32a accepts the selection of the communication session (step S22). Then, the transmission / reception unit 31a transmits a participation request to the virtual conference room to the communication management system 5 (step S23). This participation request includes a session ID indicating the communication session whose selection was accepted in step S22, and the IP address of the video conference terminal 3a, which is the request source terminal. As a result, the transmission / reception unit 51 of the communication management system 5 receives the participation request.

[0122] Next, the storage / read unit 59 performs a process of participating in the communication session by adding the IP address received in step S23 to the participating terminal IP address field in the record of the session management DB 5001 (see FIG. 21) having the same session ID as the session ID received in step S23 (step S24). Then, the transmission / reception unit 51 transmits a participation request response to the video conference terminal 3a (step S25). This participation request response includes the session ID received in step S23 and the result of the participation process. As a result, the transmission / reception unit 31a of the video conference terminal 3a receives the participation request response. Hereinafter, a case where the participation process is successful will be described.

[0123] <Image type information management process> Next, the management process of image type information will be described with reference to Fig. 27. Fig. 27 is a sequence diagram showing the management process of image type information.

[0124] First, when a user at site A (e.g., user A1) connects the USB cable of the cradle 2a with the image capturing device 1a attached to the video conference terminal 3a, the memory / read unit 19a of the image capturing device 1a reads the GUID of its own device (image capturing device 1a) stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of its own device to the communication unit 38a of the video conference terminal 3a (step S51). As a result, the communication unit 38a of the video conference terminal 3a receives the GUID of the image capturing device 1a.

[0125] Next, the determination unit 35a of the video conference terminal 3a determines whether the same vendor ID and product ID as the vendor ID and product ID in the GUID received in step S51 are managed in the photographing device management DB 3002a (see FIG. 17), thereby determining the image type (step S52). Specifically, if the same vendor ID and product ID are managed in the photographing device management DB 3002a, the determination unit 35a determines that the photographing device 1a is a photographing device that captures special images (here, a celestial sphere panoramic image). On the other hand, if the same vendor ID and product ID are not managed in the photographing device management DB 3002a, the determination unit 35a determines that the photographing device 1a is a photographing device that captures general images.

[0126] Next, the storage / reading unit 39a stores the IP address of the terminal (video conference terminal 3a) which is the source terminal, in association with the image type information which is the result of the determination made in step S52 in the image type management DB 3001a (see FIG. 16) (step S53). In this state, the image data ID is not associated. The image type information is, for example, a source name determined according to a predetermined naming rule and an image type (general image, special image).

[0127] Next, the transmitting / receiving unit 31a transmits a request to add image type information to the communication management system 5 (step S54). This request to add image type information includes the IP address of the own terminal, which is the transmission source terminal stored in step S53, and the image type information. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request to add image type information.

[0128] Next, the storage / read unit 59 of the communication management system 5 searches the session management DB 5001 (see FIG. 21) using the IP address of the source terminal received in step S54 as a search key, and reads out the corresponding session ID (step S55).

[0129] Next, the generating unit 56 generates a unique image data ID (step S56). Then, the storing / reading unit 59 stores the session ID read out in step S55, the image data ID generated in step S56, and the IP address of the transmission source terminal and the image type information received in step S54 in association with each other as a new record in the image type management DB 5002 (see FIG. 22) (step S57). Then, the transmitting / receiving unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3a. As a result, the transmitting / receiving unit 31a of the video conference terminal 3a receives the image data ID (step S58).

[0130] Next, the storage / reading unit 39a of the video conference terminal 3a stores the image data ID received in step S58 in the image type management DB 3001a (see FIG. 16) in association with the IP address and image type information of the terminal itself (video conference terminal 3a), which is the sending terminal, stored in step S53 above (step S59).

[0131] Meanwhile, the transmitting / receiving unit 51 of the communication management system 5 transmits an addition notification of the image type information to another communication terminal (here, the smartphone 9) (step S60). This addition notification of the image type information includes the image data ID generated in step S56, and the IP address and image type information of the terminal itself (the video conference terminal 3a) which is the transmission source terminal stored in step S53. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the addition notification of the image type information. Note that the transmission destination of the transmitting / receiving unit 51 is another IP address associated with the same session ID as the IP address of the video conference terminal 3a in the session management DB 5001 (see FIG. 21). In other words, the transmission destination is another communication terminal in the same virtual conference room as the video conference terminal 3a.

[0132] Next, the storage / readout unit 99 of the smartphone 9 associates the image data ID, the IP address of the transmission source terminal, and the image type information received in step S60 and stores them as a new record in the image type management DB 9001 (see FIG. 16) (step S61). Similarly, an additional communication of the image type information is transmitted to the other communication terminals, the video conference terminal 3d and the PC 7, and the image type information is also stored in the image type management DBs 3001d and 7001 of the video conference terminals 3d and PC 7, respectively. As described above, the same information can be shared in the image type management DBs 3001a, 3001d, 7001, and 9001 of the communication terminals.

[0133] <Communication processing of captured image data> Next, a communication process of captured image data and material image data in a video call will be described with reference to Fig. 28 to Fig. 36. Fig. 28 is an image diagram showing the state of a video call using site A as an example.

[0134] As shown in Fig. 28, the video conference terminal 3a displays the document image generated by the PC 6 on the display 4 by superimposing it on the spherical panoramic image captured by the image capturing device 1a. Note that a whiteboard W on which the user A1 or the like can write characters, pictures, etc. is provided on the right side of Fig. 28.

[0135] Next, with reference to Fig. 29, a process will be described in which the photographed image data, audio data, and material image data obtained at the site A shown in Fig. 28 are transmitted to other communication terminals (the smartphone 9, the PC 7, and the video conference terminal 3d) via the communication management system 5. Fig. 29 is a sequence diagram showing the communication process of the photographed image data, audio data, and material image data in a video call.

[0136] First, the communication unit 18a of the photographing device 1a transmits photographed image data obtained by photographing an object or scenery and sound data obtained by collecting sound to the communication unit 38a of the video conference terminal 3a (step S101). In this case, since the photographing device 1a is a device capable of obtaining two hemispherical images that are the basis of a celestial sphere panoramic image, the photographed image data is composed of data of two hemispherical images as shown in Fig. 3(a) and (b). As a result, the communication unit 38a of the video conference terminal 3a receives the photographed image data and sound data.

[0137] Next, the communication unit 68 of the PC 6 transmits the material image data to be displayed by the display control unit 64 to the communication unit 38a of the video conference terminal 3a (step S102).

[0138] Next, the transmitting / receiving unit 31a of the video conference terminal 3a transmits the captured image data, sound data, and material image data sent from the image capturing device 1a to the communication management system 5 (step S103). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the captured image data, sound data, and material image data. Here, the captured image data includes an image data ID for identifying the captured image data to be transmitted.

[0139] Next, the transmission / reception unit 51 of the communication management system 5 transmits the captured image data, sound data, and material image data to the communication terminals (smartphone 9, PC 7, video conference terminal 3d) participating in the same video call as the video conference terminal 3a (steps S104, S105, S106). In each of these transmissions, the transmission of captured image data includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmission / reception unit 91 of the smartphone 9, the transmission / reception unit 71 of the PC 7, and the transmission / reception unit 31d of the video conference terminal 3d each receive the captured image data and image data ID, and further receive the sound data and material image data.

[0140] Next, a display example of the display 917 at the site B will be described with reference to FIG. 30. FIG. 30 is a display example of the display at the site D. Among these, FIG. 30(a) shows a case where the captured image data transmitted from the camera 1a at the site A via the video conference terminal 3a and the captured image data transmitted from the camera 1b at the site B are displayed as is without creating a celestial sphere panorama image and a predetermined area image. On the other hand, FIG. 30(b) shows a case where the celestial sphere panorama image and a predetermined area image are created from the captured image data transmitted from the camera devices 1a and 1b. Note that an image of the site A is displayed in the left display area (layout number "1") of the display 4d, and an image of the site B is displayed in the upper right display area (layout number "2"). Furthermore, an image of the site C is displayed in the middle right display area (layout number "3") of the display 4d, and an image of the site D (own site) is displayed in the lower right display area (layout number "4"). The display area with layout number "1" is the main display area, and the display areas with layout numbers "2," "3," and "4" are the sub-display areas. The images in the main display area and the sub-display area can be changed on each communication terminal. Usually, at each location, an image of the location where the central person in the video call is located is displayed in the main display area.

[0141] Here, when the captured image data transmitted from the photographing devices 1a and 1b capable of photographing spherical panoramic images is displayed as is, as shown in FIG. 30(a), the images of sites A and B are displayed as a front hemispherical image and a rear hemispherical image, as shown in FIGS. 3(a) and (b), respectively.

[0142] In contrast, when the image / sound processor 93 creates a celestial sphere panorama image from the captured image data output by the photographing devices 1a and 1b capable of obtaining two hemispherical images that are the basis of the celestial sphere panorama image, and further creates a predetermined area image, the predetermined area image, which is a planar image, is displayed as shown in Fig. 30(b). Note that, at site C, the photographing device 8 that obtains a general image photographs, and at site D, the video conference terminal 3d that obtains a general image photographs, so that a general image (here, a planar image) is displayed in both Figs. 30(a) and (b).

[0143] Furthermore, the users at each site can change the predetermined area related to the predetermined area image in the same spherical panoramic image. For example, the user B1 can operate the touch panel 921 or the like to have the reception unit 92 receive the movement of the predetermined area image, and the display control unit 94 can shift, rotate, reduce, or enlarge the predetermined area image. As a result, the predetermined area image can be displayed as shown in FIG. 30(c) from the predetermined area image in which some users A1 and A2 at the site A are displayed in the initial setting (default) as shown in FIG. 30(b). Specifically, FIG. 30(c) shows a state in which the predetermined area image including users A1 and A2 is changed to the predetermined area image including the whiteboard W in the captured image at the site A shown in FIG. 28.

[0144] 30(b) and (c) are examples of special image identification icons for indicating that the icons are predetermined area images showing a predetermined area T of a celestial sphere panoramic image. The display positions of the celestial sphere icons 191 and 192 may be anywhere, such as the upper left, lower left, or lower right, instead of the upper right. The types of the celestial sphere icons 191 and 192 are not limited to those shown in FIG. 30(b) and (c). Instead of the celestial sphere icons 191 and 192, characters such as "celestial sphere image" may be used, or a combination of an icon and characters may be used.

[0145] Next, the process in the image communication system when the predetermined area image is displayed as in Fig. 30(b) and when the predetermined area image is changed from Fig. 30(b) to Fig. 30(c) will be described with reference to Fig. 31. Fig. 31 is a sequence diagram showing the process of sharing predetermined area information. In Fig. 31, the video conference terminal 3a at site A is the third communication terminal, the video conference terminal 3d at site D is the other communication terminal, and the smartphone 9 at site B is the communication terminal (own terminal).

[0146] First, as shown in Fig. 30(b), when a user D1 or the like at site D uses a video conference terminal 3d to display a predetermined area image at site A, the transmitting / receiving unit 31d of the video conference terminal 3d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S111). This predetermined area information includes the IP address of the video conference terminal 3a which is the sender of the captured image data, and the IP address of the video conference terminal 3d which is the destination of the captured image data (the sender of the predetermined area information). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the predetermined area information.

[0147] Next, the storage / read unit 59 of the communication management system 5 stores the predetermined area information received in step S111 and the source and destination IP addresses in association with each other in the predetermined area management DB 5003 (step S112). Note that the processes of steps S111 and S112 are performed every time the predetermined area image is changed in the video conference terminal 3d, as shown in Fig. 30(b) to Fig. 30(c).

[0148] Next, the storage / reading unit 59 of the communication management system 5 reads out the latest pair of the predetermined area information and each IP address (the pair stored latest) at that time among the pairs of the predetermined area information and each IP address stored in the predetermined area management DB 5003 at regular intervals (e.g., 30 seconds) (step S113). Then, the transmission / reception unit 51 distributes (transmits) the predetermined area information including each IP address read out in step S113 to other communication terminals (video conference terminal 3a, smartphone 9, PC 7) that are performing the same video call as the video conference terminal 3d that is the sender of the predetermined area information (steps S114, S116, S118). As a result, the transmission / reception unit 31a of the video conference terminal 3a receives the predetermined area information. Then, the storage / reading unit 39a stores the predetermined area information and each IP address received in step S114 in the predetermined area management DB 3003a while keeping them associated with each other (step S115). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives the predetermined area information, the storage / reading unit 99 stores the predetermined area information received in step S116 in association with each IP address in the predetermined area management DB 9003 (step S117). Furthermore, in the PC 7, after the transmitting / receiving unit 71 receives the predetermined area information, the storage / reading unit 79 stores the predetermined area information received in step S118 in association with each IP address received in step S119 in the predetermined area management DB 7003.

[0149] Next, another process for sharing predetermined area information will be described with reference to Fig. 32. Fig. 32 shows another example of the process shown in Fig. 31, and is a sequence diagram showing another process for sharing predetermined area information. In Fig. 32, the video conference terminal 3a at the site A is a communication terminal (own terminal), and the video conference terminal 3d at the site D is another communication terminal.

[0150] In the process shown in Fig. 31, the communication management system 5 temporarily manages each predetermined area information transmitted from each communication terminal (see S112), and transmits each predetermined area information to each communication terminal other than the transmission source at regular intervals (see S114 to S119). In contrast, in this embodiment, as shown in Fig. 33, the communication terminal (here, the video conference terminal 3a) that transmits the captured image data temporarily manages the predetermined area information instead of the communication management system 5 (see S213), and transmits each predetermined area information to each communication terminal other than the own terminal at regular intervals (see S215 to S221). That is, in this embodiment, the communication terminal that transmits the captured image data manages what predetermined area image of the predetermined area T1 the captured image data transmitted by the own terminal (here, the video conference terminal 3a) is displayed as at the other communication terminals.

[0151] The process shown in Fig. 32 is realized by a hardware configuration common to the process shown in Fig. 31. Therefore, a description of the hardware configuration for realizing the process in Fig. 32 will be omitted.

[0152] 32, first, when a user D1 or the like at site D uses a video conference terminal 3d to display a predetermined area image at site A, the transmitting / receiving unit 31d of the video conference terminal 3d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S211). This predetermined area information includes the IP address of the video conference terminal 3a which is the sender of the captured image data, and the IP address of the video conference terminal 3d which is the destination of the captured image data (the sender of the predetermined area information). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the predetermined area information.

[0153] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits the predetermined area information including each IP address received in step S211 to the video conference terminal 3a that is the transmission source of the captured image data (step S212). As a result, the transmitting / receiving unit 31a of the video conference terminal 3a receives the predetermined area information.

[0154] Next, the storage / read unit 39a of the video conference terminal 3a stores the predetermined area information received in step S212 and the source and destination IP addresses in association with each other in the predetermined area management DB 3003a (step S213). The process of step S213 is a process for managing how the captured image data transmitted by the terminal itself (here, the video conference terminal 3a) is displayed on other communication terminals. The processes of steps S211 to S213 are performed every time the predetermined area image is changed on the video conference terminal 3d.

[0155] Next, the storage / reading unit 39a of the video conference terminal 3a reads out the latest pair of predetermined area information and each IP address at that time (the pair stored latest) from among the pairs of predetermined area information and each IP address stored in the predetermined area management DB 3003a at regular intervals (e.g., 30 seconds) (step S214). Then, the transmitting / receiving unit 31a transmits the predetermined area information including each IP address read out in step S214 to the communication management system 5 (step S215). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the predetermined area information.

[0156] Next, the transmission / reception unit 51 of the communication management system 5 transmits (distributes) the predetermined area information including each IP address received in step S215 to each communication terminal (video conference terminal 3d, smartphone 9, PC 7) (steps S216, S218, S220). As a result, the transmission / reception unit 31d of the video conference terminal 3d receives the predetermined area information. Then, the storage / reading unit 39d stores the predetermined area information received in step S216 in the predetermined area management DB 3003d in association with each IP address received in the same manner (step S217). Similarly, in the smartphone 9, after the transmission / reception unit 91 receives the predetermined area information, the storage / reading unit 99 stores the predetermined area information received in step S218 in the predetermined area management DB 9003 in association with each IP address received in the same manner (step S219). Furthermore, in PC 7, after the transmission / reception unit 71 receives the specified area information, the storage / reading unit 79 stores the specified area information received in step S220 in the specified area management DB 7003 in association with each of the IP addresses also received (step S221).

[0157] As a result, the predetermined area information indicating the predetermined area image changed at location A is transmitted to each of the communication terminals at other locations B, C, and D during the same video call, so that the predetermined area information indicating the predetermined area image being displayed at location A is also shared at other locations B, C, and D. This process is also performed in the same way when the predetermined area image is changed at locations B, C, and D, and the predetermined area information indicating the predetermined area image being displayed at the other locations is shared among all locations during the same video call.

[0158] Next, the setting of the superimposition position of the document image will be described with reference to Fig. 33 to Fig. 35. Fig. 33 is a sequence diagram showing a process in the image communication system when the superimposition position at which the document image is superimposed on the spherical panoramic image is changed at the site A that is the image distribution source.

[0159] First, the reception unit 32a receives the selection of a document image in response to an operation by the user A1 or the like at the site A (step S71). Examples of a method for selecting the document image include performing an operation such as "right-clicking", "double-clicking", or "entering a corresponding key" on the document image portion on the preview screen of the full-sphere panoramic image distributed by the user A1 or the like at the site A. In the case of a touch-enabled terminal, the operation may be realized by an operation such as "long tapping" or "double tapping" on the normal image in the preview area. Next, the reception unit 32a receives a change in the superimposition position of the document image in response to an operation by the user A1 or the like (step S72).

[0160] Next, the accepting unit 32a accepts the decision of the superimposition position of the document image in response to the operation of the user A1, etc., and the calculating unit 37a generates the superimposition position information (step S73). As an example of a method for deciding the superimposition position, the superimposition position may be automatically determined after the operation of step S72 is completed, or a method may be used in which a decision button on a separate screen is pressed or a predetermined key operation is performed.

[0161] Here, an example of a screen on which user A or the like changes the superimposition position of a document image will be described with reference to Fig. 34. Fig. 34 shows a preview screen of a spherical panoramic image at site A. The preview screen here is a screen for checking a video distributed by a certain site at one's own site without going through the communication network 100. As shown in Fig. 34, the preview screen shows a predetermined area image including users A2 and A4 and a whiteboard W in the spherical panoramic image. Characters and a pie chart are written on the whiteboard W.

[0162] 34 further shows an area R1 shown within a dotted frame and an area R2 shown within a solid frame. Areas R1 and R2 respectively show the overlay position before and after the change. By changing the overlay position of the document image to area 2 within the whiteboard W, the bar graph, which is the document image displayed by the PC 6, can be displayed alongside other characters and pie charts written within the whiteboard W, making it easier for the meeting participants to see.

[0163] 34, the operations of the user A1, etc. will be described in correspondence with the above-mentioned steps 71 to 73. First, the user A1, etc. selects the area R1 (step S71), then drags and drops the area in the direction of the arrow to change the superimposition position of the document image (step 72). Then, after moving the area to the desired area R2, the user performs an operation to determine the changed superimposition position (step 73).

[0164] The method of changing the superimposition position is not limited to the "drag and drop" shown in FIG. 34. As another example, the coordinates of the start and end points of the superimposed image may be changed by a predetermined key operation, or by pinching in and out in the case of a device that allows touch input. Also, if the size of the document image before and after the change is different, that is, the size of area R1 and area R2, the image may be enlarged or reduced to fit the changed size. Also, the dotted frame and arrow indicating area R1 may or may not be displayed to assist the user in the operation.

[0165] 33, the communication unit 68 then transmits the changed superimposition position information to the transmission / reception unit 51 of the communication system 5 (step S74). This superimposition position information includes the superimposition position coordinates, the IP address of the video conference terminal 3a which is the transmission source of the captured image data, and the IP address of the video conference terminal 3d which is the transmission destination of the captured image data (the transmission source of the superimposition position information). As a result, the transmission / reception unit 51 of the communication management system 5 receives the superimposition position information.

[0166] Next, the transmitting / receiving unit 51 distributes (transmits) the superimposition position information to other communication terminals (video conference terminal 3d, smartphone 9, PC 7) that are performing the same video call as the video conference terminal 3d that is the source of the predetermined area information (steps S75, S77, S79). As a result, the transmitting / receiving unit 31d of the video conference terminal 3d receives the superimposition position information. Then, the storing / reading unit 39a stores the superimposition position information received in step S75 in the superimposition position management DB 3004d in association with the source address (step S76). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives the superimposition position information, the storing / reading unit 99 stores the superimposition position information received in step S77 in the superimposition position management DB 9004 in association with the source IP address (step S78). Furthermore, in PC 7, after the transmission / reception unit 71 receives the superimposition position information, the storage / reading unit 79 stores the specific area information received in step S79 in the specific area management DB 7004 in association with each of the IP addresses also received (step S80).

[0167] 35 is a flowchart showing the display process of the document image in the receiving terminal. Note that, since the process of each receiving terminal is the same, the process of the smartphone 9 at the site B will be described here.

[0168] First, in the smartphone 9, the memory / reading unit 99 searches the image type management DB 9001 (see FIG. 16) using the image data ID received in step S106 shown in FIG. 29 as a search key, thereby reading out the corresponding image type information (source name) (step S131).

[0169] Next, the determination unit 95 determines whether the image type information read out in step S131 indicates a "special image" (step S132). Furthermore, if the determination result in step S132 is a special image, the determination unit 95 checks whether the document image data is received (step S133). In the check in step S133, for example, if there is image data that is not managed by an image data ID, it is determined to be document image data. If the determination result in step S133 is not received, that is, No, the image / sound processing unit 93 displays a spherical image and ends the process (step S137). On the other hand, if the determination result in step 133 is received, that is, Yes, the determination unit 95 determines whether the source IP address of the image information is stored in the superimposition position management DB 9004d (step 135). If the determination result in step 135 is not stored, that is, No, the image / sound processing unit 93 superimposes the material image data at an initial superimposition position (default superimposition position) that is a predetermined superimposition position of the omnidirectional image (step S136), and displays the omnidirectional image (step S137). If the determination result in step 135 is stored, that is, Yes, the storage / reading unit 99 acquires the superimposition position from the superimposition position management DB 9004d (step S138). Next, the image / sound processing unit 93 superimposes the material image data at the acquired superimposition position on the omnidirectional image (step 139), and displays the omnidirectional image (step S137).

[0170] Fig. 36 is an example of a display screen at location B where the display process described in Fig. 35 has been executed. As shown in Fig. 36, at location B, a document image (superimposed image) is displayed in area R2, which is the superimposed position to which the superimposed image in Fig. 34 has been changed at location A.

[0171] As a result, the overlay position information changed at location A is transmitted to the communication terminals of other locations B, C, and D that are participating in the same video call, and is then displayed and shared at the other locations B, C, and D in the same position as the document screen being displayed at location A.

[0172] <Image quality control for photographed images and document images> The image quality control of the captured image and the document image executed by the communication management system 5 will be described with reference to Fig. 36 to Fig. 39. Fig. 37 is a sequence diagram showing the process of changing the image quality. Fig. 38 is a diagram showing the case where the predetermined area image including the document image is changed. Fig. 39 is a diagram showing the predetermined area after the change.

[0173] First, the calculation unit 97 of the smartphone 9 calculates the display ratio in the display area of ​​the site A with respect to the predetermined area image of the site A and the superimposed image (document image) R2 shown in Fig. 36 (step S151). In Fig. 36, the superimposed image R2 is entirely displayed.

[0174] Next, the transmitting / receiving unit 91 of the smartphone 9 transmits ratio information indicating the ratio calculated by the calculation unit 97 to the communication management system 5 (step S152). At this time, the IP address of the communication terminal that transmitted the image (here, the video conference terminal 3a) and the IP address of the communication terminal that transmitted the image (here, the smartphone 9) are also transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the ratio information and each IP address.

[0175] Next, in the communication management system 5, the storage / reading unit 59 associates the ratio information received by the transmission / reception unit 51 with each IP address, and stores and manages the information in the ratio management DB 5004 (step S153).

[0176] Next, similar to step S103 in FIG. 29, the transceiver unit 51 of the communication management system 5 receives the captured image data, audio data, and document image data accompanied by the image data ID transmitted from the transceiver unit 31a of the video conference terminal 3a (step S154).

[0177] Next, the storage / reading unit 59 searches the ratio management DB 5004 using the IP address of the communication terminal that sent the image (here, the video conference terminal 3a) and the IP address of the communication terminal that sent the image (here, the smartphone 9) as search keys to read out the corresponding ratio information, and the change unit 53 changes the image quality of the captured image related to the captured image data received in step S154 and the document image related to the document image data according to the ratio indicated by the ratio information. In this case, if the ratio indicated in the ratio information is large, the image quality is increased, and if the ratio is small, the image quality is decreased.

[0178] Next, the transmitting / receiving unit 51 transmits the photographed image data, the document image data, and the sound data according to the ratio to the smartphone 9 (step S156). In this case, the photographed image data is also accompanied by an image data ID. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the photographed image data, the document image data, and the sound data.

[0179] Next, in the smartphone, the image / sound processing unit 93 performs image processing to superimpose a superimposed image (document image) related to the document image data on the whole image related to the photographed image data received by the transmission / reception unit 91 (step S157). Then, the display control unit 94 causes the display 917 to display an image as shown in FIG.

[0180] Meanwhile, when users B1 and B2 at site B change the specified area image at site A as shown in FIG. 38, the ratio of the entire image to the document image (superimposed image) in the display area of ​​the image at site A changes. In FIG. 38, the ratio of the superimposed image R2 is smaller than in FIG. 36. FIG. 39 shows the range of the display area T at site A in the case of FIG. 38. In such a case, the process is started again from step S151. Here, since the ratio of the superimposed image R2 is smaller, the image quality of the captured image related to the captured image data transmitted by step S156 increases, and the image quality of the document image related to the document image data also transmitted decreases.

[0181] The users B1 and B2 at the site B can change the predetermined position where the superimposed screen R3 is superimposed by using the touch panel 921. In this case, the reception unit 92 receives the change of the predetermined position, and the image / sound processing unit 93 superimposes the document image at the predetermined position.

[0182] <<Main Effects of This Embodiment>> As described above, according to this embodiment, the process of steps S151 to S154 has the effect of solving the problem that the receiving user overlooks a document image that the sending user wants to draw attention to.

[0183] In addition, a communication terminal such as the video conference terminal 3a can create a celestial sphere panoramic image and a predetermined area image according to the corresponding image type information based on the image data ID sent together with the image data. This has the effect of preventing the front hemispherical image and the rear hemispherical image from being displayed as shown in Fig. 30(a).

[0184] Furthermore, according to this embodiment, in a conference system or the like in which a receiver displays a certain image (a superimposition source image) by superimposing another image (a superimposition destination image) on a part of the image, the sender can change the display position of the superimposition source image in the superimposition destination image. Therefore, by changing the superimposition position when the superimposition source image is superimposed in a place not intended by the sender or when the sender wants to superimpose the image in a different position during the conference, it is possible to obtain an effect of making the image easier to see and understand.

[0185] 〔supplement〕 In the above embodiment, a three-dimensional celestial sphere panoramic image is described as an example of a panoramic image to be superimposed, but a two-dimensional panoramic image may also be used.

[0186] In the above embodiment, the communication management system 5 relays the predetermined area information transmitted from each communication terminal, but this is not limiting, and each communication terminal may directly transmit and receive the predetermined area information to each other.

[0187] Each function of the above-mentioned embodiment can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU (Graphics Processing Unit), or a conventional circuit module designed to execute each function described above. [Explanation of symbols]

[0188] 1a Imaging device 1b Imaging device 3a Video conferencing terminal (an example of another communication terminal) 3d Video conferencing terminal (an example of a communication terminal) 4a Display 4d Display (an example of a display method) 5. Communications Management System 6 PC 7 PC 8. Imaging Equipment 9 Smartphones (an example of another communication device) 91 Transmitting / receiving unit (an example of a transmitting means, an example of a receiving means) 92 Reception unit (an example of reception means) 93 Image / sound processing unit (an example of image processing means) 94 Display control unit (an example of a display control means) 97 Calculation unit (an example of a calculation means) 53 Change unit (an example of a change means) 3004a Superimposition position management DB (an example of a superimposition position management means) 3004d Superimposition position management DB (an example of a superimposition position management means) 5004 Ratio management DB (an example of ratio management means) 7004 Superimposition position management DB (an example of a superimposition position management means) 9004 Superimposition position management DB (an example of a superimposition position management means) [Prior art documents] [Patent documents]

[0189] [Patent Document 1] JP 2012-178135 A [Patent Document 2] JP 2011-223076 A [Patent Document 3] JP 2012-156820 A

Claims

1. A program to be executed by a communication terminal, a display control unit that displays an image of a first region that is a part of the first image; The display control means is a program for superimposing and displaying the second image on the first image based on superimposition position information set in another communication terminal, the superimposition position information indicating where the second image is to be superimposed on the first image.

2. The program described in Claim 1, wherein the display control means changes the image of the first area, which is a partial area in the first image, to an image of a second area, which is a partial area in the first image, based on the operation of the user of the communication terminal, and displays the image.

3. The program described in Claim 2, wherein the display control means changes the ratio of the first image to the second image in the display area of ​​the communication terminal when the image is changed to the second area, which is a part of the first image, based on the operation of the user of the communication terminal.

4. A program described in any one of claims 1 to 3, wherein the display control means changes the position at which the second image is superimposed based on the operation of the user of the communication terminal.

5. A program described in any one of claims 1 to 4, wherein the second image is an image selected by a user of the other communication terminal.

6. A program described in any one of claims 1 to 5, wherein the first image is a spherical panoramic image.

7. a display control means for displaying an image of a first region that is a partial region of the first image; The display control means is a communication terminal that displays the second image superimposed on the first image based on superimposition position information set by another communication terminal, the superimposition position information indicating where the second image is to be superimposed on the first image.

8. An image communication system having a communication terminal and another communication terminal, the communication terminal includes a display control means for displaying an image of a first area that is a partial area of ​​a first image; the other communication terminal sets superimposition position information indicating where on the first image the second image is to be superimposed; The display control means displays the second image superimposed on the first image based on the superimposition position information set by the other communication terminal.

9. A display method for a communication terminal, comprising: a display control step of displaying an image of a first region that is a partial region of the first image, The display control step is a display method in which the second image is superimposed on the first image based on superimposition position information set in another communication terminal, the superimposition position information indicating where the second image is to be superimposed on the first image.

Images