Communication terminal, image communication system, display method, and program

Abstract

To solve the problems that a photographing device capable of acquiring an entire-celestial-sphere is connected to each communication terminal, the entire-celestial-sphere acquired by this photographing device is transmitted to a mating communication terminal, a prescribed region image indicating a prescribed region as a part of the image from the entire-celestial-sphere is displayed on a display in the mating communication terminal, and thus, a user of each point can determine the prescribed region image that interests himself from all images of the entire-celestial-sphere and display it, but the user of an arbitrary point cannot understand which part of the prescribed region image from all images at the other point is displayed when the predetermined region image is displayed by originally determining it at each point.SOLUTION: A communication terminal (an own terminal) displays a gaze point mark m41 that indicates where a predetermined region image in a display is located by using the prescribed region image in the display in an own terminal as a reference by the other communication terminal in a whole image or a direction mark m11 or the like indicating their direction.SELECTED DRAWING: Figure 34

Description

[Technical Field]

[0001] The present invention relates to a communication terminal, an image communication system, a display method, and a program. [Background technology]

[0002] Conference systems that hold remote conferences between remote locations via communication networks such as the Internet are becoming widespread. In these conference systems, in a conference room where one of the parties, such as attendees, is present, a communication terminal of the remote conference system is used to capture and collect images of the conference room and audio, such as speech, of the parties, and converts these into digital data and transmits them to the communication terminal of the other party. This allows a video call to be conducted with images displayed on a display and audio output from a speaker in the other party's conference room, making it possible to hold a conference between remote locations in an environment similar to a real conference (see Patent Document 1).

[0003] Also, a technique is known in which a communication terminal is connected to an imaging device capable of acquiring a celestial sphere panoramic image in real time, the celestial sphere panoramic image acquired from the imaging device is transmitted to each communication terminal of the other party, and each communication terminal of the other party displays a predetermined area image indicating a predetermined area that is a part of the celestial sphere panoramic image on a display or the like (see Patent Document 2). This allows users at each location to independently determine and display a predetermined area image indicating a predetermined area that interests them within the entire image of the celestial sphere panoramic image. Summary of the Invention [Problem to be solved by the invention]

[0004] However, when each location independently determines and displays a predetermined area image for the same overall image, a user at a given location cannot know which portion of the overall image is being displayed at another location. Therefore, if a predetermined area image unrelated to the topic of a meeting or the like is still displayed during the meeting or the like, the user at that location may have difficulty following the topic of the meeting or the like. [Means for solving the problem]

[0005] The invention of claim 1 is a communication terminal capable of displaying an image of a predetermined area in an overall image, characterized in that it has: a receiving means for receiving first predetermined information for identifying the first predetermined area, which is transmitted from another communication terminal displaying a first predetermined area image, which is an image of a first predetermined area in the overall image; a calculation means for, when the communication terminal is displaying a second predetermined area image, which is an image of a second predetermined area in the overall image, calculating the position of the first predetermined area relative to the second predetermined area in the overall image based on the second predetermined information for identifying the second predetermined area and the first predetermined information received by the receiving means; and a display control means for displaying position information indicating the position calculated by the calculation means, including it in the second predetermined area image. [Effects of the Invention]

[0006] As described above, according to the present invention, a user at a given location can grasp which portion of the entire image is being displayed at another location as a specific region image, which has the effect of making it easier to follow topics in a meeting, for example, compared to the conventional method. [Brief explanation of the drawings]

[0007] [Figure 1] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. 1C is a plan view of the imaging device. [Figure 2] FIG. 10 is an image diagram of the imaging device in use. [Figure 3](a) is a hemispherical image (before) taken with the imaging device, (b) is a hemispherical image (after) taken with the imaging device, and (c) is an image represented by the Mercator projection. [Figure 4] (a) A conceptual diagram showing how a sphere is covered with a Mercator image, and (b) a diagram showing a spherical panoramic image. [Figure 5] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere panoramic image is a three-dimensional sphere. [Figure 6] 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 7] 10 is a diagram showing the relationship between predetermined area information and an image of a predetermined area T. FIG. [Figure 8] A diagram showing points in three-dimensional Euclidean space in spherical coordinates. [Figure 9] 1 is a schematic diagram of an image communication system according to an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating a hardware configuration of the imaging device. [Figure 11] FIG. 2 is a hardware configuration diagram of a video conference terminal. [Figure 12] FIG. 2 is a hardware configuration diagram of a communication management system and a PC. [Figure 13] FIG. 1 is a hardware configuration diagram of a smartphone. [Figure 14] FIG. 1 is a functional block diagram of a portion of an image communication system. [Figure 15] FIG. 1 is a functional block diagram of a portion of an image communication system. [Figure 16] FIG. 10 is a conceptual diagram illustrating an image type management table. [Figure 17] FIG. 10 is a conceptual diagram illustrating a photographing device management table. [Figure 18] FIG. 10 is a conceptual diagram illustrating a predetermined area management table. [Figure 19] FIG. 10 is a conceptual diagram illustrating a session management table. [Figure 20] FIG. 10 is a conceptual diagram illustrating an image type management table. [Figure 21] FIG. 10 is a conceptual diagram illustrating a predetermined area management table. [Figure 22] FIG. 10 is a sequence diagram illustrating a process of joining a specific communication session. [Figure 23] FIG. 10 is a diagram showing a selection screen for a communication session (virtual conference room). [Figure 24] FIG. 10 is a sequence diagram showing a management process of image type information. [Figure 25] 1A and 1B are conceptual diagrams showing the state of a video call, in which (a) shows a case where the photographing device 1a is not used, and (b) shows a case where the photographing device 1a is used. [Figure 26] FIG. 10 is a sequence diagram showing a communication process of captured image data and sound data during a video call. [Figure 27] 27(a) shows display examples on the display at site B, where FIG. 27(a) shows a case where the image data sent from the photographing devices 1a and 1b is displayed as is without creating a celestial sphere panoramic image or a predetermined area image, FIG. 27(b) shows a case where the celestial sphere panoramic image and a predetermined area image are created from the image data sent from the photographing devices 1a and 1b, and FIG. 27(c) shows a case where the predetermined area image of FIG. 27(b) is changed. [Figure 28] FIG. 10 is a sequence diagram showing a process of sharing predetermined area information. [Figure 29] 10 is a flowchart showing a display process of a predetermined area image. [Figure 30] 10 is a diagram showing a method for deriving the position of the gaze point of another base in a predetermined area image of the own base. FIG. [Figure 31] (a) A diagram showing the definition of angles, and (b) a diagram showing the definition of angle ranges. [Figure 32] 10 shows an image of the main display area, which is an example of a display of a predetermined area image including a display direction mark. [Figure 33] 10 shows an image of the main display area, which is an example of a display of a predetermined area image including a gaze point mark. [Figure 34] 10 shows an image of the main display area, which is an example of a display of a predetermined area image including a gaze point mark and a display direction mark. [Figure 35] FIG. 10 is a sequence diagram showing another process for sharing predetermined area information. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] [First embodiment] First, the first embodiment will be described with reference to FIGS.

[0010] <<Outline of the embodiment>> <How to generate a spherical panoramic image> A method for generating a spherical panoramic image will be described with reference to FIGS.

[0011] First, the appearance of the imaging device 1 will be described using Fig. 1. The imaging device 1 is a digital camera for obtaining captured images that serve as the basis for a three-dimensional spherical (360°) panoramic image. Fig. 1(a) is a left side view of the imaging device, Fig. 1(b) is a front view of the imaging device, and Fig. 1(c) is a plan view of the imaging device.

[0012] As shown in FIG. 1(a), the photographing device 1 is small enough to be held in one hand. As shown in FIGS. 1(a), 1(b), and 1(c), an image sensor 103a is provided on the front side (front side) of the upper portion of the photographing device 1, and an image sensor 103b is provided on the rear side (rear side). These image sensors 103a and 103b are used in conjunction with optical components (e.g., fisheye lenses 102a and 102b in FIG. 10, which will be described later) capable of capturing hemispherical images (with an angle of view of 180° or more). As shown in FIG. 1(b), an operation unit 115, such as a shutter button, is provided on the surface of the photographing device 1 opposite the front side.

[0013] Next, the usage of the image capturing device 1 will be described with reference to Fig. 2. Fig. 2 is an image diagram of the image capturing device in use. As shown in Fig. 2, the image capturing device 1 is used, for example, by being held by a user and capturing images of subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image capturing element 103a and the image capturing element 103b shown in Fig. 1, respectively.

[0014] Next, an outline of the process of creating a celestial sphere panoramic image from images captured by the image capturing device 1 will be described with reference to Fig. 3 and Fig. 4. Fig. 3(a) is a diagram showing a hemispherical image (front side) captured by the image capturing device, Fig. 3(b) is a diagram showing a hemispherical image (rear side) captured by the image capturing device, and Fig. 3(c) is a diagram showing an image expressed by Mercator projection (hereinafter referred to as "Mercator image"). Fig. 4(a) is a conceptual diagram showing a state in which a sphere is covered with a Mercator image, and Fig. 4(b) is a diagram showing a celestial sphere panoramic image.

[0015] As shown in Fig. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a fisheye lens 102a (described later). Also, as shown in Fig. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a fisheye lens 102b (described later). Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are combined by the image capturing device 1 to create a Mercator image as shown in Fig. 3(c).

[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted to cover the spherical surface as shown in FIG. 4(a), and a spherical panoramic image as shown in FIG. 4(b) is created. In this way, the spherical panoramic image is expressed as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the spherical panoramic image may be a still image or a video.

[0017] As described above, a spherical panoramic image is an image pasted to cover the spherical surface, which gives a sense of incongruity to people. Therefore, by displaying a predetermined region of the spherical panoramic image (hereinafter referred to as a "predetermined region image") as a flat image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to people. This will be described with reference to FIGS. 5 and 6.

[0018] FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user viewing the celestial sphere image CE displayed as a three-dimensional sphere. FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing the predetermined area image when displayed on a display. In FIG. 6(a), the celestial sphere image CE shown in FIG. 4 is represented by a three-dimensional sphere CS. If the celestial sphere image CE generated in this way is the celestial sphere CS, then the virtual camera IC is located inside the celestial sphere image CE as shown in FIG. 5. The predetermined area T in the celestial sphere image CE is the shooting area of ​​the virtual camera IC, and is specified by predetermined area information that indicates the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE.

[0019] The predetermined area image Q shown in FIG. 6(a) is then displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined area image represented by the initial (default) predetermined area information. Note that the predetermined area information may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the position coordinates of the virtual camera IC. The following explanation will be given using the shooting direction (rH, rV) and angle of view (α) of the virtual camera IC.

[0020] The relationship between the predetermined region information and the image of the predetermined region T will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating the relationship between the predetermined region information and the image of the predetermined region T. As shown in FIG. 7, rH represents the horizontal radius, rV represents the vertical radius, and α represents the angle of view (Angle). That is, the posture of the virtual camera IC is changed so that the gaze point of the virtual camera IC, indicated by the shooting direction (rH, rV), becomes the center point CP of the predetermined region T, which is the shooting region of the virtual camera IC. The predetermined region image Q is an image of the predetermined region T in the celestial sphere image CE. f represents the distance from the virtual camera IC to the center point CP. L represents the distance between any vertex of the predetermined region T and the center point CP (2L is the diagonal). In FIG. 7, the trigonometric function represented by the following (Equation 1) generally holds.

[0021] Lf=tan(α / 2) (Equation 1) FIG. 8 is a diagram showing points in a three-dimensional Euclidean space in spherical coordinates. The position coordinates of the center point CP when expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius vector, polar angle, and azimuth angle, respectively. The radius vector r is the distance from the origin of the three-dimensional virtual space containing the omnidirectional panoramic image to the center point CP, and is therefore equal to f. FIG. 8 is a diagram showing these relationships. Hereinafter, the description will be given using the position coordinates (r, θ, φ) of the virtual camera IC.

[0022] <Outline of the image communication system> Next, an outline of the configuration of the image communication system of this embodiment will be described with reference to Fig. 9. Fig. 9 is a schematic diagram of the configuration of the image communication system of this embodiment.

[0023] 9, the image communication system of this embodiment is configured with photographing devices 1a and 1b, video conference terminals 3a and 3b, displays 4a and 4b, a communication management system 5, a PC (Personal Computer) 7, a photographing device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

[0024] Of these, the image capturing devices 1a and 1b are special digital cameras for capturing images of subjects, scenery, etc. to obtain two hemispherical images that are the basis for a celestial sphere panoramic image, as described above, while the image capturing device 8 is a general digital camera for capturing images of subjects, scenery, etc. to obtain a general planar image.

[0025] The video conference terminals 3a and 3b are dedicated to video conferences and display images of video calls on their displays 4a and 4b, respectively, via wired cables such as USB (Universal Serial Bus) cables. The video conference terminal 3a normally captures images using a camera 312 (see FIG. 11 ), which will be described later. However, when the video conference terminal 3a is connected to a cradle 2a (which holds the image capture device 1a) via a wired cable, the image capture device 1a is prioritized, and two hemispherical images that form the basis of a celestial sphere panoramic image can be obtained. When a wired cable is used, the cradle 2a not only communicates between the image capture device 1a and the video conference terminal 3a, but also serves to supply power to and support the image capture device 1a. Here, the image capture device 1a, the cradle 2a, the video conference terminal 3a, and the display 4a are all located at the same site, Site A. Four users, A1, A2, A3, and A4, are participating in a video call at Site A. Meanwhile, the video conference terminal 3d and the display 4d are located at the same site, Site D. At site D, three users D1, D2, and D3 are participating in a video call.

[0026] The communication management system 5 manages and controls communications between the video conference terminals 3a and 3b, the PC 7, and the smartphone 9, and manages the types of image data (general images and special images) sent and received. Therefore, the communication management system is also a communication control system. Here, the special images are spherical panoramic images. The communication management system 5 is installed at a service company that provides video communication services. The communication management system 5 may be constructed by a single computer, or may be constructed by multiple computers in which each section (function, means, or storage section) is divided and arbitrarily assigned.

[0027] The PC 7 is capable of video calls by attaching the camera device 8. In this example, the PC 7 and the camera device 8 are located at the same location, site C. At site C, one user C is participating in the video call.

[0028] The smartphone 9 displays an image of the video call on a display 917 (described later) provided on the smartphone 9. The smartphone 9 normally captures images using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 (described later) provided on the smartphone 9, but can also acquire two hemispherical image data that form the basis of the omnidirectional panoramic image acquired by the image capture device 1b using wireless communication technology such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark). When using wireless communication technology, the cradle 2b only serves to supply power to the image capture device 1b and support the image capture device 1b. Here, the image capture device 1b, the cradle 2b, and the smartphone 9 are all located at the same site, site B. At site B, two users B1 and B2 are participating in the video call.

[0029] The video conference terminals 3 a and 3 d, the PC 7, and the smartphone 9 are examples of communication terminals. OpenGL ES is installed in each communication terminal, and the communication terminals can create predetermined area information indicating a partial area of ​​a celestial sphere panoramic image and create a predetermined area image from a celestial sphere panoramic image transmitted from another communication terminal.

[0030] The arrangement of the terminals (communication terminal, display, and image capture device), devices, and users shown in FIG. 9 is one example, and other examples may be used. For example, at site C, instead of image capture device 8, an image capture device capable of capturing a celestial sphere panoramic image may be used. Furthermore, communication terminals also include digital televisions, smartwatches, car navigation devices, and the like. Furthermore, hereinafter, any image capture device among image capture devices 1a and 1b will be referred to as "image capture device 1." Furthermore, any video conference terminal among video conference terminals 3a and 3b will be referred to as "video conference terminal 3." Furthermore, any display among displays 4a and 4b will be referred to as "display 4."

[0031] <<Hardware configuration of the embodiment>> 10 to 13, the hardware configurations of the image capturing device 1, the video conference terminal 3, the communication management system 5, the PC 7, and the smartphone 9 of this embodiment will be described in detail. Note that the image capturing device 8 is a general camera, and therefore a detailed description thereof will be omitted.

[0032] <Hardware configuration of the imaging device 1> First, the hardware configuration of the image capturing device 1 will be described using Fig. 10. Fig. 10 is a hardware configuration diagram of the image capturing device 1. In the following, the image capturing device 1 is assumed to be an omnidirectional (omnidirectional) image capturing device using two image capturing elements, but the number of image capturing elements may be any number greater than or equal to two. Furthermore, the image capturing device does not necessarily have to be a device dedicated to omnidirectional image capturing; an ordinary digital camera, smartphone, or the like may be equipped with an omnidirectional image capturing unit as an add-on, so that it has substantially the same functions as the image capturing device 1.

[0033] As shown in FIG. 10, the photographing device 1 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I / F 116, a communication unit 117, and an antenna 117a.

[0034] Of these, imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b, each with a field angle of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each wide-angle lens. Imaging elements 103a and 103b include an image sensor such as a CMOS sensor or a CCD (Charge Coupled Device) sensor that converts optical images captured by fisheye lenses 102a and 102b into image data in the form of electrical signals and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers that set various commands and parameters required for the operation of the imaging elements.

[0035] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are connected via a serial I / F bus (such as an I2C bus) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to a CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like.

[0036] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then synthesizes the image data to create Mercator image data such as that shown in Figure 3(c).

[0037] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0038] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 1 have a preview display function or a function for displaying moving images on a display (for example, the display of the video conference terminal 3a). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0039] As will be described later, the imaging control unit 105 also functions as a synchronization control means that synchronizes the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the photographing device 1 is not provided with a display unit, but may be provided with a display unit.

[0040] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I / F bus and performs predetermined processing on the sound data.

[0041] The CPU 111 controls the overall operation of the imaging device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and processed Mercator image data.

[0042] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. The user operates the operation buttons to input various shooting modes, shooting conditions, etc.

[0043] The network I / F 116 is a general term for an interface circuit (such as a USB I / F) with external media such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. The Mercator image data stored in the DRAM 114 is recorded on external media via the network I / F 116, or transmitted to an external device such as the video conference terminal 3a via the network I / F 116 as needed.

[0044] The communication unit 117 communicates with an external device such as the video conference terminal 3a by short-range wireless technology such as WiFi or NFC (Near Field Communication) via an antenna 117a provided in the image capturing device 1. The communication unit 117 can also transmit Mercator image data to an external device of the video conference terminal 3a.

[0045] The electronic compass 118 calculates the orientation and tilt (roll rotation angle) of the image capturing device 1 from the Earth's magnetism and outputs the orientation and tilt information. This orientation and tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of captured images. The related information also includes data such as the date and time the image was captured and the data size of the image data.

[0046] <Videoconferencing terminal hardware configuration> Next, the hardware configuration of the video conference terminal 3 will be described with reference to Fig. 11. Fig. 11 is a diagram showing the hardware configuration of the video conference terminal. As shown in Fig. 11, the video conference terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, an operation button 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an image sensor I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319.

[0047] Of these, the CPU 301 controls the overall operation of the video conference terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as an IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and audio data. The SSD (Solid State Drive) 305 controls the reading and writing of various data from and to the flash memory 304 under the control of the CPU 301. Note that an HDD may be used instead of an SSD. The media I / F 307 controls the reading and writing (storage) of data from and to a recording medium 306, such as a flash memory. The operation button 308 is a button that is operated when selecting a destination for the video conference terminal 3, etc. The power switch 309 is a switch for turning the power of the video conference terminal 3 ON / OFF.

[0048] The network I / F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 301. The image sensor I / F 313 is a circuit that controls the driving of the camera 312. The microphone 314 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 316 is a circuit that processes input and output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I / F 317 is a circuit that transmits image data to the external display 4 under the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) or Bluetooth (registered trademark).

[0049] 11. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 301 shown in FIG.

[0050] The display 4 is a type of display means configured with a liquid crystal or organic electroluminescence (EL) display that displays an image of a subject, operation icons, etc. The display 4 is connected to the display I / F 317 by a cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0051] The camera 312 includes a lens and a solid-state imaging element that converts light into an electric charge to digitize an image (video) of a subject, and a CMOS sensor, CCD sensor, or the like is used as the solid-state imaging element. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 via a USB (Universal Serial Bus) cable, or the like. When an external camera is connected, the external camera is driven under the control of the CPU 301 in preference to the built-in camera 312. Similarly, when an external microphone or an external speaker is connected, the external microphone or external speaker is driven under the control of the CPU 301 in preference to the built-in microphone 314 or built-in speaker 315, respectively.

[0052] The recording medium 306 is detachable from the video conference terminal 3. As long as it is a non-volatile memory that reads or writes data under the control of the CPU 301, it is not limited to the flash memory 304, and an EEPROM (Electrically Erasable and Programmable ROM) or the like may also be used.

[0053] <Communication management system, PC hardware configuration> Next, the hardware configuration of the communication management system 5 and the PC 7 will be described with reference to Fig. 12. Fig. 12 is a diagram showing the hardware configuration of the communication management system and the PC. Note that since the communication management system 5 and the PC 7 are both computers with the same configuration, the following will describe the configuration of the communication management system 5, and will omit a description of the configuration of the PC 7.

[0054] The communication management system 5 includes a CPU 501 for controlling the overall operation of the communication management system 5, a ROM 502 storing a program used to drive the CPU 501, such as an IPL, a RAM 503 used as a work area for the CPU 501, a HD 504 for storing various data such as programs for the communication management system 5, a HDD (Hard Disk Drive) 505 for controlling reading or writing of various data from or to the HD 504 under the control of the CPU 501, a media drive 507 for controlling reading or writing (storing) of data from or to a recording medium 506, such as a flash memory, a display 508 for displaying various information such as a cursor, menu, window, character, or image, a network I / F 509 for data communication using the communication network 100, a keyboard 511 having multiple keys for inputting characters, numbers, various instructions, etc., a mouse 512 for selecting and executing various instructions, selecting a processing target, moving the cursor, etc., and a CD-RW (Compact Disc The CD-RW drive 514 controls the reading of various data from the CD-ROM (CD-RW / CD-Rewritable) 513, and bus lines 510 such as an address bus and a data bus for electrically connecting the above components as shown in FIG. 12.

[0055] <Hardware configuration of smartphone> Next, the hardware of the smartphone will be described with reference to Fig. 13. Fig. 13 is a hardware configuration diagram of the smartphone. As shown in Fig. 13, the smartphone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration / direction sensor 906, a media I / F 908, and a GPS receiver 909.

[0056] Of these, the CPU 901 controls the overall operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901, such as the IPL. The RAM 903 is used as a work area for the CPU 901. The EEPROM 904 reads and writes various data, such as smartphone programs, under the control of the CPU 901. The CMOS sensor 905 captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 901. The acceleration / azimuth sensor 906 is a variety of sensors, such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor. The media I / F 908 controls the reading and writing (storage) of data from and to a recording medium 907, such as a flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0057] The smartphone 9 also includes a long-distance communication circuit 911, a camera 912, an image sensor I / F 913, a microphone 914, a speaker 915, an audio input / output I / F 916, a display 917, an external device connection I / F 918, a short-distance communication circuit 919, an antenna 919a of the short-distance communication circuit 919, and a touch panel 921.

[0058] Of these, the long-distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 901. The image sensor I / F 913 is a circuit that controls the operation of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs audio. The sound input / output I / F 916 is a circuit that processes the input and output of audio signals between the microphone 914 and the speaker 915 under the control of the CPU 901. The display 917 is a type of display means, such as an LCD or organic EL, that displays an image of a subject, various icons, etc. The external device connection I / F 918 is an interface for connecting various external devices. The short-distance communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means that allows a user to operate the smartphone 9 by pressing the display 917.

[0059] The smartphone 9 also includes a bus line 910. The bus line 910 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 901.

[0060] It should be noted that the recording media such as CD-ROMs on which the above programs are stored, and the HDDs on which these programs are stored, can be provided domestically or internationally as program products.

[0061] <<Functional configuration of the embodiment>> Next, the functional configuration of this embodiment will be described with reference to Figures 14 to 20. Figures 14 and 15 are functional block diagrams of a portion of the image communication system.

[0062] <Functional configuration of the imaging device 1a> 14, the image capturing device 1a has a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a storage / readout unit 19a. Each of these units is a function or means realized by operating any of the components shown in FIG. 10 in response to an instruction from the CPU 111 in accordance with a program for the image capturing device that has been loaded from the SRAM 113 onto the DRAM 114.

[0063] The image capturing device 1 also has a storage unit 1000a configured by the ROM 112, SRAM 113, and DRAM 114 shown in Fig. 10. The storage unit 1000a stores a GUID (Globally Unique Identifier) ​​of the image capturing device 1.

[0064] The photographing device 1b has a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, a memory / readout unit 19b, and a memory unit 1000b, but since these have the same functions as the reception unit 12a, the imaging unit 13a, the sound collection unit 14a, the communication unit 18a, the memory / readout unit 19a, and the memory unit 1000a in the photographing device 1a, their descriptions will be omitted.

[0065] (Functional configuration of the imaging device 1a) Next, each functional configuration of the image capturing device 1a will be described in more detail with reference to FIGS.

[0066] The reception unit 12a of the photographing device 1 is mainly realized by the processing of the operation unit 115 and the CPU 111 shown in FIG. 10, and receives operation inputs from the user.

[0067] The imaging section 13a is mainly realized by the imaging unit 101, image processing unit 104, imaging control unit 105, and processing by the CPU 111 shown in FIG. 10, and captures images of scenery and the like to obtain captured image data.

[0068] The sound collection unit 14a is realized by the processing of the sound processing unit 108 and the sound processing unit 109 shown in FIG. 10, and the CPU 111, and collects sounds around the image capture device 1a.

[0069] The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38a of the video conference terminal 3a by short-range wireless communication technology such as the NFC standard, BlueTooth, or WiFi.

[0070] The storage / reading unit 19a is mainly realized by the processing of the CPU 111 shown in FIG. 10, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.

[0071] <Functional configuration of the video conference terminal 3a> 14, the video conference terminal 3a includes a transmitting / receiving unit 31a, a receiving unit 32a, an image / sound processing unit 33a, a display control unit 34a, a determining unit 35a, a creating unit 36a, a calculating unit 37a, a communication unit 38a, and a storing / reading unit 39a. Each of these units is a function or means realized by operating any of the components shown in FIG. 11 in response to an instruction from the CPU 301 in accordance with the program for the video conference terminal 3a that has been loaded from the flash memory 304 onto the RAM 303.

[0072] The video conference terminal 3a also has a storage unit 3000a configured with a ROM 302, a RAM 303, and a flash memory 304 shown in Fig. 11. The storage unit 3000a contains an image type management DB 3001a, a photographing device management DB 3002a, and a predetermined area management DB 3003a. Of these, the image type management DB 3001a is configured with the image type management table shown in Fig. 16. The photographing device management DB 3002a is configured with the photographing device management table shown in Fig. 17. The predetermined area management DB 3003a is configured with the predetermined area management table shown in Fig. 18.

[0073] The video conference terminal 3d has a transmitting / receiving unit 31d, a receiving unit 32d, an image / sound processing unit 33d, a display control unit 34d, a judgment unit 35d, a creation unit 36d, a calculation unit 37d, a communication unit 38d, a memory / readout unit 39d, and a memory unit 3000d, but because these units respectively realize the same functions as the transmitting / receiving unit 31a, the receiving unit 32a, the image / sound processing unit 33a, the display control unit 34a, the judgment unit 35a, the creation unit 36a, the calculation unit 37a, the communication unit 38a, the memory / readout unit 39a, and the memory unit 3000a in the video conference terminal 3a, their descriptions will be omitted. In addition, the memory unit 3000d in the video conference terminal 3d has an image type management DB 3001d, a photographing device management DB 3002d, and a specified area management DB 3003d constructed therein. However, since these have the same data structures as the image type management DB 3001a, the photographing device management DB 3002a, and the specified area management DB 3003a in the video conference terminal 3a, their explanations will be omitted.

[0074] (Image type management table) FIG. 16 is a conceptual diagram showing an image type management table. In this image type management table, image data IDs, IP addresses (which are an example of destinations of source terminals), and source names are stored and managed in association with each other. Of these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is attached to image data transmitted from the same source terminal. This allows the destination terminal (receiving communication terminal) to identify the source terminal of the received image data. The IP address of the source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for identifying the imaging device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal, such as the videoconference terminal 3a, in accordance with a predetermined naming convention.

[0075] For example, it is shown that four communication terminals with IP addresses "1.2.1.3," "1.2.2.3," "1.3.1.3," and "1.3.2.3" are respectively transmitting image data indicated by image data IDs "RS001," "RS002," "RS003," and "RS004." Furthermore, the image types indicated by the source names of each communication terminal are "Video_Theta," "Video_Theta," "Video," and "Video," which indicate the image types as "special image," "special image," "general image," and "general image," respectively. Note that the special image here is a spherical panoramic image.

[0076] It should be noted that data other than image data may also be managed in association with an image data ID. Data other than image data may be, for example, sound data or material data used when sharing a screen. It should be noted that data other than image data may also be managed in association with an image data ID. Data other than image data may be, for example, sound data or material data used when sharing a screen.

[0077] (Radiation equipment management table) 17 is a conceptual diagram showing an image capturing device management table. In this image capturing device management table, the vendor ID and product ID of the GUID of an image capturing device that can obtain two hemispherical images that are the basis of a celestial sphere panoramic image are stored and managed. As the GUID, for example, a vendor ID (VID) and product ID (PID) used in a USB device can be used. The vendor ID and product ID are stored in a communication terminal such as a videoconferencing terminal when it is shipped from the factory, but may also be stored after shipping.

[0078] (Predetermined area management table) FIG. 18 is a conceptual diagram showing a predetermined area management table. In this predetermined area management table, the IP address of the communication terminal that is the sender of the captured image data, the IP address of the communication terminal that is the destination of the captured image data, and predetermined area information indicating the predetermined area image currently being displayed on the communication terminal that is the destination of the captured image data are stored and managed in association with each other. The communication terminal that is the destination of the captured image data is also the communication terminal that is the sender of the predetermined area information. As shown in FIGS. 6 and 7, the predetermined area information is a conversion parameter for converting a captured image into an image of a predetermined area T in the captured image (predetermined area image). The IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for identifying the communication terminal, and the like. Here, the IP address is expressed as a simplified version of an IPv4 address. The IP address may also be an IPv6 address.

[0079] For example, in the predetermined area management table of FIG. 18, when the IP address of the video conference terminal 3a is "1.2.1.3," the first to third rows of the table indicate that the captured image data transmitted from the video conference terminal 3a is transmitted via the communication management system 5 to the video conference terminal 3d having the IP address "1.2.2.3," the PC 7 having the IP address "1.3.1.3," and the smartphone 9 having the IP address "1.3.2.3." Furthermore, the video conference terminal 3d is also identified as the source communication terminal of the predetermined area information (r=10, θ=20, φ=30). Similarly, the PC 7 is also identified as the source communication terminal of the predetermined area information (r=10, θ=30, φ=40). Furthermore, the smartphone 9 is also identified as the source communication terminal of the predetermined area information (r=30, θ=40, φ=50).

[0080] In addition, when the transmitting / receiving unit 31a receives new specified area information that includes the same set of IP addresses as the IP address of the communication terminal that sent the captured image data and the IP address of the communication terminal that sent the captured image data, which are already managed, the storage / reading unit 39a rewrites the specified area information that is already managed with the newly received specified area information.

[0081] (Functional configuration of the video conference terminal 3a) Next, each functional configuration of the video conference terminal 3a will be described in more detail with reference to FIGS.

[0082] The transmission / reception unit 31a of the video conference terminal 3a is mainly realized by the processing of the network I / F 311 and CPU 301 shown in Figure 11, and transmits and receives various data (or information) to and from the communication management system 5 via the communication network 100.

[0083] The reception unit 32a receives various selections or inputs from the user, and is realized mainly by the operation buttons 308 and processing by the CPU 301. In addition to the operation buttons 308, a touch panel or the like may be used as another input means.

[0084] 11, and performs image processing on image data obtained by capturing an image of a subject with camera 312. After the user's voice is converted into an audio signal by microphone 314, image / audio processing unit 33a performs audio processing on the audio data related to this audio signal.

[0085] Furthermore, the image / sound processing unit 33a performs image processing on image data received from another communication terminal based on image type information such as a source name, so that the display control unit 34 can display an image on the display 4. Specifically, when the image type information indicates that the image is a special image, the image / sound processing unit 33a creates omnidirectional panoramic image data by converting the image data (for example, data of each hemispherical image shown in FIGS. 3(a) and 3(b)) into omnidirectional panoramic image data as shown in FIG. 4(b), and further creates a predetermined area image as shown in FIG. 6(b). In addition, the image / sound processing unit 33a outputs an audio signal related to audio data received from another communication terminal via the communication management system 5 to the speaker 315, and causes the speaker 315 to output audio.

[0086] The display control unit 34a is realized mainly by the processing of the display I / F 317 and the CPU 301, and performs control for displaying various images, characters, and the like on the display 4.

[0087] The determination unit 35a is realized mainly by the processing of the CPU 301, and determines the image type associated with the image data received from the photographing device 1a, for example.

[0088] The creation unit 36a is mainly realized by the processing of the CPU 301, and creates a source name, which is an example of image type information, in accordance with the naming rules described above based on the determination result of the determination unit 35a as being a general image or a special image (here, a spherical panoramic image). For example, if the determination unit 35a determines that the image is a general image, the creation unit 36a creates a source name "Video" indicating that it is a general image. On the other hand, if the determination unit 35a determines that the image is a special image, the creation unit 36a creates a source name "Video_Theta" indicating that it is a special image.

[0089] The calculation unit 37a is mainly realized by the processing of the CPU 301, and calculates the position (position information) of the predetermined area T1 relative to the predetermined area T2 in the captured image based on the predetermined area information (i2) indicating the predetermined area T2 and the predetermined area information (i1) received from another communication terminal by the transmission / reception unit 31a. Note that the predetermined area information (i1) indicates the predetermined area T1 in the captured image. An image in which the entire captured image is displayed is also referred to as an "entire image."

[0090] The communication unit 38a is mainly realized by the short-range communication circuit 319, the antenna 318a, and processing by the CPU 301, and can communicate with the communication unit 18a of the photographing device 1a using short-range wireless technology such as NFC, Bluetooth, WiFi, etc. Although the communication unit 38a and the transceiver unit 31a have been described as having separate communication units, they may also share a common communication unit.

[0091] The storage / reading unit 39a is mainly realized by the processing of the CPU 301 shown in FIG. 11, and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000.

[0092] <Functional configuration of communication management system> Next, each functional configuration of the communication management system 5 will be described in detail with reference to Fig. 12 and Fig. 15. The communication management system 5 has a transmitting / receiving unit 51, a determining unit 55, a generating unit 56, and a storing / reading unit 59. Each of these units is a function or means realized when any of the components shown in Fig. 12 operates in response to an instruction from the CPU 501 in accordance with the program for the communication management system 5 loaded from the HD 504 onto the RAM 503.

[0093] The communication management system 5 also has a storage unit 5000 constructed by the RAM 503 and HD 504 shown in Fig. 12. A session management DB 5001, an image type management DB 5002, and a predetermined area management DB 5003 are constructed in this storage unit 5000. Of these, the session management DB 5001 is constructed by the session management table shown in Fig. 19. The image type management DB 5002 is constructed by the image type management table shown in Fig. 20. The predetermined area management DB 5003 is constructed by the predetermined area management table shown in Fig. 21.

[0094] (Session management table) FIG. 19 is a conceptual diagram showing a session management table. In this session management table, session IDs and IP addresses of participating communication terminals are stored and managed in association with each other. Of these, the session ID is an example of session identification information for identifying a communication session that realizes a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal, such as the video conference terminal 3a, and is used when selecting a communication session in each communication terminal. The IP addresses of participating communication terminals indicate the IP addresses of communication terminals that have participated in the virtual conference room identified by the associated session ID.

[0095] (Image type management table) FIG. 20 is a conceptual diagram illustrating an image type management table. In addition to the various pieces of information managed in the image type management table shown in FIG. 16, the image type management table shown in FIG. 20 associates and manages session IDs identical to those managed in the session management table. Here, three communication terminals with IP addresses "1.2.1.3," "1.2.2.3," and "1.3.1.3" are participating in a virtual conference room identified by the same session ID "se101." The communication management system 5 manages the same image data ID, source terminal IP address, and image type information as those managed by a communication terminal, such as the video conference terminal 3a, in order to transmit image type information, etc., to communication terminals already in a video call and the newly participating communication terminal when a new communication terminal joins the virtual conference room. This eliminates the need to transmit image type information, etc., between a communication terminal already in a video call and the newly participating communication terminal.

[0096] (Predetermined area information management table) Fig. 21 is a conceptual diagram showing a predetermined area management table. This predetermined area management table basically has the same data structure as the predetermined area management table shown in Fig. 18. However, as will be described later, the transmitting / receiving unit 51 transmits the latest predetermined area information to each communication terminal at regular intervals (e.g., 30 seconds), so all predetermined area information received by the transmitting / receiving unit 51 is saved without being deleted until the predetermined area information is transmitted at regular intervals. In Fig. 21, newer predetermined area information is managed at a higher level.

[0097] (Functional configuration of communication management system) Next, each functional configuration of the communication management system 5 will be described in detail with reference to FIGS.

[0098] The transmission / reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I / F 509 and CPU 501 shown in FIG. 12, and transmits and receives various data (or information) with the video conference terminals 3a, 3d, or PC 7 via the communication network 100.

[0099] The determination unit 55 is mainly realized by the processing of the CPU 501 and makes various determinations.

[0100] The generation unit 56 is mainly realized by the processing of the CPU 501 and generates an image data ID.

[0101] The storage / reading unit 59 is mainly realized by the HDD 505 shown in FIG. 12 and the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 or reads out various data (or information) from the storage unit 5000.

[0102] <Functional Configuration of PC> Next, the functional configuration of the PC 7 will be described in detail using FIGS. 12 and 14. The PC 7 basically has the same functions as the video conference terminal 3a. That is, as shown in FIG. 14, the PC 7 has a transmission / reception unit 71, a reception unit 72, an image / audio processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a calculation unit 77, a communication unit 78, and a storage / reading unit 79. Each of these units is a function or means realized by any of the components shown in FIG. 12 operating according to an instruction from the CPU 501 in accordance with the program for the PC 7 developed from the HD 50 (should this be 504?) onto the RAM 503.

[0103] Also, the PC 7 has a storage unit 7000 constructed by the ROM 502, RAM 503, and HD 504 shown in FIG. 12. In this storage unit 7000, an image type management DB 7001, a photographing device management DB 7002, and a predetermined area management DB 7003 are constructed. Since the image type management DB 7001, the photographing device management DB 7002, and the predetermined area management DB 7003 have the same data structure as the image type management DB 3001a, the photographing device management DB 3002a, and the predetermined area management DB 3003a, respectively, the descriptions thereof are omitted.

[0104] (Functional Configurations of PC) The transmitting / receiving unit 71 of the PC 7 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 12, and realizes the same function as the transmitting / receiving unit 31a.

[0105] The reception unit 72 is mainly realized by processing by the keyboard 511, the mouse 512, and the CPU 501, and realizes the same function as the reception unit 32a. The image / sound processing unit 73 is mainly realized by instructions from the CPU 501, and realizes the same function as the image / sound processing unit 33a. The display control unit 74 is mainly realized by processing by the CPU 501, and realizes the same function as the display control unit 34a. The determination unit 75 is mainly realized by processing by the CPU 501, and realizes the same function as the determination unit 35a. The creation unit 76 is mainly realized by processing by the CPU 501, and realizes the same function as the creation unit 36a. The calculation unit 77 is mainly realized by processing by the CPU 501, and realizes the same function as the calculation unit 37a. The communication unit 78 is mainly realized by processing by the CPU 501, and realizes the same function as the communication unit 38a. The storage / readout unit 79 a is realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 7000 and reads out various data (or information) from the storage unit 7000 .

[0106] <Smartphone functional configuration> Next, the functional configuration of the smartphone 9 will be described in detail with reference to Figures 13 and 14. The smartphone 9 basically has the same functions as the video conference terminal 3a. That is, as shown in Figure 14, the smartphone 9 has a transmission / reception unit 91, a reception unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a calculation unit 97, a communication unit 98, and a storage / readout unit 99. Each of these units is a function or means realized when any of the components shown in Figure 13 operates in response to an instruction from the CPU 901 in accordance with a program for the smartphone 9 loaded from the EEPROM 904 onto the RAM 903.

[0107] 13. An image type management DB 9001, an imaging device management DB 9002, and a predetermined area management DB 9003 are configured in the storage unit 9000. Note that the image type management DB 9001, the imaging device management DB 9002, and the predetermined area management DB 9003 have the same data configurations as the image type management DB 3001a, the imaging device management DB 3002a, and the predetermined area management DB 3003a, respectively, and therefore descriptions thereof will be omitted.

[0108] (Smartphone functional configuration) The transmitter / receiver 91 of the smartphone 9 is mainly realized by the processing of the long-distance communication circuit 911 and the CPU 901 shown in FIG. 13, and realizes the same functions as the transmitter / receiver 31a.

[0109] The reception unit 92 is realized mainly by the processing of the touch panel 921 and the CPU 901, and realizes the same functions as the reception unit 32a.

[0110] The image / sound processing unit 93 is mainly implemented by instructions from the CPU 901, and realizes the same functions as the image / sound processing unit 33a. The display control unit 94 is mainly implemented by processing by the CPU 901, and realizes the same functions as the display control unit 34a. The determination unit 95 is mainly implemented by processing by the CPU 901, and realizes the same functions as the determination unit 35a. The creation unit 96 is mainly implemented by processing by the CPU 901, and realizes the same functions as the creation unit 36a. The calculation unit 97 is mainly implemented by processing by the CPU 901, and realizes the same functions as the calculation unit 37a. The communication unit 98 is mainly implemented by processing by the CPU 901, and realizes the same functions as the communication unit 38a. The storage / readout unit 99 is mainly implemented by processing by the CPU 901, and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000.

[0111] <<Processing or Operation of the Embodiment>> Next, the processing or operation of this embodiment will be described with reference to FIGS.

[0112] <Participation process> First, the process of joining a specific communication session will be described with reference to Fig. 22 and Fig. 23. Fig. 22 is a sequence diagram showing the process of joining a specific communication session. Fig. 23 is a diagram showing a selection screen for a communication session (virtual conference room).

[0113] First, when a user at site A (e.g., user A1) operates the videoconference terminal 3a to display a selection screen for a communication session (virtual conference room), the reception unit 32a accepts the operation to display the selection screen, and the display control unit 34a displays the selection screen as shown in Fig. 23 on the display 4a (step S21). This selection screen displays selection buttons b1, b2, b3, etc., which indicate the virtual conference rooms R1, R2, R3, etc., to be selected. Each selection button b1, etc., is associated with a session ID.

[0114] When user A1 selects a desired selection button (here, selection button b1) in the virtual conference room, the reception unit 32a receives the selection of the communication session (step S22). Then, the transmission / reception unit 31a transmits a request to join the virtual conference room to the communication management system 5 (step S23). This participation request includes a session ID indicating the communication session whose selection was received in step S22 and the IP address of the video conference terminal 3a, which is the requesting terminal. As a result, the transmission / reception unit 51 of the communication management system 5 receives the participation request.

[0115] Next, the storage / readout unit 99 performs a process of joining the communication session by adding the IP address received in step S23 to the joining terminal IP address field of the record in the session management DB 5001 (see FIG. 19) that has the same session ID as the session ID received in step S23 (step S24). Then, the transmission / reception unit 51 transmits a join request response to the video conference terminal 3a (step S25). This join request response includes the session ID received in step S23 and the result of the join processing. As a result, the transmission / reception unit 31a of the video conference terminal 3a receives the join request response. The following describes the case where the join processing is successful.

[0116] <Image type information management process> Next, the management process of image type information will be described with reference to Fig. 24. Fig. 24 is a sequence diagram showing the management process of image type information.

[0117] First, when a user at site A (e.g., user A1) connects the USB cable of the cradle 2a with the image capturing device 1a attached to the video conference terminal 3a, the memory / readout unit 19a of the image capturing device 1a reads the GUID of the image capturing device 1a stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of the image capturing device 1a to the communication unit 38a of the video conference terminal 3a (step S51). As a result, the communication unit 38a of the video conference terminal 3a receives the GUID of the image capturing device 1a.

[0118] Next, the determination unit 35a of the video conference terminal 3a determines the image type by determining whether the same vendor ID and product ID as those in the GUID received in step S51 are managed in the camera device management DB 3002a (see FIG. 17) (step S52). Specifically, if the same vendor ID and product ID are managed in the camera device management DB 3002a, the determination unit 35a determines that the camera device 1a is a camera device that captures special images (here, omnidirectional panoramic images). On the other hand, if the same vendor ID and product ID are not managed in the camera device management DB 3002a, the determination unit 35a determines that the camera device 1a is a camera device that captures general images.

[0119] Next, the storage / readout unit 39a associates the IP address of the terminal (video conference terminal 3a) that is the source terminal with the image type information determined in step S52 and stores them in the image type management DB 3001a (see FIG. 16) (step S53). In this state, no image data ID is associated. The image type information is, for example, a source name determined according to a predetermined naming rule and an image type (general image, special image).

[0120] Next, the transmitting / receiving unit 31a transmits a request to add image type information to the communication management system 5 (step S54). This request to add image type information includes the IP address of the terminal itself, which is the transmission source terminal stored in step S53, and the image type information. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request to add image type information.

[0121] Next, the storage / reading unit 59 of the communication management system 5 searches the session management DB 5001 (see FIG. 19) using the IP address of the source terminal received in step S54 as a search key, and reads out the corresponding session ID (step S55).

[0122] Next, the generation unit 56 generates a unique image data ID (step S56). Then, the storage / reading unit 59 associates the session ID read out in step S55, the image data ID generated in step S56, and the IP address of the transmission source terminal and image type information received in step S54 and stores them as a new record in the image type management DB 5002 (see FIG. 20) (step S57). Then, the transmission / reception unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3a. As a result, the transmission / reception unit 31a of the video conference terminal 3a receives the image data ID (step S58).

[0123] Next, the storage / readout unit 39a of the video conference terminal 3a stores the image data ID received in step S58 in the image type management DB 3001a (see FIG. 16) in association with the IP address and image type information of the terminal (video conference terminal 3a) that is the source terminal stored in step S53 (step S59).

[0124] Meanwhile, the transmitter / receiver 51 of the communication management system 5 transmits an addition notification of the image type information to another communication terminal (here, the smartphone 9) (step S60). This addition notification of the image type information includes the image data ID generated in step S56, as well as the IP address and image type information of the terminal (video conference terminal 3a) that is the sender terminal stored in step S53. As a result, the transmitter / receiver 91 of the smartphone 9 receives the addition notification of the image type information. Note that the destination of the transmitter / receiver 51 is another IP address that is associated in the session management DB 5001 (see FIG. 19) with the same session ID as the IP address of the video conference terminal 3a. In other words, the destination is another communication terminal that is in the same virtual conference room as the video conference terminal 3a.

[0125] Next, the storage / readout unit 99 of the smartphone 9 associates the image data ID, the IP address of the sending terminal, and the image type information received in step S60 and stores them as a new record in the image type management DB 9001 (see FIG. 16) (step S61). Similarly, an additional communication of the image type information is sent to the other communication terminals, the video conference terminal 3d and the PC 7, and the image type information is also stored in the image type management DBs 3001d and 7001 of the video conference terminals 3d and PC 7, respectively. As described above, the same information can be shared among the communication terminals in the image type management DBs 3001a, 3001d, 7001, and 9001.

[0126] <Communication processing of captured image data> Next, the communication process of captured image data during a video call will be described with reference to Fig. 25 to Fig. 34. Fig. 25 is an image diagram showing the state of a video call. Of these, Fig. 25(a) shows a case where the image capturing device 1a is not used, and Fig. 25(b) shows a case where the image capturing device 1a is used.

[0127] First, as shown in FIG. 25(a), if the camera 312 (see FIG. 11) pre-installed in the video conference terminal 3a is used without using the image capture device 1a, the angle of view is 125 degrees horizontally and 70 degrees vertically, so the video conference terminal 3a must be placed at the edge of the desk where each user A1, etc. can be seen. This means that each user A1, etc. must face the video conference terminal 3a when speaking. Furthermore, because each user A1, etc. faces the video conference terminal 3a, the display 4a must also be placed next to the video conference terminal 3a. As a result, users A2 and A4, who are far from the video conference terminal 3a, are far from the microphone 314 (see FIG. 11), so they must speak relatively loudly and have difficulty seeing the content displayed on the display 4a.

[0128] 25(b), when the image capturing device 1a is used, two hemispherical images that are the basis for the omnidirectional panoramic image can be obtained, and therefore the video conference terminal 3a and the display 4a can be placed relatively close to the center of the desk. This allows each user A1, etc. to be close to the microphone 314, so they can speak in a relatively low voice and the contents displayed on the display 4a can be easily seen. A whiteboard 6 is installed on the right side of the site A, on which the user A1, etc. can write text, draw pictures, etc.

[0129] Next, with reference to Fig. 26, a process will be described in which the captured image data and sound data obtained at site A shown in Fig. 25(b) are transmitted to other communication terminals (smartphone 9, PC 7, video conference terminal 3d) via the communication management system 5. Fig. 26 is a sequence diagram showing the communication process of captured image data and sound data during a video call.

[0130] First, the communication unit 18a of the image capturing device 1a transmits captured image data obtained by capturing an image of a subject, scenery, or the like, and audio data obtained by collecting sound to the communication unit 38a of the video conference terminal 3a (step S101). In this case, since the image capturing device 1a is a device capable of obtaining two hemispherical images that are the basis for a celestial sphere panoramic image, the captured image data is composed of data for the two hemispherical images, as shown in Figures 3(a) and 3(b). As a result, the communication unit 38a of the video conference terminal 3a receives the captured image data and audio data.

[0131] Next, the transmitting / receiving unit 31a of the video conference terminal 3a transmits the captured image data and sound data sent from the image capturing device 1a to the communication management system 5 (step S102). This transmission includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the captured image data and image data ID.

[0132] Next, the transmitter / receiver 51 of the communication management system 5 transmits the captured image data and audio data to the communication terminals (smartphone 9, PC 7, and videoconference terminal 3d) participating in the same video call as the videoconference terminal 3a (steps S103, S104, and S105). Each of these transmissions includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmitter / receiver 91 of the smartphone 9, the transmitter / receiver 71 of the PC 7, and the transmitter / receiver 31d of the videoconference terminal 3d each receive the captured image data, image data ID, and audio data.

[0133] Next, a display example of the display 917 at the site B will be described with reference to FIG. 27. FIG. 27 shows a display example of the display at the site B. Of these, FIG. 27(a) shows a case where the captured image data transmitted from the camera 1a at the site A via the video conference terminal 3a and the captured image data transmitted from the camera 1b at the site B are displayed as is without creating a celestial sphere panoramic image and a predetermined area image. On the other hand, FIG. 27(b) shows a case where the celestial sphere panoramic image and a predetermined area image are created from the captured image data transmitted from the camera devices 1a and 1b. Note that an image of site A is displayed in the left display area (layout number "1") of the display 4d, and an image of site B is displayed in the upper right display area (layout number "2"). Furthermore, an image of site C is displayed in the middle right display area (layout number "3") of the display 4d, and an image of site D (the user's site) is displayed in the lower right display area (layout number "4"). The display area for layout number "1" is the main display area, and the display areas for layout numbers "2," "3," and "4" are sub-display areas. The images in the main display area and the sub-display area can be changed on each communication terminal. Normally, at each location, the image of the location where the central person in the video call is located is displayed in the main display area.

[0134] Here, when the captured image data sent from the photographing devices 1a and 1b capable of photographing a celestial sphere panoramic image is displayed as it is, as shown in FIG. 27(a), the images of the sites A and B are displayed as a front hemispherical image and a rear hemispherical image, as shown in FIGS. 3(a) and 3(b), respectively.

[0135] In contrast, when the image / sound processing unit 93 creates a celestial sphere panoramic image from the captured image data output by the photographing devices 1a and 1b that can obtain two hemispherical images that are the basis of the celestial sphere panoramic image, and further creates a predetermined area image, the predetermined area image, which is a planar image, is displayed as shown in Fig. 27(b). Note that, at site C, photographing is performed by the photographing device 8 that obtains a general image, and at site D, photographing is performed by the video conference terminal 3d that obtains a general image, so general images (here, planar images) are displayed in both Figs. 27(a) and (b).

[0136] Furthermore, users at each location can change the predetermined area associated with the predetermined area image in the same omnidirectional panoramic image. For example, user B1 operates the touch panel 921, etc., so that the reception unit 92 receives a request to move the predetermined area image, and the display control unit 94 can shift, rotate, reduce, or enlarge the predetermined area image. As a result, the predetermined area image, which displays users A1 and A2 at location A as the default setting as shown in FIG. 28(b), can be changed to display the predetermined area image as shown in FIG. 28(c). Specifically, FIG. 28(c) illustrates a state in which the predetermined area image including users A1 and A2 in the image captured at location A shown in FIG. 25(b) has been changed to a predetermined area image including a whiteboard 6.

[0137] 27(b) and 27(c) are examples of special image identification icons for indicating that the celestial sphere icons 191 and 192 are predetermined area images that indicate a predetermined area T that is a part of a celestial sphere panoramic image. The display positions of the celestial sphere icons 191 and 192 do not have to be the upper right, but may be anywhere, such as the upper left, lower left, or lower right. The types of the celestial sphere icons 191 and 192 are not limited to those shown in FIGS. 27(b) and 27(c). Instead of the celestial sphere icons 191 and 192, characters such as "celestial sphere image" may be used, or a combination of an icon and characters may be used.

[0138] Next, using Fig. 28, we will explain the processing in the image communication system when the predetermined area image is displayed as in Fig. 27(b) and when the predetermined area image is changed from Fig. 27(b) to Fig. 27(c). Fig. 28 is a sequence diagram showing the processing for sharing predetermined area information. In Fig. 28, the video conference terminal 3a at site A is the third communication terminal, the video conference terminal 3d at site D is the other communication terminal, and the smartphone 9 at site B is the communication terminal (own terminal).

[0139] 27(b), when a user D1 or the like at site D uses a video conference terminal 3d to display a predetermined area image of site A, the transmitter / receiver 31d of the video conference terminal 3d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S111). This predetermined area information includes the IP address of the video conference terminal 3a that is the sender of the captured image data, and the IP address of the video conference terminal 3d that is the destination of the captured image data (the sender of the predetermined area information). As a result, the transmitter / receiver 51 of the communication management system 5 receives the predetermined area information.

[0140] Next, the storage / readout unit 59 of the communication management system 5 associates the predetermined area information received in step S111 with the source and destination IP addresses and stores them in the predetermined area management DB 5003 (step S112). Note that the processes of steps S111 and S112 are performed every time the predetermined area image is changed on the video conference terminal 3d, as shown in Figures 27(b) to 27(c).

[0141] Next, the storage / read unit 59 of the communication management system 5 reads out the most recent pair of predetermined area information and IP addresses (the pair stored most recently) at that time from among the pairs of predetermined area information and IP addresses stored in the predetermined area management DB 5003 at regular intervals (e.g., 30 seconds) (step S113). Then, the transmission / reception unit 51 distributes (transmits) the predetermined area information including the IP addresses read out in step S113 to other communication terminals (video conference terminal 3a, smartphone 9, PC 7) that are performing the same video call as the video conference terminal 3d that sent the predetermined area information (steps S114, S116, S118). As a result, the transmission / reception unit 31a of the video conference terminal 3a receives the predetermined area information. Then, the storage / read unit 39a stores the predetermined area information and the IP addresses received in step S114 in the predetermined area management DB 3003a while associating them with each other (step S115). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives the predetermined area information, the storage / reading unit 99 stores the predetermined area information received in step S116 in association with each IP address in the predetermined area management DB 9003 (step S117). Furthermore, in the PC 7, after the transmitting / receiving unit 71 receives the predetermined area information, the storage / reading unit 79 stores the predetermined area information received in step S118 in association with each IP address also received in the predetermined area management DB 7003 (step S119).

[0142] As described above, the predetermined area information indicating the predetermined area image changed at location A is transmitted to each communication terminal at other locations B, C, and D that are participating in the same video call, and the predetermined area information indicating the predetermined area image being displayed at location A is also shared at other locations B, C, and D. This process is also performed in the same way when the predetermined area image is changed at locations B, C, and D, and the predetermined area information indicating the predetermined area image being displayed at other locations is shared among all locations that are participating in the same video call.

[0143] Next, a method for using the predetermined area information shared among the locations will be described with reference to Figures 29 to 34. Figure 29 is a flowchart showing the display process of a predetermined area image. Note that the process is the same for each communication terminal, so the process of the smartphone 9 at location B will be described here. Specifically, when captured image data transmitted from the video conference terminal 3a at location A is displayed as a predetermined area image by the video conference terminal 3d at location D, the video conference terminal 3d transmits predetermined area information indicating the predetermined area image to other communication terminals participating in the same video call, and the process executed by the smartphone 9 at location B will be described.

[0144] First, in the smartphone 9, the storage / reading unit 99 searches the image type management DB 9001 (see FIG. 16) using the image data ID received in step S103 shown in FIG. 26 as a search key, thereby reading out the corresponding image type information (source name) (step S131).

[0145] Next, the determination unit 95 determines whether the image type information read out in step S131 indicates a "special image" (step S132). If the image type information indicates a "special image" (step S132; YES), the storage / reading unit 99 further searches the predetermined area management DB 9003 for predetermined area information indicating a predetermined area image displayed by a communication terminal at another base (step S133). Then, the determination unit 95 determines whether the predetermined area management DB 9003 manages predetermined area information indicating a predetermined area image displayed by a communication terminal at another base (step S134). If the predetermined area information indicating the predetermined area image displayed by the communication terminal at the other location is managed (step S134; YES), the calculation unit 97 calculates the position of the predetermined area T1 relative to the predetermined area T2 in the entire image based on the predetermined area information (i2) indicating the predetermined area image of the predetermined area T2 currently displayed by the smartphone 9 (own terminal) and the predetermined area information (i1) indicating the predetermined area image of the predetermined area T1 received from the other communication terminal by the transmission / reception unit 91 and managed in the predetermined area management DB 9003 (step S135). Strictly speaking, the position in this case indicates the point of gaze of the predetermined area T1 relative to the point of gaze of the predetermined area T2. The point of gaze is the center point as described above, but may also be the upper left corner (or the lower left corner, upper right corner, or lower right corner) of the rectangle of each predetermined area. Alternatively, the point of gaze may be a specific point within each predetermined area.

[0146] Here, a method for calculating the gaze point of a predetermined area T1 relative to a predetermined area T2 in an entire image will be described using Fig. 30. Fig. 30(a) is a diagram showing the definition of the angle of the virtual camera, and Fig. 30(b) is a diagram showing a method for calculating the position of the gaze point of another base in the predetermined area image of the own base by parallel projection as viewed from above.

[0147] 30(a), the calculation unit 97 acquires the radius vector r, the polar angle θ, and the azimuth angle φ from the predetermined area information indicating the predetermined area image being displayed by the display control unit 94 of the own terminal (smartphone 9), and sets this as CP1(r0, θ1, φ1). Next, the calculation unit 97 acquires the radius vector r, the polar angle θ, and the azimuth angle φ from the predetermined area information of the other base read in step S133, and sets this as CP2(r0, θ2, φ2).

[0148] Considering a predetermined area T2 centered on the gaze point CP1 of the own terminal (smartphone 9), the width w and height h of the predetermined area T2 are projected as w in Figure 30(b), which is a parallel projection from the polar direction, and the height is projected as a length of hcosθ1.

[0149] In addition, the radius of gaze point CP1 is projected onto the length of r0 sinθ1, and the radius of gaze point CP2 is projected onto the length of r0 sinθ2, so gaze point CP1 is located at the coordinates (r0 sinθ1 · r0 cosφ1, r0 sinθ1 · r0 sinφ1), and gaze point CP2 is located at the coordinates (r0 sinθ2 · r0 cosφ2, r0 sinθ2 · r0 cosφ2).

[0150] As a result of the above, in Figure 30(b), the coordinates of the gaze points CP1 and CP2 can be derived, and the position of the gaze point CP2 on the plane of the predetermined area T2 with width w and height h can be derived using general coordinate transformation.

[0151] Here, a method for calculating the direction of a predetermined area T1 relative to a predetermined area T2 in an entire image will be described with reference to Fig. 31. Fig. 31(a) is a diagram showing the definition of an angle, and Fig. 31(b) is a diagram showing the definition of an angle range.

[0152] 31(a), the calculation unit 97 acquires the azimuth angle φ from the predetermined area information indicating the predetermined area image being displayed by the display control unit 94 of the own terminal (smartphone 9), and sets this as the rotation angle φ1. Next, the calculation unit 97 acquires the azimuth angle φ from the predetermined area information of the other location read out in step S133, and sets this as the rotation angle φ2. Furthermore, the calculation unit 97 sets the difference between the rotation angle φ2 and the rotation angle φ1 as the rotation angle φ3.

[0153] Furthermore, as shown in FIG. 31(b), if the angle range centered on the rotation angle φ of the own base is defined as α1 and the angle range centered on an angle obtained by adding 180 degrees to the horizontal angle of the own base is defined as α2, the calculation unit 97 calculates the direction of the specified area T1 relative to the specified area T2 in the overall image as follows: (1) If the rotation angle φ3 is included in the angle range α1, the positional relationship is determined to be “forward.” (2) If φ3 is included in the angle range α2, the positional relationship is determined to be “backward.” (3) If φ3 is not included in the angle range α1 or the angle range α2 and is greater than 0 degrees and less than 180 degrees, the positional relationship is determined to be "rightward." (4) If φ3 is not included in the angle range α1 or the angle range α2 and is greater than 180 degrees and less than 360 degrees, the positional relationship is determined to be "leftward."

[0154] Next, the image / sound processing unit 93 creates a predetermined area image including a fixation point mark indicating the fixation point calculated by the calculation unit 97 and a display direction mark indicating the direction (step S136). The display position of the fixation point mark is directly determined from the position of the predetermined area T1 relative to the predetermined area T2 in the entire image. The display position of the display direction mark is determined by the above-mentioned determination processes (1) to (5) using the position of the predetermined area T1 relative to the predetermined area T2 in the entire image.

[0155] At this time, the image / sound processing unit 93 combines celestial sphere icons 191 and 192 indicating that the image is a celestial sphere panoramic image with the predetermined area image based on the fact that the image type information indicates a "special image." Then, the display control unit 94 displays the predetermined area image created in step S136 (step S137) as shown in FIGS. 32(a), (b), and (c), 33(a), (b), and 34. Note that FIGS. 32(a), (b), and (c) show images of the main display area, illustrating three display examples of the predetermined area image including a display direction mark. Note that FIGS. 33(a) and (b) show images of the main display area, illustrating two display examples of the predetermined area image including a gaze point mark. Note that FIG. 34 shows an image of the main display area, illustrating one display example of the predetermined area image including a gaze point mark and a display direction mark. As shown in FIG. 27, images of all locations during the video call are displayed, but in FIGS. 32, 33, and 34, only the image of location A is displayed due to the limited area of ​​the drawings.

[0156] As shown in Fig. 32(a), in the predetermined area image which is a part of the image at site A, display direction marks m11, m13, and m14 are displayed in the entire image, indicating the direction of the predetermined area image displayed at other sites with reference to the predetermined area image being displayed at site B (own site). Note that the display direction marks m21, m23, and m24 shown in Fig. 32(b) and the display direction marks m31, m33, and m34 shown in Fig. 32(c) correspond to the display direction marks m11, m13, and m14 shown in Fig. 32(a), respectively.

[0157] The display direction mark is an example of directional information, and other forms of directional information may be used. Furthermore, the directional information may not be indicated by an arrow, but may be indicated by characters such as "right," "left," "rear," and "front."

[0158] 33(a), in the predetermined area image, which is a part of the image at site A, point of interest marks m41 and m42 are displayed in the entire image, indicating the point of interest of the predetermined area image displayed at another site, with the predetermined area image displayed at site B (the site itself) as the reference. The point of interest marks may be displayed semi-transparently so as not to obscure the predetermined area image. Note that the display direction marks m51 and m52 shown in FIG. 33(b) correspond to the display direction marks m41 and m42 shown in FIG. 33(a), respectively.

[0159] In FIG. 33(a), a point of gaze mark m41 showing the location name "C" and a point of gaze mark m42 showing the location name "D" are displayed so that it is possible to identify which location the point of gaze is. In contrast, in FIG. 33(b), the location names are not displayed, and different locations are indicated by point of gaze marks with different patterns. In this case, if a table that associates patterns with location names is prepared at each location, users at each location can identify the location from the pattern of the point of gaze mark. The table may be printed on paper or stored as electronic data at each location.

[0160] Instead of using a pattern to distinguish the point of interest marks, the marks may be distinguished by changing the color or the type of line. The point of interest marks are an example of corresponding position information.

[0161] In FIG. 34, when the point of gaze of another location is included in the range of the predetermined area image, a display mark m41 indicating the point of gaze is displayed, and when it is not included in the range of the predetermined area image, display direction marks m11 and m14 are displayed.

[0162] 29, if it is determined in step S134 that the predetermined area management DB 9003 does not manage the predetermined area information indicating the predetermined area image displayed by the communication terminal at the other location (step S134; NO), the image / sound processing unit 93 creates the predetermined area image without including the point of interest mark and the display direction mark (step S138). After that, the process proceeds to step S137.

[0163] Furthermore, if it is determined in step S132 that the image type information does not indicate a "special image" (step S132; NO), that is, if the image type information indicates a "general image," the image / sound processing unit 93 does not create a spherical panoramic image from the captured image data received in step S103, and the display control unit 94 displays a general image (step S139).

[0164] As described above, users B1 and B2 at site B can grasp the positional relationship between the predetermined area image displayed at their own site and the predetermined area image displayed at the other site, thereby preventing users B1 and B2 at site B from being unable to keep up with the topics of a meeting, etc.

[0165] <<Main Effects of This Embodiment>> As described above, according to this embodiment, a communication terminal such as the video conference terminal 3a can create a celestial sphere panoramic image and further create a predetermined area image based on the image data ID transmitted together with the image data and corresponding image type information. This has the effect of preventing the front hemispherical image and the rear hemispherical image from being displayed in a separate image, as shown in Fig. 27(a).

[0166] Furthermore, a user at any location can grasp which part of the predetermined area image of the entire spherical panoramic image is being displayed at another location, which has the effect of making it easier to follow the topic of a meeting, etc., compared to conventional methods.

[0167] 28, if the communication management system 5 were to transfer the predetermined area information to other communication terminals every time it received it from the video conference terminal 3d, users B1 and B2 would not be able to concentrate on the video call due to the flickering of the gaze point marks and display direction marks shown in Fig. 34. Therefore, as shown in steps S112 to S114 above, the communication management system 5 periodically distributes the latest predetermined area image at that time and each IP address pair, allowing each user to concentrate on the video call.

[0168] Second Embodiment Next, a second embodiment will be described with reference to Fig. 35. Fig. 35 is a sequence diagram showing another example of the process shown in Fig. 28, illustrating another process for sharing predetermined area information. In Fig. 35, the video conference terminal 3a at site A is the communication terminal (own terminal), and the video conference terminal 3d at site D is the other communication terminal.

[0169] In the first embodiment, as shown in Fig. 28, the communication management system 5 temporarily manages each predetermined area information transmitted from each communication terminal (see S112) and periodically transmits each predetermined area information to each communication terminal other than the transmission source (see S114 to S119). In contrast, in the present embodiment, as shown in Fig. 35, the communication terminal (here, the video conference terminal 3a) that transmitted the captured image data temporarily manages the predetermined area information on behalf of the communication management system 5 (see S213) and periodically transmits each predetermined area information to each communication terminal other than the own terminal (see S215 to S221). That is, in the present embodiment, the communication terminal that transmitted the captured image data manages how the captured image data transmitted from the own terminal (here, the video conference terminal 3a) is displayed on the other communication terminals as a predetermined area image of the predetermined area T1.

[0170] This embodiment has the same configuration as the first embodiment, but differs in the processing shown in Fig. 28. Therefore, in the following, the same components are denoted by the same reference numerals, and the description is omitted, and processing that differs from the first embodiment will be described using Fig. 35.

[0171] First, when a user D1 or the like at site D uses the video conference terminal 3d to display a predetermined area image of site A, the transmitter / receiver 31d of the video conference terminal 3d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S211). This predetermined area information includes the IP address of the video conference terminal 3a that is the sender of the captured image data, and the IP address of the video conference terminal 3d that is the destination of the captured image data (the sender of the predetermined area information). As a result, the transmitter / receiver 51 of the communication management system 5 receives the predetermined area information.

[0172] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits the predetermined area information including each IP address received in step S211 to the video conference terminal 3a that is the sender of the captured image data (step S212). As a result, the transmitting / receiving unit 31a of the video conference terminal 3a receives the predetermined area information.

[0173] Next, the storage / readout unit 39a of the video conference terminal 3a associates the predetermined area information received in step S212 with the IP addresses of the sender and the recipient and stores them in the predetermined area management DB 3003a (step S213). The process of step S213 is a process for managing how the captured image data sent by the terminal itself (here, the video conference terminal 3a) is displayed on other communication terminals. The processes of steps S211 to S213 are performed every time the predetermined area image is changed on the video conference terminal 3d.

[0174] Next, the storage / readout unit 39a of the video conference terminal 3a reads out the latest pair of predetermined area information and each IP address at that time (the pair stored most recently) from among the pairs of predetermined area information and each IP address stored in the predetermined area management DB 3003a at regular intervals (e.g., 30 seconds) (step S214). Then, the transmission / reception unit 31a transmits the predetermined area information including each IP address read out in step S214 to the communication management system 5 (step S215). As a result, the transmission / reception unit 51 of the communication management system 5 receives the predetermined area information.

[0175] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits (distributes) the predetermined area information including each IP address received in step S215 to each communication terminal (videoconference terminal 3d, smartphone 9, PC 7) (steps S216, S218, S220). As a result, the transmitting / receiving unit 31d of the videoconference terminal 3d receives the predetermined area information. The storage / reading unit 39d then stores the predetermined area information received in step S216 in the predetermined area management DB 3003d in association with each IP address (step S217). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives the predetermined area information, the storage / reading unit 99 stores the predetermined area information received in step S218 in the predetermined area management DB 9003 in association with each IP address (step S219). Furthermore, in PC7, after the transmitting / receiving unit 71 receives the specified area information, the storage / reading unit 79 stores the specified area information received in step S220 in the specified area management DB 7003 in association with each IP address also received (step S221).

[0176] <<Main Effects of This Embodiment>> As described above, according to this embodiment, the communication terminal that has transmitted the captured image data collects, based on the captured image data transmitted from its own terminal, predetermined area information indicating how the captured image data is displayed on each communication terminal, and distributes the information to each communication terminal. This has the effect of preventing the burden on the communication management system 5 from being concentrated when the same conference or the like is held at a large number of communication terminals, in addition to the effect of the first embodiment.

[0177] Third Embodiment Next, a third embodiment will be described.

[0178] In FIG. 10, the image capture device 1a is equipped with one microphone 108, but in this embodiment, the image capture device 1a is equipped with multiple directional microphones. By using multiple directional microphones in the image capture device 1a, audio data from each directional microphone is transmitted from the image capture device 1a to the video conference terminal 3a. As a result, the calculation unit 37a of the video conference terminal 3a calculates the direction of the sound source (microphone angle) from the audio data of each microphone, and the calculated direction is used to identify the position of the speaker (who transmitted the captured image). The same is true for the image capture device 1b. The image capture device 1b, which is equipped with multiple directional microphones, transmits audio data from each microphone to the smartphone 9, and the calculation unit 97 of the smartphone 9 calculates the direction of the sound source (microphone angle) from the audio data of each microphone, and the calculated angle is used to identify the position of the speaker (who transmitted the captured image).

[0179] <<Main Effects of This Embodiment>> As a result, it is possible to more accurately display at another location which portion of the entire image of the omnidirectional panoramic image is being displayed as a predetermined region image.

[0180] 〔supplement〕 In the above embodiment, the predetermined region T is identified by predetermined region information indicating the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE, but this is not limiting. For example, when using a constant angle of view, the predetermined region T may be identified by predetermined point information indicating the center point CP or any of the four corner points of the rectangular predetermined region T in FIG. 7. Note that the "predetermined information" includes the predetermined region information and the predetermined information.

[0181] In the above embodiment, a three-dimensional celestial sphere panoramic image is described as an example of a panoramic image, but a two-dimensional panoramic image may also be used.

[0182] Furthermore, in the above embodiment, the communication management system 5 relays the predetermined area information transmitted from each communication terminal, but this is not limiting, and each communication terminal may transmit and receive the predetermined area information directly to each other.

[0183] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), a SOC (System on a Chip), a GPU (Graphics Processing Unit), and conventional circuit modules designed to execute each of the above-described functions. [Explanation of symbols]

[0184] 1a Imaging device 1b Imaging device 3a Video conference terminal (an example of a third communication terminal / an example of a communication terminal (own terminal)) 3D video conferencing terminal (an example of another communication terminal) 4a Display 4D display (an example of a display method) 5. Communications Management System 7 PC 8. Imaging Device 9 Smartphone (an example of a communication terminal (own terminal)) 31d Transmitter / receiver unit (an example of a transmitting means, an example of a receiving means) 37d Calculation unit (an example of a change means) 3001d Image type management DB (an example of image type management means) 3002d Imaging device management DB (an example of imaging device management means) 3003d Predetermined area management DB (an example of a predetermined area management means) 3004d Required resolution management DB (an example of required resolution management means) m11 Display direction mark (example of direction information) m14 Display direction mark (example of direction information) m41 Point of interest mark (example of corresponding position information) [Prior art documents] [Patent documents]

[0185] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-178135 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-223076

Claims

1. A communication terminal that displays a first region image indicating a first region in an entire image, and communicates with another communication terminal that displays a second region image indicating a second region in the entire image at another location, a receiving means for receiving second information for identifying the second area transmitted from the other communication terminal; a display control means for displaying the first region image including direction information indicating a direction of the second region relative to the first region in the entire image, based on first information and the second information for specifying the first region; A communication terminal having the above configuration.

2. When the second region is not included in the first region, the display control means displays the first region image including the direction information. The communication terminal according to claim 1 .

3. When there are a plurality of second regions, the display control means displays identification information corresponding to the second regions in association with the direction information.

3. The communication terminal according to claim 1 or 2.

4. The first area and the second area are changeable by a user operation. A communication terminal according to any one of claims 1 to 3.

5. The overall image is a spherical panoramic image. A communication terminal according to any one of claims 1 to 4.

6. A display method executed by a communication terminal that displays a first region image indicating a first region in an entire image and communicates with another communication terminal that displays a second region image indicating a second region in the entire image at another location, the method comprising: a receiving step of receiving second information for identifying the second area transmitted from the other communication terminal; a display control step of displaying the first region image including direction information indicating a direction of the second region relative to the first region in the entire image based on first information and the second information for identifying the first region; A display method having the following.

7. A program executed on a communication terminal that displays a first region image indicating a first region in an entire image and communicates with another communication terminal that displays a second region image indicating a second region in the entire image at another location, the program comprising: a receiving step of receiving second information for identifying the second area transmitted from the other communication terminal; a display control step of displaying the first region image including direction information indicating a direction of the second region relative to the first region in the entire image based on first information and the second information for identifying the first region; A program with.

8. An image communication system including: a communication terminal that displays a first region image indicating a first region in an entire image transmitted via a communication network, and communicates with another communication terminal that displays a second region image indicating a second region in the entire image at another location; and another communication terminal that displays the second region image indicating the second region in the entire image, a receiving means for receiving second information for identifying the second area transmitted from the other communication terminal; a display control means for displaying the first region image by including direction information indicating a direction of the second region relative to the first region in the entire image, based on first information and the second information for specifying the first region; An image communication system having:

Images