Image communication system, communication terminal, image communication method, communication method, display method, and program

Abstract

The present invention provides a communication terminal, an image communication system, a display method, and a program that can suppress delays in image data even when transmitting and receiving data for both 360-degree images and superimposed images. [Solution] In an image communication system, a communication terminal capable of displaying a predetermined region image, which is an image of a predetermined region in an overall image, on a display means includes: a receiving means that receives data of the overall image, data of the superimposed image, and superimposed position information indicating the position in which the superimposed image is superimposed within the overall image, transmitted by another communication terminal; an image processing means that superimposes the superimposed image onto the overall image based on the superimposed position information; and a display control means that causes the predetermined region image in the overall image to be displayed on the display means. The display control means causes the predetermined region image and the superimposed image to be displayed on the display means with image quality changed according to the ratio of the predetermined region image and the superimposed image within the display area of ​​the display means.

Description

Technical Field

[0001] The present invention relates to a communication terminal, an image communication system, a display method, and a program.

Background Art

[0002] Conference systems for holding remote conferences with remote locations via communication networks such as the Internet have become widespread. In this conference system, in a conference room where one of the parties such as the attendees of the remote conference is present, the communication terminal of the remote conference system is used to photograph and collect the image of the conference room and the voice such as the speech of the parties of the conference, and these are converted into digital data and transmitted to the communication terminal of the other party. As a result, since the image can be displayed on the display of the other party's conference room and the voice can be output by the speaker, a video call can be made, and thus a conference between remote locations can be held in a state close to an actual conference (see Patent Document 1).

[0003] In addition, a photographing device capable of acquiring an omnidirectional panoramic image in real time is connected to the communication terminal, and the omnidirectional panoramic image acquired from this photographing device is transmitted to each communication terminal of the other party, and in each communication terminal of the other party, a predetermined area image indicating a predetermined area that is a part of the omnidirectional panoramic image is displayed on a display or the like (see Patent Document 2). Thereby, the users at each base can independently determine and display a predetermined area image indicating a predetermined area that they are interested in among the entire image of the omnidirectional panoramic image.

[0004] In addition, it is disclosed that the transmitting-side communication terminal transmits the original video and the attention object image to the server, and the receiving-side communication terminal detects the coordinates of the attention object in the current video, so that the user on the transmitting side can superimpose a material image (superimposed image) such as graphic information on the area in the original video that the user wants to pay attention to. (See Patent Document 3).

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when transmitting and receiving both panoramic image data and superimposed image data between multiple communication terminals via a communication network such as the internet, a problem arises in that the communication load increases, causing delays in image data. [Means for solving the problem]

[0006] The invention according to claim 1 is a communication terminal capable of displaying a predetermined region image, which is an image of a predetermined region in an overall image, on a display means, comprising: a receiving means for receiving data of the overall image, data of an overlaid image, and overlaid position information indicating the position in which the overlaid image is overlaid within the overall image, transmitted by another communication terminal; an image processing means for overlaying the overlaid image onto the overall image based on the overlaid position information; and a display control means for displaying the predetermined region image in the overall image on the display means, wherein the display control means causes the display means to display the predetermined region image and the overlaid image with image quality changed according to the ratio of the predetermined region image and the overlaid image within the display area of ​​the display means. [Effects of the Invention]

[0007] As described above, the present invention has the effect of suppressing delays in image data, even when transmitting and receiving data for both a 360-degree image and a superimposed image. [Brief explanation of the drawing]

[0008] [Figure 1] (a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a top view of the imaging device. [Figure 2] This is a diagram illustrating the use of the imaging device. [Figure 3] (a) is a hemispherical image (front) taken with the imaging device, (b) is a hemispherical image (back) taken with the imaging device, and (c) is an image represented using the Mercator projection. [Figure 4] (a) A conceptual diagram showing the state of the sphere being covered by a Mercator image, and (b) A diagram showing a panoramic image of the entire sky. [Figure 5] This diagram shows the positions of a virtual camera and a predetermined area when a full-spherical panoramic image is treated as a three-dimensional sphere. [Figure 6] (a) is a stereoscopic perspective view of Figure 5, and (b) is a diagram showing the state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 7] This figure shows the relationship between information from a predetermined region and an image of a predetermined region T. [Figure 8] This is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. [Figure 9] This is a schematic diagram of an image communication system according to an embodiment of the present invention. [Figure 10] This is a hardware configuration diagram of the imaging device. [Figure 11] This is a hardware configuration diagram for a video conferencing terminal. [Figure 12] This is a hardware configuration diagram of the communication management system and PC. [Figure 13] This is a hardware configuration diagram of a smartphone. [Figure 14] This is a functional block diagram of a part of the image communication system. [Figure 15] This is a functional block diagram of a part of the image communication system. [Figure 16] This is a conceptual diagram showing the image type management table. [Figure 17] This is a conceptual diagram showing the management table for the imaging equipment. [Figure 18] This is a conceptual diagram showing a designated area management table. [Figure 19] This is a conceptual diagram showing the superposition position management table. [Figure 20] This is an explanatory diagram of the overlapping positions. [Figure 21] This is a conceptual diagram showing the session management table. [Figure 22] This is a conceptual diagram showing the image type management table. [Figure 23] This is a conceptual diagram showing a designated area information management table. [Figure 24]It is a conceptual diagram showing a ratio management table. [Figure 25] It is a sequence diagram showing the participation process in a specific communication session. [Figure 26] It is a diagram showing the selection screen of a communication session (virtual conference room). [Figure 27] It is a sequence diagram showing the management process of image type information. [Figure 28] It is an image diagram showing the state of a video call. [Figure 29] It is a sequence diagram showing the communication process of captured image data, audio data, and material image data in a video call. [Figure 30] An example of the display on the display at Site B is shown. (a) shows the case where the image data sent from the imaging devices 1a and 1b is displayed as it is without creating an omnidirectional panorama image and a predetermined area image. Figure 30(b) shows the case where an omnidirectional panorama image and a predetermined area image are created from the image data sent from the imaging devices 1a and 1b. Figure 30(c) is a diagram showing the case where the predetermined area image in Figure 30(b) is changed. [Figure 31] It is a sequence diagram showing the process of sharing predetermined area information. [Figure 32] It is a sequence diagram showing another process of sharing predetermined area information. [Figure 33] It is a sequence diagram showing the process of sharing superimposed position information. [Figure 34] It is an example screen for changing the superimposed position. [Figure 35] It is a flowchart showing the display process of the superimposed image. [Figure 36] An example of the display of the superimposed image (material image) at Site B is shown. [Figure 37] It is a sequence diagram showing the process of changing the image quality. [Figure 38] It is a diagram when the predetermined area image including the material image is changed. [Figure 39] It is a diagram showing the changed predetermined area. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to Figures 1 to 36.

[0010] <<Outline of the Embodiment>> <Method for generating a full-spherical panoramic image> The method for generating a full-sphere panoramic image will be explained using Figures 1 to 7.

[0011] First, Figure 1 will be used to describe the appearance of the imaging device 1. The imaging device 1 is a digital camera used to obtain images that will serve as the basis for a three-dimensional 360° panoramic image. Figure 1(a) is a left side view of the imaging device, Figure 1(b) is a front view of the imaging device, and Figure 1(c) is a top view of the imaging device.

[0012] As shown in Figure 1(a), the imaging device 1 is small enough to be held in one hand. Also, as shown in Figures 1(a), 1(b), and 1(c), the imaging device 1 has an image sensor 103a on the front side and an image sensor 103b on the rear side. These image sensors 103a and 103b are used in conjunction with an optical component (for example, a fisheye lens 102a or 102b, which will be described later) that can capture hemispherical images (angle of view of 180° or more). Also, as shown in Figure 1(b), an operating section 115, such as a shutter button, is provided on the side of the imaging device 1 opposite to the front side.

[0013] Next, the usage of the imaging device 1 will be explained using Figure 2. Figure 2 is an illustrative diagram of the imaging device in use. As shown in Figure 2, the imaging device 1 is used, for example, by a user holding it in their hand to photograph subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image sensors 103a and 103b shown in Figure 1.

[0014] Next, using Figures 3 and 4, we will outline the process from the image captured by the imaging device 1 to the creation of a full-sphere panoramic image. Figure 3(a) shows the hemispherical image (front) captured by the imaging device, Figure 3(b) shows the hemispherical image (back) captured by the imaging device, and Figure 3(c) shows the image represented by the Mercator projection (hereinafter referred to as the "Mercator image"). Figure 4(a) is a conceptual diagram showing the state in which the sphere is covered by the Mercator image, and Figure 4(b) shows the full-sphere panoramic image.

[0015] As shown in Figure 3(a), the image obtained by the image sensor 103a becomes a curved hemispherical image (front side) by the fisheye lens 102a described later. Similarly, as shown in Figure 3(b), the image obtained by the image sensor 103b becomes a curved hemispherical image (back side) by the fisheye lens 102b described later. The hemispherical image (front side) and the hemispherical image (back side) which is inverted 180 degrees are superimposed by the imaging device 1 to create a Mercator image, as shown in Figure 3(c).

[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is superimposed to cover the sphere, as shown in Figure 4(a), creating a full-sphere panoramic image as shown in Figure 4(b). In this way, the full-sphere panoramic image is represented as an image where the Mercator image is oriented towards the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. The full-sphere panoramic image can be a still image or a video.

[0017] As described above, a full-sphere panoramic image is an image pasted over a sphere, which can cause discomfort to the human eye. Therefore, by displaying a predetermined area of ​​the full-sphere panoramic image (hereinafter referred to as the "predetermined area image") as a flat image with less curvature, it is possible to display it in a way that does not cause discomfort to the human eye. This will be explained using Figures 5 and 6.

[0018] Figure 5 shows the positions of the virtual camera and the predetermined region when the 360-degree image is treated as a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of the user viewing the 360-degree image CE, which is displayed as a three-dimensional sphere. Figure 6(a) is a stereoscopic perspective view of Figure 5, and Figure 6(b) shows the predetermined region image as it appears on a display. In Figure 6(a), the 360-degree image CE shown in Figure 4 is represented as a three-dimensional sphere CS. If the generated 360-degree image CE is a sphere CS, then, as shown in Figure 5, the virtual camera IC is located inside the 360-degree image CE. The predetermined region T in the 360-degree image CE is the shooting area of ​​the virtual camera IC and is identified by predetermined region information indicating the shooting direction and field of view of the virtual camera IC in the three-dimensional virtual space including the 360-degree image CE.

[0019] Then, the predetermined region image Q shown in Figure 6(a) is displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC, as shown in Figure 6(b). The image shown in Figure 6(b) is a predetermined region image represented by the predetermined region information that has been initially set (default). Note that the predetermined region information may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined region T, rather than the position coordinates of the virtual camera IC. In the following explanation, we will use the shooting direction (rH, rV) and field of view (α) of the virtual camera IC.

[0020] The relationship between the predetermined region information and the image of the predetermined region T will be explained using Figure 7. Figure 7 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 7, rH is the Horizontal Radian, rV is the Vertical Radian, and α is the field of view (Angle). That is, the orientation of the virtual camera IC will be changed so that the point of fixation of the virtual camera IC, indicated by the shooting direction (rH, rV), becomes the center point CP of the predetermined region T, which is the shooting area of ​​the virtual camera IC. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined region T and the center point CP (2L is the diagonal). And, in Figure 7, the trigonometric function shown in (Equation 1) below generally holds true.

[0021] Lf = tan(α / 2) ... (Equation 1) Figure 8 shows a point in a three-dimensional Euclidean space using spherical coordinates. Let (r, θ, φ) be the position coordinates of the center point CP when expressed in spherical polar coordinates. (r, θ, φ) are the radial, polar angle, and azimuth angle, respectively. The radial r is equal to f because it is the distance from the origin of the three-dimensional virtual space containing the panoramic image to the center point CP. Figure 8 illustrates these relationships. Hereafter, we will use the position coordinates (r, θ, φ) of the virtual camera IC for explanation.

[0022] <Overview of the image communication system> Next, the general configuration of the image communication system of this embodiment will be explained using Figure 9. Figure 9 is a schematic diagram of the configuration of the image communication system of this embodiment.

[0023] As shown in Figure 9, the image communication system of this embodiment consists of a camera 1a, 1b, video conferencing terminals 3a, 3b, displays 4a, 4b, a communication management system 5, a PC (Personal Computer) 6, a PC 7, a camera 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection method of the communication network 100 may be either wireless or wired.

[0024] Of these, the shooting devices 1a and 1b are special digital cameras used to capture subjects, landscapes, etc., and obtain two hemispherical images that form the basis of a full-sphere panoramic image, as described above. On the other hand, the shooting device 8 is a general-purpose digital camera used to capture subjects, landscapes, etc., and obtain a general planar image.

[0025] The video conferencing terminals 3a and 3b are dedicated to video conferencing and display the video call image on displays 4a and 4b, respectively, via a wired cable such as a USB (Universal Serial Bus) cable. Normally, the video conferencing terminal 3a captures images using the camera 312 shown in Figure 11 (described later). However, when connected via a wired cable to the cradle 2a, which houses the camera 1a, the camera 1a takes priority, allowing the terminal to obtain two hemispherical images that form the basis of a full-spherical panoramic image. Furthermore, the video conferencing terminal 3a can be connected to PC 6, allowing access to the PC 6's screen. When using a wired cable, the cradle 2a not only facilitates communication between the camera 1a and the video conferencing terminal 3a but also supplies power to and supports the camera 1a. In this example, the camera 1a, cradle 2a, video conferencing terminal 3a, display 4a, and PC 6 are all located at the same location, site A. Additionally, four users, A1, A2, A3, and A4, are participating in the video call at site A. Meanwhile, the video conferencing terminal 3d and display 4d are located at the same location, location D. Three users, D1, D2, and D3, are participating in the video call at location D.

[0026] The communication management system 5 manages and controls the communication between the video conferencing terminals 3a, 3b, PC 7, and smartphone 9, and also manages the types of image data transmitted and received (general images and special images). Therefore, the communication management system is also a communication control system. Here, special images are 360-degree panoramic images. The communication management system 5 is installed in service companies that provide video communication services. Furthermore, the communication management system 5 may be constructed by a single computer, or it may be constructed by multiple computers in which each part (function, means, or memory unit) is divided and arbitrarily assigned.

[0027] PC6 generates the data (image data) of the materials displayed in the video conference. These materials include, but are not limited to, images displayed, created, or edited using general-purpose applications running on PC6, or images played back on PC6 using a general digital camera, etc.

[0028] PC7 becomes capable of video calls when the camera 8 is attached to it. In this case, both PC7 and camera 8 are located at the same location, location C. Additionally, one user, C, is participating in the video call at location C.

[0029] Smartphone 9 displays the video call image on a display 917 located on its own device, as described later. Normally, smartphone 9 captures images using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 located on its own device, as described later. However, it can also acquire two hemispherical image data, which form the basis of the full-sphere panoramic image obtained by the imaging device 1b, using wireless communication technologies such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark). When using wireless communication technology, cradle 2b only serves to supply power to imaging device 1b and support it. In this case, imaging device 1b, cradle 2b, and smartphone 9 are all located at the same location, location B. Two users, B1 and B2, are participating in the video call at location B.

[0030] Furthermore, video conferencing terminals 3a and 3d, PC 7, and smartphone 9 are examples of communication terminals. Each communication terminal has OpenGL ES installed and can create predetermined region information that indicates a portion of a full-spherical panoramic image, or create predetermined region images from full-spherical panoramic images sent from other communication terminals.

[0031] Note that the arrangement of each terminal (communication terminal, display, camera), device, and user shown in Figure 9 is just one example, and other examples are also possible. For example, at site C, a camera capable of capturing 360-degree panoramic images may be used instead of camera 8. Communication terminals also include digital televisions, smartwatches, car navigation devices, etc. Furthermore, from now on, any camera among camera 1a and 1b will be referred to as "camera 1". Similarly, any video conferencing terminal among video conferencing terminals 3a and 3b will be referred to as "video conferencing terminal 3". In addition, any display among displays 4a and 4b will be referred to as "display 4".

[0032] <<Hardware configuration of the embodiment>> Next, the hardware configuration of the shooting device 1, video conferencing terminal 3, communication management system 5, PC 6, PC 7, and smartphone 9 of this embodiment will be described in detail with reference to Figures 10 to 13. Since the camera 8 is a standard camera, a detailed explanation will be omitted.

[0033] <Hardware configuration of imaging device 1> First, the hardware configuration of the imaging device 1 will be explained using Figure 10. Figure 10 is a hardware configuration diagram of the imaging device 1. In the following, the imaging device 1 will be assumed to be a 360-degree (omnidirectional) imaging device using two image sensors, but it may have more than two image sensors. Furthermore, it is not necessarily required to be a device dedicated to omnidirectional imaging; a regular digital camera or smartphone can be fitted with an aftermarket omnidirectional imaging unit to achieve essentially the same functionality as the imaging device 1.

[0034] As shown in Figure 10, the imaging device 1 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network interface 116, a communication unit 117, and an antenna 117a.

[0035] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b, each having a field of view of 180° or more for forming hemispherical images, and two image sensors 103a and 103b, each corresponding to the wide-angle lens. The image sensors 103a and 103b include an image sensor such as a CMOS sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lenses 102a and 102b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the image sensors are set.

[0036] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are connected separately from the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via bus 110. Furthermore, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are also connected to bus 110.

[0037] The image processing unit 104 receives image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then superimposes these image data to create Mercator image data as shown in Figure 3(c).

[0038] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 acting as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0039] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 1, there may also be functions to display a preview or video on a display (for example, the display of the video conferencing terminal 3a). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0040] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 1 is not provided with a display unit, but a display unit may be provided.

[0041] Microphone 108 converts sound into sound (signal) data. Sound processing unit 109 receives the sound data output from microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0042] The CPU 111 controls the overall operation of the imaging device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress and processed Mercator image data from the image processing unit 104.

[0043] The control unit 115 is a collective term for various operation buttons, a power switch, a shutter button, and a touch panel that combines display and operation functions. The user inputs various shooting modes and shooting conditions by operating the operation buttons.

[0044] The network interface 116 is a general term for interface circuits (such as USB interfaces) that connect to external media such as SD cards or personal computers. The network interface 116 can be wireless or wired. The Mercator image data stored in the DRAM 114 is recorded to external media via this network interface 116, or transmitted to external devices such as video conferencing terminals 3a via the network interface 116 as needed.

[0045] The communication unit 117 communicates with external devices such as the video conferencing terminal 3a via the antenna 117a provided on the imaging device 1, using short-range wireless technologies such as Wi-Fi or NFC (Near Field Communication). This communication unit 117 can also transmit Mercator image data to the external device of the video conferencing terminal 3a.

[0046] The electronic compass 118 calculates the orientation and tilt (roll angle) of the imaging device 1 from the Earth's magnetic field and outputs orientation and tilt information. This orientation and tilt information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was taken and the data size of the image data.

[0047] <Hardware configuration of video conferencing terminals> Next, the hardware configuration of the video conferencing terminal 3 will be explained using Figure 11. Figure 11 is a hardware configuration diagram of the video conferencing terminal. As shown in Figure 11, the video conferencing terminal 3 is equipped with a CPU 301, ROM 302, RAM 303, flash memory 304, SSD 305, media I / F 307, operation buttons 308, power switch 309, bus line 310, network I / F 311, camera 312, image sensor I / F 313, microphone 314, speaker 315, sound input / output I / F 316, display I / F 317, external device connection I / F 318, short-range communication circuit 319, and antenna 319a of the short-range communication circuit 319.

[0048] Of these components, the CPU 301 controls the overall operation of the video conferencing terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as the IPL (Initial Program Loader). The RAM 303 is used as the work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and sound data. The SSD (Solid State Drive) 305 controls the reading or writing of various data to the flash memory 304 according to the control of the CPU 301. Note that an HDD may be used instead of the SSD. The media I / F 307 controls the reading or writing (storage) of data to the recording media 306, such as the flash memory. The operation button 308 is used to select a destination on the video conferencing terminal 3, etc. The power switch 309 is a switch for turning the power of the video conferencing terminal 3 ON / OFF.

[0049] Furthermore, the network I / F 311 is an interface for data communication using a communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 301. The image sensor I / F 313 is a circuit that controls the driving of the camera 312. The microphone 314 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 316 is a circuit that processes the input and output of sound signals between the microphone 314 and the speaker 315 according to the control of the CPU 301. The display I / F 317 is a circuit that transmits image data to an external display 4 according to the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) or Bluetooth (registered trademark).

[0050] Furthermore, the bus line 310 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 301 shown in Figure 11.

[0051] Display 4 is a type of display means composed of liquid crystal or organic EL that displays images of the subject, operation icons, etc. Display 4 is connected to display I / F 317 by cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0052] The camera 312 includes a lens and a solid-state image sensor that converts light into electric charge to create an image (video) of the subject. CMOS sensors and CCD sensors are used as the solid-state image sensor. External devices such as external cameras, external microphones, and external speakers can be connected to the external device connection I / F 318 via USB (Universal Serial Bus) cables, etc. When an external camera is connected, it is driven in priority over the built-in camera 312, according to the control of the CPU 301. Similarly, when an external microphone or external speaker is connected, it is driven in priority over the built-in microphone 314 and built-in speaker 315, respectively, according to the control of the CPU 301.

[0053] Furthermore, the recording medium 306 is configured to be detachable from the video conferencing terminal 3. In addition, any non-volatile memory that reads or writes data according to the control of the CPU 301 may be used, not limited to flash memory 304, but also including EEPROM (Electrically Erasable and Programmable ROM), etc.

[0054] <Communication management system, PC hardware configuration> Next, Figure 12 will be used to explain the hardware configuration of the communication management system 5, PC 6, and PC 7. Figure 12 is a hardware configuration diagram of the communication management system and the PCs. Since the communication management system 5, PC 6, and PC 7 are all computers with the same configuration, the following explanation will focus on the configuration of the communication management system 5, and the explanations of the configurations of PC 6 and PC 7 will be omitted.

[0055] The communication management system 5 includes a CPU 501 that controls the overall operation of the communication management system 5, a ROM 502 that stores programs used to drive the CPU 501 such as an IPL, a RAM 503 used as the work area of ​​the CPU 501, an HD 504 that stores various data such as programs for the communication management system 5, an HDD (Hard Disk Drive) 505 that controls the reading or writing of various data to the HD 504 according to the control of the CPU 501, a media drive 507 that controls the reading or writing (storage) of data to a recording medium 506 such as flash memory, a display 508 that displays various information such as cursors, menus, windows, characters, or images, a network I / F 509 for data communication using the communication network 100, a keyboard 511 with multiple keys for inputting characters, numbers, and various instructions, a mouse 512 for selecting and executing various instructions, selecting processing targets, moving the cursor, etc., and a CD-RW (Compact Disc) as an example of a removable recording medium. The system includes a CD-RW drive 514 that controls the reading of various data to the ReWritable) 513, and bus lines 510 such as an address bus and a data bus for electrically connecting the above components as shown in Figure 12.

[0056] <Smartphone Hardware Configuration> Next, the hardware of the smartphone will be described using Figure 13. Figure 13 is a hardware configuration diagram of the smartphone. As shown in Figure 13, the smartphone 9 is equipped with a CPU 901, ROM 902, RAM 903, EEPROM 904, CMOS sensor 905, accelerometer / compass sensor 906, media I / F 908, and GPS receiver 909.

[0057] Of these, the CPU 901 controls the overall operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901, such as the IPL. The RAM 903 is used as the work area for the CPU 901. The EEPROM 904 reads or writes various data, such as smartphone programs, according to the control of the CPU 901. The CMOS sensor 905 captures images of a subject (mainly a self-portrait) and obtains image data according to the control of the CPU 901. The acceleration / direction sensor 906 is a variety of sensors, such as an electronic magnetic compass, gyrocompass, and acceleration sensor that detect the Earth's magnetic field. The media I / F 908 controls the reading or writing (storage) of data to or from the recording medium 907, such as flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0058] The smartphone 9 also includes a long-range communication circuit 911, a camera 912, an image sensor interface 913, a microphone 914, a speaker 915, an audio input / output interface 916, a display 917, an external device connection interface 918, a short-range communication circuit 919, an antenna 919a for the short-range communication circuit 919, and a touch panel 921.

[0059] Of these, the long-range communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 901. The image sensor interface 913 is a circuit that controls the driving of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs sound. The sound input / output interface 916 is a circuit that processes the input and output of sound signals between the microphone 914 and the speaker 915 according to the control of the CPU 901. The display 917 is a type of display means such as a liquid crystal or organic EL that displays images of the subject and various icons. The external device connection interface 918 is an interface for connecting various external devices. The short-range communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means that allows the user to operate the smartphone 9 by pressing the display 917.

[0060] Furthermore, the smartphone 9 is equipped with a bus line 910. The bus line 910 is an address bus, data bus, etc., for electrically connecting various components such as the CPU 901.

[0061] Furthermore, recording media such as CD-ROMs on which the above programs are stored, as well as hard drives on which these programs are stored, may be provided domestically or internationally as program products.

[0062] <<Embodiment Functional Configuration>> Next, the functional configuration of this embodiment will be described using Figures 14 to 21. Figures 14 and 15 are functional block diagrams of a part of the image communication system.

[0063] <Functional configuration of imaging device 1a> As shown in Figure 14, the imaging device 1a includes a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a storage / reading unit 19a. Each of these units is a function or means realized by any of the components shown in Figure 10 operating according to instructions from the CPU 111 that follow a program for the imaging device deployed on SRAM 113 to DRAM 114.

[0064] Furthermore, the imaging device 1 has a storage unit 1000a constructed from ROM 112, SRAM 113, and DRAM 114 as shown in Figure 10. The storage unit 1000a stores the GUID (Globally Unique Identifier) ​​of the device itself.

[0065] Although the imaging device 1b includes a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, a memory / reading unit 19b, and a memory unit 1000b, these functions are the same as those of the reception unit 12a, imaging unit 13a, sound collection unit 14a, communication unit 18a, memory / reading unit 19a, and memory unit 1000a in the imaging device 1a, so their descriptions are omitted.

[0066] (Functional configuration of each part of the imaging device 1a) Next, the functional configurations of the imaging device 1a will be explained in more detail using Figures 10 and 14.

[0067] The reception unit 12a of the imaging device 1 is mainly implemented by the operation unit 115 and the CPU 111 shown in Figure 10, and receives operation input from the user.

[0068] The imaging unit 13a is mainly realized by the imaging unit 101, image processing unit 104, and imaging control unit 105 shown in Figure 10, as well as the processing of the CPU 111, to capture images of landscapes, etc., and obtain captured image data (captured image data).

[0069] The sound collection unit 14a is realized by the processing of the 108 and sound processing unit 109 shown in Figure 10, as well as the CPU 111, and collects sound from the surrounding area of ​​the imaging device 1a.

[0070] The communication unit 18a is primarily implemented by the processing of the CPU 111 and can communicate with the communication unit 38a of the video conferencing terminal 3a using short-range wireless communication technologies such as NFC, Bluetooth, and WiFi.

[0071] The storage and reading unit 19a is mainly implemented by the processing of the CPU 111 shown in Figure 10, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.

[0072] <Functional Configuration of Video Conferencing Terminal 3a> As shown in Figure 14, the video conferencing terminal 3a includes a transmitting / receiving unit 31a, a receiving unit 32a, an image / sound processing unit 33a, a display control unit 34a, a determination unit 35a, a creation unit 36a, a calculation unit 37a, a communication unit 38a, and a storage / reading unit 39a. Each of these units is a function or means realized by any of the components shown in Figure 11 operating according to instructions from the CPU 301 that follow the program for the video conferencing terminal 3a deployed from the flash memory 304 onto the RAM 303.

[0073] Furthermore, the video conferencing terminal 3a has a storage unit 3000a constructed from a ROM 302, RAM 303, and flash memory 304 as shown in Figure 11. This storage unit 3000a contains an image type management DB 3001a, an imaging device management DB 3002a, a predetermined area management DB 3003a, and an overlay position management DB 3004a. Of these, the image type management DB 3001a is composed of the image type management table shown in Figure 16. The imaging device management DB 3002a is composed of the imaging device management table shown in Figure 17. The predetermined area management DB 3003a is composed of the predetermined area management table shown in Figure 18. The overlay position management DB 3004a is composed of the overlay position management table shown in Figure 19.

[0074] Although the video conferencing terminal 3d has a transmitting / receiving unit 31d, a receiving unit 32d, an image / sound processing unit 33d, a display control unit 34d, a judgment unit 35d, a creation unit 36d, a calculation unit 37d, a communication unit 38d, a storage / reading unit 39d, and a storage unit 3000d, these units perform the same functions as the transmitting / receiving unit 31a, the receiving unit 32a, the image / sound processing unit 33a, the display control unit 34a, the judgment unit 35a, the creation unit 36a, the calculation unit 37a, the communication unit 38a, the storage / reading unit 39a, and the storage unit 3000a in the video conferencing terminal 3a, their descriptions are omitted. Furthermore, the memory unit 3000d in the video conferencing terminal 3d contains the image type management DB 3001d, the imaging device management DB 3002d, the predetermined area management DB 3003d, and the superimposed position management DB 3004d. However, since these have the same data structure as the image type management DB 3001a, imaging device management DB 3002a, predetermined area management DB 3003a, and superimposed position management DB 3004a in the video conferencing terminal 3a, their explanations will be omitted.

[0075] (Image type management table) Figure 16 is a conceptual diagram showing the image type management table. In this image type management table, the image data ID, the IP address (an example of the destination of the sending terminal), and the source name are stored and managed in association. Of these, the image data ID is an example of image data identification information used to identify image data when performing video communication. Image data sent from the same sending terminal is assigned the same image data ID. This allows the receiving terminal (the receiving communication terminal) to identify the sending terminal of the received image data. The IP address of the sending terminal indicates the IP address of the communication terminal that sends the image data indicated by the associated image data ID. The source name is a name used to identify the imaging device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal, such as the video conferencing terminal 3a, according to a predetermined naming convention.

[0076] For example, the four communication terminals with IP addresses "1.2.1.3", "1.2.2.3", "1.3.1.3", and "1.3.2.3" respectively are shown to be transmitting image data indicated by image data IDs "RS001", "RS002", "RS003", and "RS004", respectively. Furthermore, the image types indicated by the source names of each communication terminal are "Video_Theta", "Video_Theta", "Video", and "Video", which indicate that the image types are "Special Image", "Special Image", "General Image", and "General Image", respectively. In this case, the special image is a 360-degree panoramic image.

[0077] Furthermore, data other than image data may also be managed in association with image data IDs. Examples of data other than image data include sound data. (Imaging equipment management table) Figure 17 is a conceptual diagram showing the imaging device management table. This imaging device management table stores and manages the vendor ID and product ID of the GUID of the imaging device that can obtain the two hemispherical images that form the basis of a full-spherical panoramic image. As GUIDs, for example, the vendor ID (VID) and product ID (PID) used in USB devices can be used. These vendor ID and product ID are stored from the time of factory shipment of communication terminals such as video conferencing terminals, but they may also be stored after factory shipment.

[0078] (Designated area management table) Figure 18 is a conceptual diagram showing a predetermined area management table. In this predetermined area management table, the IP address of the communication terminal that sends the captured image data, the IP address of the communication terminal to which the captured image data is sent, and predetermined area information indicating the predetermined area image currently displayed on the communication terminal to which the captured image data is sent are stored and managed in association. Note that the communication terminal to which the captured image data is sent is also the communication terminal that sends the predetermined area information. As shown in Figures 6 and 7, the predetermined area information is a conversion parameter for converting the captured image into an image of the predetermined area T in that captured image (predetermined area image). Note that the IP address is an example of destination information, and destination information includes MAC (Media Access Control) address, terminal ID (Identification) to identify the communication terminal, etc. Also, here the IP address is represented in a simplified form of IPv4 address. The IP address may also be IPv6.

[0079] For example, in the first to third rows of the predetermined area management table in Figure 18, it is managed that if the IP address of video conferencing terminal 3a is "1.2.1.3", the captured image data transmitted from video conferencing terminal 3a was sent via the communication management system 5 to video conferencing terminal 3d with IP address "1.2.2.3", PC 7 with IP address "1.3.1.3", and smartphone 9 with IP address "1.3.2.3". Furthermore, it is managed that video conferencing terminal 3d is the communication terminal that sent the predetermined area information (r=10,θ=20,φ=30). Similarly, it is managed that PC 7 is the communication terminal that sent the predetermined area information (r=10,θ=30,φ=40). Also, it is managed that smartphone 9 is the communication terminal that sent the predetermined area information (r=30,θ=40,φ=50).

[0080] Furthermore, if the transmitting / receiving unit 31a receives new predetermined area information that includes the same set of IP addresses as the IP address of the communication terminal that is the source of the captured image data and the IP address of the communication terminal to which the captured image data is sent, the storage / reading unit 39a overwrites the predetermined area information that is already being managed with the newly received predetermined area information.

[0081] (Overlapping Position Management Table) Figure 19 is a conceptual diagram showing the superposition position management table. In this superposition position management table, the IP address of the communication terminal that sent the captured image data and the superposition position information indicating where the source image (the reference image) should be superimposed on the destination image (the full-spherical panoramic image) are stored and managed in association with each other.

[0082] Figure 20 illustrates superimposed coordinates as an example of superimposed position information. Figure 20 shows a Mercator image of a 360-degree video with a source image (the source image) superimposed on it. Superimposed coordinates consist of a start point coordinate and an end point coordinate. When the top-left vertex of the source image (the source image) is taken as the start point, the start point coordinate can be expressed as (0,0) and the end point coordinate as (xn,yn). In this case, when superimposing the start point coordinate (0,0) and the end point coordinate (xn,yn) to positions corresponding to coordinates (x1,y1) and (x2,y2) in the Mercator image, respectively, they are associated with (x1,x2) as the start point coordinate and (x2,y2) as the end point coordinate in the superimposed position management table. Note that the IP address is an example of destination information, and destination information includes MAC (Media Access Control) address, terminal ID (Identification) to identify the communication terminal, etc. Also, here the IP address is represented in a simplified form of IPv4 address. The IP address can also be IPv6.

[0083] (Functional configuration of video conferencing terminal 3a) Next, we will explain in more detail the functional configuration of the video conferencing terminal 3a using Figures 11 and 14.

[0084] The transmitting / receiving unit 31a of the video conferencing terminal 3a is mainly realized by the processing of the network interface 311 and CPU 301 shown in Figure 11, and transmits and receives various data (or information) with the communication management system 5 via the communication network 100.

[0085] The reception unit 32a is primarily implemented by the operation buttons 308 and the CPU 301, and accepts various selections or inputs from the user. In addition to the operation buttons 308, other input means such as a touch panel may also be used.

[0086] The image and sound processing unit 33a is implemented by instructions from the CPU 301 shown in Figure 11, and performs image processing on the image data obtained when the camera 312 captures a subject. In addition, the image and sound processing unit 33a performs audio processing on the sound data related to the audio signal after the user's voice is converted into an audio signal by the microphone 314.

[0087] Furthermore, the image / sound processing unit 33a performs image processing on image data received from other communication terminals based on image type information such as the source name, in order for the display control unit 34 to display the image on the display 4. Specifically, if the image type information indicates that it is a special image, the image / sound processing unit 33a creates a full-sphere panoramic image by converting the image data (for example, the data of each hemispherical image as shown in Figures 3(a) and (b)) into a full-sphere panoramic image as shown in Figure 4(b). Furthermore, if there is a source image to be superimposed on the full-sphere panoramic image, it generates a superimposed image, which is an image in which the source image is superimposed on the full-sphere panoramic image, based on the source image data and superimposition position information. The image / sound processing unit 33a also creates a predetermined region image as shown in Figure 6(b). In addition, the image / sound processing unit 33a outputs an audio signal related to the sound data received from other communication terminals via the communication management system 5 to the speaker 315, and outputs sound from the speaker 315.

[0088] The display control unit 34a is mainly implemented by the processing of the display I / F 317 and the CPU 301, and controls the display of various images, characters, etc. on the display 4.

[0089] The determination unit 35a is mainly implemented by the processing of the CPU 301 and, for example, determines the type of image data received from the imaging device 1a, but is not limited to this and performs various other determinations related to the image data.

[0090] The creation unit 36a is mainly implemented by the CPU 301 and, based on the determination by the judgment unit 35a that it is a general image or a special image (in this case, a full-sphere panoramic image), creates a source name, which is an example of image type information, according to the naming rules described above. For example, if the judgment unit 35a determines that it is a general image, the creation unit 36a creates a source name "Video" that indicates it is a general image. On the other hand, if the judgment unit 35a determines that it is a special image, the creation unit 36a creates a source name "Video_Theta" that indicates it is a special image.

[0091] The calculation unit 37a is mainly implemented by the CPU 301 and calculates superimposed position information.

[0092] The communication unit 38a is mainly realized by the processing of the short-range communication circuit 319, antenna 318a, and CPU 301, and can communicate with the communication unit 18a of the imaging device 1a using short-range wireless technologies such as NFC, Bluetooth, and WiFi. Although the communication unit 38a and the transceiver unit 31a have been described as having separate communication units, they may also be in a shared configuration.

[0093] The storage and reading unit 39a is mainly implemented by the processing of the CPU 301 shown in Figure 11, and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000.

[0094] <Functional Configuration of Communication Management System> Next, the functional configurations of the communication management system 5 will be described in detail using Figures 12 and 15. The communication management system 5 includes a transmitting / receiving unit 51, a determination unit 55, a generation unit 56, and a storage / reading unit 59. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 501 that follow the program for the communication management system 5 deployed from the HD 504 onto the RAM 503.

[0095] Furthermore, the communication management system 5 has a storage unit 5000 constructed from the RAM 503 and HD 504 shown in Figure 12. This storage unit 5000 contains a session management DB 5001, an image type management DB 5002, a predetermined area management DB 5003, and a percentage management DB 5004. Of these, the session management DB 5001 is composed of the session management table shown in Figure 21. The image type management DB 5002 is composed of the image type management table shown in Figure 22. The predetermined area management DB 5003 is composed of the predetermined area information management table shown in Figure 23. The percentage management DB 5004 is composed of the percentage management table shown in Figure 24.

[0096] (Session management table) Figure 21 is a conceptual diagram showing the session management table. In this session management table, the session ID and the IP address of the participating communication terminal are stored and managed in association. The session ID is an example of session identification information used to identify a communication session that enables video calls, and is generated for each virtual conference room. The session ID is also managed by each communication terminal, such as the video conferencing terminal 3a, and is used when selecting a communication session at each communication terminal. The IP address of the participating communication terminal indicates the IP address of the communication terminal that joined the virtual conference room indicated by the associated session ID.

[0097] (Image type management table) Figure 22 is a conceptual diagram showing the image type management table. The image type management table shown in Figure 22 manages the same information as the image type management table shown in Figure 16, as well as the same session ID as the session ID managed in the session management table. Here, it is shown that three communication terminals with IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1.3" are participating in the virtual conference room indicated by the same session ID "se101". Note that the reason why the communication management system 5 manages the same image data ID, source terminal IP address, and image type information for communication terminals such as the video conferencing terminal 3a is to send image type information etc. to the communication terminal already in a video call and the newly joined communication terminal when a new communication terminal enters the virtual conference room. This eliminates the need to send and receive image type information etc. between the communication terminal already in a video call and the newly joined communication terminal.

[0098] (Designated area information management table) Figure 23 is a conceptual diagram showing the predetermined area information management table. This predetermined area information management table basically has the same data structure as the predetermined area information management table shown in Figure 18. However, as will be described later, the transmitting / receiving unit 51 transmits the latest predetermined area information to each communication terminal at regular intervals (e.g., 30 seconds), so all predetermined area information received by the transmitting / receiving unit 51 is stored without being deleted until the predetermined area information is transmitted at regular intervals. In Figure 21, newer predetermined area information is managed at a higher level.

[0099] (Percentage Management Table) FIG. 24 is a conceptual diagram showing a ratio management table. In this ratio management table, the IP address of the communication terminal that is the transmission source of the photographed image data, the IP address of the communication terminal that is the transmission destination of the photographed image data, and ratio information are associated, stored, and managed. The ratio information indicates the ratio of the display areas of the predetermined area image and the material image (an example of an overlapping image) in the omnidirectional panorama image data sent from the communication terminal of the transmission source to the display means such as a display in the communication terminal of the transmission destination (reception side) of the photographed image data when displaying the material image.

[0100] (Functional Configurations of Communication Management System) Next, with reference to FIGS. 12 and 15, the functional configurations of the communication management system 5 will be described in detail.

[0101] The transmission / reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 12, and transmits and receives various data (or information) to and from the video conference terminals 3a, 3d, or the PC 7 via the communication network 100.

[0102] The determination unit 55 is mainly realized by the processing of the CPU 501 and makes various determinations.

[0103] The generation unit 56 is mainly realized by the processing of the CPU 501 and generates an image data ID.

[0104] The storage / reading unit 59 is mainly realized by the processing of the HDD 505 shown in FIG. 12 and the CPU 501, and stores various data (or information) in the storage unit 5000 or reads out various data (or information) from the storage unit 5000.

[0105] <Functional Configuration of PC>[ Next, with reference to FIGS. 12 and 14, the functional configurations of the PC 6 will be described in detail.

[0106] As shown in Figure 14, PC6 includes a reception unit 62, a display control unit 64, a communication unit 68, and a storage / reading unit 69. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 501 in accordance with the PC6 program deployed from HD 504 onto RAM 503.

[0107] Furthermore, PC6 has a storage unit 6000 constructed from ROM 502, RAM 503, and HD 504 as shown in Figure 12.

[0108] (The functional configuration of a PC) The reception unit 62 of PC6 is mainly implemented by the keyboard 511, mouse 512, and CPU 501, and performs the same functions as the reception unit 32a. The display control unit 64 is mainly implemented by the CPU 501 and controls the display 508 to display various images, characters, etc. The communication unit 68 is mainly implemented by the CPU 501 and can communicate with the communication unit 38a of the video conferencing terminal 3a using short-range wireless communication technologies such as NFC, Bluetooth, and WiFi. The storage / reading unit 69a is implemented by the CPU 501 and stores various data (or information) in the storage unit 6000 and reads various data (or information) from the storage unit 6000.

[0109] <Functional Configuration of a PC as a Communication Terminal> Next, the functional configuration of PC7 will be described in detail using Figures 12 and 15. PC7 basically has the same functions as video conferencing terminal 3a. That is, as shown in Figure 14, PC7 has a transmitting / receiving unit 71, a receiving unit 72, an image / sound processing unit 73, a display control unit 74, a judgment unit 75, a creation unit 76, a calculation unit 77, a communication unit 78, and a storage / reading unit 79. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 501 in accordance with the PC7 program deployed from HD 504 onto RAM 503.

[0110] Furthermore, PC7 has a storage unit 7000 constructed from ROM 502, RAM 503, and HD 504 as shown in Figure 12. This storage unit 7000 contains an image type management DB 7001, an imaging device management DB 7002, a predetermined area management DB 7003, and an overlay position management DB 7004. Note that the image type management DB 7001, imaging device management DB 7002, predetermined area management DB 7003, and overlay position management DB 7004 have the same data structure as image type management DB 3001a, imaging device management DB 3002a, predetermined area management DB 3003a, and overlay position management DB 3004a, respectively, so their explanations are omitted.

[0111] (Functional configuration of a PC as a communication terminal) The transmit / receive unit 71 of PC7 is mainly implemented by the processing of the network interface 509 and CPU 501 shown in Figure 12, and performs the same functions as the transmit / receive unit 31a.

[0112] The reception unit 72 is mainly implemented by the keyboard 511, mouse 512, and CPU 501, and performs the same functions as the reception unit 32a. The image / sound processing unit 73 is mainly implemented by instructions from the CPU 501, and performs the same functions as the image / sound processing unit 33a. The display control unit 74 is mainly implemented by the CPU 501, and performs the same functions as the display control unit 34a. The judgment unit 75 is mainly implemented by the CPU 501, and performs the same functions as the judgment unit 35a. The creation unit 76 is mainly implemented by the CPU 501, and performs the same functions as the creation unit 36a. The calculation unit 77 is mainly implemented by the CPU 501, and performs the same functions as the calculation unit 37a. The communication unit 78 is mainly implemented by the CPU 501, and performs the same functions as the communication unit 38a. The memory / read unit 79a is implemented by the CPU 501 and stores various data (or information) in the memory unit 7000 and reads various data (or information) from the memory unit 7000.

[0113] <Smartphone Functionality Configuration> Next, the functional configuration of the smartphone 9 will be described in detail using Figures 13 and 14. The smartphone 9 basically has the same functions as the video conferencing terminal 3a. That is, as shown in Figure 14, the smartphone 9 has a transmitting / receiving unit 91, a receiving unit 92, an image / sound processing unit 93, a display control unit 94, a judgment unit 95, a creation unit 96, a calculation unit 97, a communication unit 98, and a storage / reading unit 99. Each of these units is a function or means realized by any of the components shown in Figure 13 operating according to instructions from the CPU 901 that follow the smartphone 9 program deployed from the EEPROM 904 onto the RAM 903.

[0114] Furthermore, the smartphone 9 has a storage unit 9000 constructed from ROM 902, RAM 903, and EEPROM 904 as shown in Figure 13. This storage unit 9000 contains an image type management DB 9001, an imaging device management DB 9002, a predetermined area management DB 9003, and an overlay position management DB 9004. Note that the image type management DB 9001, imaging device management DB 9002, predetermined area management DB 9003, and overlay position management DB 9004 have the same data structure as image type management DB 3001a, imaging device management DB 3002a, predetermined area management DB 3003a, and overlay position management DB 3004a, respectively, so their explanations will be omitted.

[0115] (Smartphone function configurations) The transmitting / receiving unit 91 of the smartphone 9 is mainly realized by the processing of the long-distance communication circuit 911 and CPU 901 shown in Figure 13, and performs the same functions as the transmitting / receiving unit 31a.

[0116] The reception unit 92 is mainly implemented by the processing of the touch panel 921 and the CPU 901, and performs the same functions as the reception unit 32a.

[0117] The image / sound processing unit 93 is mainly implemented by instructions from the CPU 901 and performs the same functions as the image / sound processing unit 33a. The display control unit 94 is mainly implemented by the processing of the CPU 901 and performs the same functions as the display control unit 34a. The judgment unit 95 is mainly implemented by the processing of the CPU 901 and performs the same functions as the judgment unit 35a. The creation unit 96 is mainly implemented by the processing of the CPU 901 and performs the same functions as the creation unit 36a. The calculation unit 97 is mainly implemented by the processing of the CPU 901 and performs the same functions as the calculation unit 37a. The communication unit 98 is mainly implemented by the processing of the CPU 901 and performs the same functions as the communication unit 38a. The storage / reading unit 99 is implemented by the processing of the CPU 901 and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000.

[0118] <<Processing or operation of the embodiment>> Next, the processing or operation of this embodiment will be described using Figures 25 to 36.

[0119] <Processing of participation> First, we will explain the process of joining a specific communication session using Figures 25 and 26. Figure 25 is a sequence diagram showing the process of joining a specific communication session. Figure 26 is a diagram showing the screen for selecting a communication session (virtual meeting room).

[0120] First, when a user at site A (for example, user A1) performs an operation on the video conferencing terminal 3a to display the selection screen for a communication session (virtual meeting room), the reception unit 32a receives the operation to display the selection screen, and the display control unit 34a displays the selection screen shown in Figure 26 on the display 4a (step S21). This selection screen displays selection buttons b1, b2, b3, etc., which indicate the virtual meeting rooms R1, R2, R3, etc. that are the target of selection. In addition, each selection button b1, etc., is associated with each session ID.

[0121] Here, when user A1 selects the desired selection button for the virtual meeting room (in this case, selection button b1), the reception unit 32a accepts the selection of the communication session (step S22). Then, the transmitting / receiving unit 31a sends a request to join the virtual meeting room to the communication management system 5 (step S23). This join request includes the session ID indicating the communication session accepted in step S22, and the IP address of the requesting terminal, the video conferencing terminal 3a. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the join request.

[0122] Next, the storage / reading unit 59 performs the process of joining the communication session by adding the IP address received in step S23 to the field of the participating terminal IP address in the record with the same session ID as the session ID received in step S23 in the session management DB 5001 (see Figure 21) (step S24). Then, the transmitting / receiving unit 51 sends a join request response to the video conferencing terminal 3a (step S25). This join request response includes the session ID received in step S23 and the result of the join process. As a result, the transmitting / receiving unit 31a of the video conferencing terminal 3a receives the join request response. The following describes the case when the join process is successful.

[0123] <Image type information management process> Next, we will explain the image type information management process using Figure 27. Figure 27 is a sequence diagram showing the image type information management process.

[0124] First, when a user at site A (for example, user A1) connects the USB cable of the cradle 2a, with the camera 1a attached, to the video conferencing terminal 3a, the memory / read unit 19a of the camera 1a reads the GUID of its own device (camera 1a) stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of its own device to the communication unit 38a of the video conferencing terminal 3a (step S51). As a result, the communication unit 38a of the video conferencing terminal 3a receives the GUID of the camera 1a.

[0125] Next, the determination unit 35a of the video conferencing terminal 3a determines the image type (step S52) by checking whether the same vendor ID and product ID as those in the GUID received in step S51 are managed in the imaging device management DB 3002a (see Figure 17). Specifically, if the same vendor ID and product ID are managed in the imaging device management DB 3002a, the determination unit 35a determines that the imaging device 1a is an imaging device that captures special images (in this case, a 360-degree panoramic image). Conversely, if the same vendor ID and product ID are not managed in the imaging device management DB 3002a, the determination unit 35a determines that the imaging device 1a is an imaging device that captures general images.

[0126] Next, the storage / reading unit 39a stores the IP address of the source terminal (video conferencing terminal 3a) and the image type information, which is the result of the determination made in step S52, in association with the image type management DB 3001a (see Figure 16) (step S53). In this state, the image data ID is not associated. The image type information is, for example, the source name determined according to a predetermined naming convention, or the image type (general image, special image).

[0127] Next, the transmitting / receiving unit 31a sends a request to the communication management system 5 for additional image type information (step S54). This request for additional image type information includes the IP address of the transmitting terminal, which was stored in step S53, and the image type information. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request for additional image type information.

[0128] Next, the storage and reading unit 59 of the communication management system 5 uses the IP address of the source terminal received in step S54 as a search key to search the session management DB 5001 (see Figure 21) and read the corresponding session ID (step S55).

[0129] Next, the generation unit 56 generates a unique image data ID (step S56). Then, the storage / reading unit 59 stores the session ID read in step S55, the image data ID generated in step S56, and the IP address and image type information of the source terminal received in step S54 as a new record in the image type management DB 5002 (see Figure 22) (step S57). Then, the transmitting / receiving unit 51 transmits the image data ID generated in step S56 to the video conferencing terminal 3a. As a result, the transmitting / receiving unit 31a of the video conferencing terminal 3a receives the image data ID (step S58).

[0130] Next, the memory and retrieval unit 39a of the video conferencing terminal 3a stores the image data ID received in step S58 in the image type management DB 3001a (see Figure 16), associating it with the IP address of the source terminal, which is the local terminal (video conferencing terminal 3a), and the image type information stored in step S53 (step S59).

[0131] Meanwhile, the transmitting / receiving unit 51 of the communication management system 5 sends an additional notification of image type information to another communication terminal (in this case, a smartphone 9) (step S60). This additional notification of image type information includes the image data ID generated in step S56, as well as the IP address of the source terminal (video conferencing terminal 3a) and the image type information stored in step S53. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the additional notification of image type information. The destination of the transmitting / receiving unit 51 is another IP address associated with the same session ID as the IP address of the video conferencing terminal 3a in the session management DB 5001 (see Figure 21). In other words, the destination is another communication terminal that is in the same virtual conference room as the video conferencing terminal 3a.

[0132] Next, the memory and retrieval unit 99 of the smartphone 9 stores the image data ID, the IP address of the sending terminal, and the image type information received in step S60 as a new record in the image type management DB 9001 (see Figure 16) (step S61). Similarly, additional communication regarding image type information is sent to other communication terminals, the video conferencing terminal 3d and the PC 7, and the information is stored in the image type management DBs 3001d and 7001, respectively, on the video conferencing terminal 3d and the PC 7. Thus, each communication terminal can share the same information in its respective image type management DBs 3001a, 3001d, 7001, and 9001.

[0133] <Communication processing of captured image data> Next, Figures 28 to 36 will be used to explain the communication processing of captured image data and document image data in video calls. Figure 28 is an illustrative diagram showing the state of a video call using location A as an example.

[0134] As shown in Figure 28, the video conferencing terminal 3a displays the document images generated by the PC 6 superimposed on the 360-degree panoramic image captured by the camera 1a on the display 4. A whiteboard W is also installed on the right side of Figure 28, where users A1 and others can write text, draw pictures, etc.

[0135] Next, using Figure 29, we will explain the process by which the captured image data, sound data, and document image data obtained at location A shown in Figure 28 are transmitted to other communication terminals (smartphone 9, PC 7, video conferencing terminal 3d) via the communication management system 5. Figure 29 is a sequence diagram showing the communication process of captured image data, sound data, and document image data in a video call.

[0136] First, the communication unit 18a of the shooting device 1a transmits captured image data obtained by shooting subjects, scenery, etc., and sound data obtained by collecting sound to the communication unit 38a of the video conferencing terminal 3a (step S101). In this case, since the shooting device 1a is a device that can obtain two hemispherical images that will become the basis of a full-sphere panoramic image, the captured image data consists of data from the two hemispherical images, as shown in Figures 3(a) and (b). As a result, the communication unit 38a of the video conferencing terminal 3a receives the captured image data and sound data.

[0137] Next, the communication unit 68 of PC6 transmits the data image data to be displayed by the display control unit 64 to the communication unit 38a of the video conferencing terminal 3a (step S102).

[0138] Next, the transmitting / receiving unit 31a of the video conferencing terminal 3a transmits the captured image data, sound data, and document image data sent from the imaging device 1a to the communication management system 5 (step S103). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the captured image data, sound data, and document image data. Here, the captured image data includes an image data ID to identify the captured image data to be transmitted.

[0139] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits captured image data, sound data, and reference image data to the communication terminals (smartphone 9, PC 7, video conferencing terminal 3d) participating in the same video call as the video conferencing terminal 3a (steps S104, S105, S106). In each of these transmissions, the transmission of captured image data includes an image data ID to identify the captured image data to be transmitted. As a result, the transmitting / receiving unit 91 of the smartphone 9, the transmitting / receiving unit 71 of the PC 7, and the transmitting / receiving unit 31d of the video conferencing terminal 3d receive the captured image data and image data ID, and also receive the sound data and reference image data.

[0140] Next, Figure 30 will be used to explain an example of the display on display 917 at site B. Figure 30 shows an example of the display on the display at site D. Of these, Figure 30(a) shows the case where the captured image data sent from the shooting device 1a at site A via the video conferencing terminal 3a, and the captured image data sent from the shooting device 1b at site B are displayed as is, without creating a full-spherical panoramic image or a predetermined area image. On the other hand, Figure 30(b) shows the case where a full-spherical panoramic image and a predetermined area image are created from the captured image data sent from the shooting devices 1a and 1b. Note that the image of site A is displayed in the left display area (layout number "1") of display 4d, and the image of site B is displayed in the upper right display area (layout number "2"). Furthermore, the image of site C is displayed in the middle right display area (layout number "3") of display 4d, and the image of site D (own site) is displayed in the lower right display area (layout number "4"). The display area for layout number "1" is the primary display area, while the display areas for layout numbers "2," "3," and "4" are secondary display areas. The images in the primary and secondary display areas can be changed on each communication terminal. Typically, at each location, the primary display area displays an image of the location where the main person in the video call is located.

[0141] Here, when the captured image data sent from the imaging devices 1a and 1b, which are capable of capturing full-spherical panoramic images, is displayed as is, as shown in Figure 30(a), the images of base A and base B are displayed as the front hemisphere image and the rear hemisphere image, respectively, as shown in Figures 3(a) and (b).

[0142] In response, the image and sound processing unit 93 creates a full-sphere panoramic image from the captured image data output by the imaging devices 1a and 1b, which can obtain two hemispherical images that will form the basis of the full-sphere panoramic image. Furthermore, it creates a predetermined region image, and as shown in Figure 30(b), the predetermined region image, which is a planar image, is displayed. Note that at site C, the imaging device 8 that obtains the general image takes the picture, and at site D, the video conferencing terminal 3d that obtains the general image also takes the picture, so in both Figures 30(a) and (b), the general image (in this case, a planar image) is displayed.

[0143] Furthermore, users at each location can change the predetermined region of the image within the same 360-degree panoramic image. For example, user B1 can operate the touch panel 921, which allows the reception unit 92 to receive a request to move the predetermined region image, and the display control unit 94 can then shift, rotate, shrink, or enlarge the predetermined region image. As a result, as shown in Figure 30(b), the predetermined region image, which initially displays some users A1 and A2 at location A, can be changed to display a predetermined region image as shown in Figure 30(c). Specifically, Figure 30(c) shows the state in the captured image of location A shown in Figure 28, where the predetermined region image has been changed from one including users A1 and A2 to one including the whiteboard W.

[0144] The 360-degree icons 191 and 192 shown in Figures 30(b) and (c) are examples of special image identification icons used to indicate that a given region T is a designated area image representing a part of a 360-degree panoramic image. The display position of the 360-degree icons 191 and 192 is not limited to the upper right; it can be anywhere, such as the upper left, lower left, or lower right. Furthermore, the types of 360-degree icons 191 and 192 are not limited to those shown in Figures 30(b) and (c). In addition, instead of the 360-degree icons 191 and 192, the text "360-degree image" or other words may be used, or a combination of icons and text may be used.

[0145] Next, using Figure 31, we will explain the processing in the image communication system when a predetermined region image is displayed as in Figure 30(b), and when the predetermined region image is changed from Figure 30(b) to Figure 30(c). Figure 31 is a sequence diagram showing the process of sharing predetermined region information. In Figure 31, the video conferencing terminal 3a at site A is the third communication terminal, the video conferencing terminal 3d at site D is another communication terminal, and the smartphone 9 at site B is a communication terminal (its own terminal).

[0146] First, as shown in Figure 30(b), when user D1 or others at site D use video conferencing terminal 3d to display a predetermined area image of site A, the transmitting / receiving unit 31d of video conferencing terminal 3d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S111). This predetermined area information includes the IP address of video conferencing terminal 3a, which is the source of the captured image data, and the IP address of video conferencing terminal 3d, which is the destination of the captured image data (source of the predetermined area information). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the predetermined area information.

[0147] Next, the storage and reading unit 59 of the communication management system 5 stores the predetermined area information received in step S111, along with the source and destination IP addresses, in the predetermined area management DB 5003 (step S112). Note that the processing in steps S111 and S112 is performed each time the predetermined area image is changed on the video conferencing terminal 3d, as shown in Figures 30(b) to 30(c).

[0148] Next, the storage / reading unit 59 of the communication management system 5 reads the most recent set of predetermined area information and IP address set stored in the predetermined area management DB 5003 at regular intervals (for example, every 30 seconds) (step S113). Then, the transmitting / receiving unit 51 distributes (transmits) the predetermined area information, including the IP addresses read in step S113, to other communication terminals (video conferencing terminal 3a, smartphone 9, PC 7) that are making the same video call as the video conferencing terminal 3d, which is the source of the predetermined area information (steps S114, S116, S118). As a result, the transmitting / receiving unit 31a of the video conferencing terminal 3a receives the predetermined area information. Then, the storage / reading unit 39a stores the predetermined area information and IP addresses received in step S114 in the predetermined area management DB 3003a while associating them (step S115). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives predetermined area information, the storage / reading unit 99 stores the predetermined area information received in step S116 and its association with each IP address in the predetermined area management DB 9003 (step S117). Furthermore, in the PC 7, after the transmitting / receiving unit 71 receives predetermined area information, the storage / reading unit 79 stores the predetermined area information received in step S118 in the predetermined area management DB 7003 and its association with each IP address that was also received (step S119).

[0149] Next, we will explain other processes for sharing predetermined area information using Figure 32. Figure 32 is a sequence diagram showing other examples of the processes shown in Figure 31, illustrating other processes for sharing predetermined area information. In Figure 32, the video conferencing terminal 3a at site A is a communication terminal (local terminal), and the video conferencing terminal 3d at site D is another communication terminal.

[0150] In the process shown in Figure 31 above, the communication management system 5 temporarily manages the predetermined area information transmitted from each communication terminal (see S112) and transmits the predetermined area information to each communication terminal other than the source at regular intervals (see S114-S119). In contrast, in this embodiment, as shown in Figure 33, the communication terminal that transmits the captured image data (here, the video conferencing terminal 3a) temporarily manages the predetermined area information on behalf of the communication management system 5 (see S213) and transmits the predetermined area information to each communication terminal other than its own at regular intervals (see S215-S221). That is, in this embodiment, the communication terminal that transmits the captured image data manages how the captured image data transmitted by its own terminal (here, the video conferencing terminal 3a) is displayed as a predetermined area image of a predetermined area T1 on other communication terminals.

[0151] Note that the process shown in Figure 32 is implemented using the same hardware configuration as the process shown in Figure 31. Therefore, a description of the hardware configuration required to implement the process in Figure 32 will be omitted.

[0152] In the process shown in Figure 32, first, when user D1 or others at site D use video conferencing terminal 3d to display a predetermined area image of site A, the transmitting / receiving unit 31d of video conferencing terminal 3d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S211). This predetermined area information includes the IP address of video conferencing terminal 3a, which is the source of the captured image data, and the IP address of video conferencing terminal 3d, which is the destination of the captured image data (source of the predetermined area information). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the predetermined area information.

[0153] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits predetermined area information, including each IP address received in step S211, to the video conferencing terminal 3a, which is the source of the captured image data (step S212). As a result, the transmitting / receiving unit 31a of the video conferencing terminal 3a receives the predetermined area information.

[0154] Next, the memory and read unit 39a of the video conferencing terminal 3a stores the predetermined area information received in step S212, along with the source and destination IP addresses, in the predetermined area management DB 3003a (step S213). This process in step S213 manages how the captured image data transmitted by its own terminal (in this case, the video conferencing terminal 3a) is displayed on other communication terminals. Note that the processes in steps S211 to S213 are performed each time the predetermined area image is changed on the video conferencing terminal 3d.

[0155] Next, the memory and retrieval unit 39a of the video conferencing terminal 3a reads the most recent set of predetermined area information and IP addresses from the predetermined area management DB 3003a (the set that was stored most recently) at regular intervals (for example, every 30 seconds) (step S214). Then, the transmitting and receiving unit 31a transmits the predetermined area information, including the IP addresses read in step S214, to the communication management system 5 (step S215). As a result, the transmitting and receiving unit 51 of the communication management system 5 receives the predetermined area information.

[0156] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits (distributes) predetermined area information, including each IP address received in step S215, to each communication terminal (video conferencing terminal 3d, smartphone 9, PC 7) (steps S216, S218, S220). As a result, the transmitting / receiving unit 31d of the video conferencing terminal 3d receives the predetermined area information. The storage / reading unit 39d then stores the predetermined area information received in step S216 in the predetermined area management DB 3003d, associating it with each IP address that was also received (step S217). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives the predetermined area information, the storage / reading unit 99 stores the predetermined area information received in step S218 in the predetermined area management DB 9003, associating it with each IP address that was also received (step S219). Furthermore, in PC7, after the transmitting / receiving unit 71 receives the predetermined area information, the storage / reading unit 79 stores the predetermined area information received in step S220 in the predetermined area management DB 7003, associating it with each IP address that was also received (step S221).

[0157] As a result, the designated area information indicating the changed designated area image at site A is transmitted to the communication terminals at other sites B, C, and D, which are in the same video call, thereby sharing the designated area information indicating the designated area image currently displayed at site A with other sites B, C, and D. This process is carried out similarly when the designated area image is changed at sites B, C, and D, so that all sites in the same video call share the designated area information indicating the designated area image currently displayed at other sites.

[0158] Next, Figures 33 to 35 will be used to explain how to set the superposition position of the source image. Figure 33 is a sequence diagram showing the processing in the image communication system when changing the superposition position in which the source image is superimposed on the full-sphere panoramic image at base A, which is the image distribution source.

[0159] First, the reception unit 32a accepts the selection of a reference image in response to an operation by user A1 or others at base A (step S71). One example of how to select an image is to perform an operation such as "right-clicking," "double-clicking," or "entering the appropriate key" on the reference image portion of the preview screen of the 360-degree panoramic image being distributed by user A1 or others at base A. In the case of a touch-enabled terminal, this can also be achieved by operations such as "long-tapping" or "double-tapping" on the normal image within the preview area. Next, the reception unit 32a accepts a change in the superimposed position of the reference image in response to an operation by user A1 or others (step S72).

[0160] Next, the reception unit 32a receives the determination of the superposition position of the material image in response to the operation of user A1, etc., and the calculation unit 37a generates superposition position information (step S73). As an example of how to determine the superposition position, it may be considered automatically determined after the operation in step S72 is completed, or it may be determined by pressing a confirmation button on the screen separately or by performing a predetermined key operation.

[0161] Here, an example of a screen in which user A etc. changes the superimposed position of the reference image will be explained using Figure 34. Figure 34 is a preview screen at location A of a 360-degree panoramic image. The preview screen referred to here is a screen used to check the video being distributed by one location at one's own location without going through the communication network 100. As shown in Figure 34, the preview screen shows a predetermined area image of the 360-degree panoramic image that includes users A2, A4 and whiteboard W. Text and pie charts are written on whiteboard W.

[0162] Figure 34 further shows area R1, indicated by a dotted line, and area R2, indicated by a solid line. Areas R1 and R2 show the overlapping position before and after the change, respectively. By changing the overlapping position of the document image to area 2, which is an area within whiteboard W, the bar graph, which is a document image displayed by PC6, can be displayed alongside other text and pie charts written on whiteboard W, making it easier for meeting participants to see.

[0163] Using Figure 34, the operations of user A1, etc., corresponding to the aforementioned steps 71 to 73, will be explained. First, user A1, etc., selects area R1 (step S71), then drags and drops in the direction of the arrow to change the superposition position of the reference image (step 72). Then, with the image moved to the desired area R2, the operation to determine the new superposition position is performed (step 73).

[0164] Note that the method for changing the superimposed position is not limited to "drag and drop" as shown in Figure 34. As an alternative, the coordinates of the start and end points of the superimposed image may be changed by predetermined key operations, or, in the case of touch-enabled devices, by pinch-in and pinch-out operations. Also, if the sizes of the source image before and after the change, i.e., the sizes of region R1 and region R2, are different, the image may be enlarged or reduced to fit the changed size. Furthermore, the dotted line frame and arrow indicating region R1 may or may not be displayed to assist the user in their operation.

[0165] Continuing the explanation of Figure 33, the communication unit 68 then transmits the modified superimposed position information to the transmitting / receiving unit 51 of the communication system 5 (step S74). This superimposed position information includes the superimposed position coordinates, the IP address of the video conferencing terminal 3a which is the source of the captured image data, and the IP address of the video conferencing terminal 3d which is the destination of the captured image data (source of the superimposed position information). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the superimposed position information.

[0166] Next, the transmitting / receiving unit 51 distributes (transmits) the superimposed location information to other communication terminals (video conferencing terminal 3d, smartphone 9, PC 7) that are having the same video call as the video conferencing terminal 3d, which is the source of the predetermined area information (steps S75, S77, S79). As a result, the transmitting / receiving unit 31d of the video conferencing terminal 3d receives the superimposed location information. Then, the storage / reading unit 39a stores the superimposed location information received in step S75 in the superimposed location management DB 3004d, associating it with the source address (step S76). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives the superimposed location information, the storage / reading unit 99 stores the superimposed location report received in step S77 in the superimposed location management DB 9004, associating it with the source IP address (step S78). Furthermore, in PC7, after the transmitting / receiving unit 71 receives the superimposed position information, the storage / reading unit 79 stores the predetermined area information received in step S79 in the predetermined area management DB 7004, associating it with each IP address that was also received (step S80).

[0167] Figure 35 is a flowchart illustrating the display process of document images on the receiving terminal. Since the processing is the same for all receiving terminals, this section will explain the process for smartphone 9 at site B.

[0168] First, in the smartphone 9, the storage and reading unit 99 uses the image data ID received in step S106 shown in Figure 29 as a search key to search the image type management DB 9001 (see Figure 16) and read the corresponding image type information (source name) (step S131).

[0169] Next, the determination unit 95 determines whether the image type information read in step S131 indicates that it is a "special image" (step S132). Furthermore, if the determination result in step 132 is that it is a special image, the determination unit 95 checks whether it has received the source image data (step S133). The check in step S133 determines that if there is image data that is not managed by an image data ID, for example, it is source image data. If the determination result in step S133 is that it has not been received, i.e., No, the image / sound processing unit 93 displays the 360-degree image and terminates (step S137). On the other hand, if the determination result in step 133 is that it has been received, i.e., Yes, the determination unit 95 determines whether the source IP address of the image information is stored in the superimposed position management DB 9004d (step 135). If the result of the determination in step 135 is not stored, i.e., No, the image / sound processing unit 93 superimposes the source image data onto the initial superimposition position (default superimposition position), which is a predetermined superimposition position of the 360-degree image (step S136), and displays the 360-degree image (step S137). If the result of the determination in step 135 is stored, i.e., Yes, the storage / reading unit 99 obtains the superimposition position from the superimposition position management DB 9004d (step S138). Next, the image / sound processing unit 93 superimposes the source image data onto the acquired superimposition position of the 360-degree image (step 139), and displays the 360-degree image (step S137).

[0170] Figure 36 shows an example of the display screen at site B after the display processing described in Figure 35 has been executed. As shown in Figure 36, at site B, the data image (superimposed image) is displayed in region R2, which is the superimposed position where the superimposed image was modified at site A in Figure 34.

[0171] As a result, the superimposed position information modified at site A is transmitted to the communication terminals at other sites B, C, and D, which are also in the same video call, so that the document screen displayed at site A is displayed and shared at sites B, C, and D in the same position.

[0172] <Image quality control of captured images and reference images> Figures 36 to 39 illustrate the image quality control of captured images and reference images performed by the communication management system 5. Figure 37 is a sequence diagram showing the process of changing the image quality. Figure 38 shows the case when a predetermined region image including the reference image is changed. Figure 39 shows the predetermined region after the change.

[0173] First, the calculation unit 97 of the smartphone 9 calculates the display ratio in the display area of ​​base A with respect to the predetermined area image of base A shown in Figure 36 and the superimposed image (reference image) R2 (step S151). In Figure 36, the entire superimposed image R2 is displayed.

[0174] Next, the transmitting / receiving unit 91 of the smartphone 9 transmits percentage information, which is calculated by the calculation unit 97, to the communication management system 5 (step S152). At this time, the IP address of the communication terminal that is the source of the image transmission (in this case, the video conferencing terminal 3a) and the IP address of the communication terminal that is the destination of the image transmission (in this case, the smartphone 9) are also transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the percentage information and each IP address.

[0175] Next, in the communication management system 5, the storage and reading unit 59 associates the percentage information and each IP address received by the transmitting and receiving unit 51, and stores and manages them in the percentage management DB 5004 (step S153).

[0176] Next, similar to step S103 in Figure 29, the transmitting / receiving unit 51 of the communication management system 5 receives the captured image data, sound data, and document image data, all of which are accompanied by an image data ID, transmitted from the transmitting / receiving unit 31a of the video conferencing terminal 3a (step S154).

[0177] Next, the storage / reading unit 59 searches the ratio management DB 5004 using the IP address of the communication terminal that transmits the image (in this case, the video conferencing terminal 3a) and the IP address of the communication terminal that receives the image (in this case, the smartphone 9) as search keys to read the corresponding ratio information. The modification unit 53 then changes the image quality of the captured image data and the data image data received in step S154 according to the ratio indicated in the ratio information. In this case, if the ratio indicated in the ratio information is large, the image quality is increased; if the ratio is small, the image quality is decreased.

[0178] Next, the transmitting / receiving unit 51 transmits the captured image data, the reference image data, and the sound data according to the proportion to the smartphone 9 (step S156). In this case, the captured image data also includes an image data ID. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the captured image data, the reference image data, and the sound data.

[0179] Next, in the smartphone, the image / sound processing unit 93 performs image processing to superimpose a superimposed image (reference image) relating to the reference image data onto the overall image relating to the captured image data received by the transmitting / receiving unit 91 (step S157). Then, the display control unit 94 displays an image on the display 917 as shown in Figure 35.

[0180] By the way, if users B1 and B2 at site B change the predetermined area image at site A, as shown in Figure 38, the ratio of the overall image to the reference image (overlaid image) in the display area of ​​the image at site A changes. In Figure 38, the proportion of the overlaid image R2 is smaller than in Figure 36. Figure 39 shows the range of the display area T at site A in the case of Figure 38. In such a case, processing from step S151 is restarted. This is because the proportion of the overlaid image R2 is smaller. The image quality of the captured image data transmitted by step S156 increases, while the image quality of the reference image data transmitted in the same manner decreases.

[0181] Furthermore, users B1 and B2 at site B can change the predetermined position where the superimposed screen R3 is superimposed by using the touch panel 921. In this case, the reception unit 92 receives the change in the predetermined position, and the image / sound processing unit 93 superimposes the data image at the predetermined position.

[0182] <<Main Effects of This Embodiment>> As described above, this embodiment has the effect of resolving the problem that the receiving user may overlook the material images that the transmitting user wants the user to pay attention to, through the processing of steps S151 to S154.

[0183] Furthermore, communication terminals such as video conferencing terminal 3a can create a full-spherical panoramic image based on the image data ID sent along with the image data, and then create a predetermined region image based on the corresponding image type information. This has the effect of preventing the display of the front hemisphere image and the rear hemisphere image, as shown in Figure 30(a).

[0184] Furthermore, according to this embodiment, in a conferencing system or the like where a receiver overlays another image (the source image) onto a part of a given image (the destination image), the sender can change the display position of the source image on the destination image. Therefore, if the source image is overlaid in a location unintended by the sender, or if the sender wants to overlay the image in a different location during a meeting, changing the overlay position makes it possible to create an image that is easier to see and understand.

[0185] 〔supplement〕 In the above embodiment, a panoramic image of three dimensions was described as an example of a panoramic image to be superimposed on, but a two-dimensional panoramic image may also be used.

[0186] Furthermore, in the above embodiment, the communication management system 5 relays predetermined area information transmitted from each communication terminal, but the system is not limited to this, and each communication terminal may directly send and receive predetermined area information from each other.

[0187] Furthermore, each of the functions of the above embodiments can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), SOCs (System on a chip), GPUs (Graphics Processing Units), and conventional circuit modules designed to execute each of the functions described above. [Explanation of symbols]

[0188] 1a Imaging device 1b Imaging device 3a Video conferencing terminal (an example of another communication terminal) 3D video conferencing terminal (an example of a communication terminal) 4a Display 4D display (an example of a display method) 5. Communication Management System 6 PC 7 PC 8. Imaging device 9. Smartphone (an example of another communication device) 91 Transmitting and Receiving Unit (Example of transmitting means, example of receiving means) 92 Reception area (an example of reception procedures) 93 Image and sound processing unit (an example of image processing means) 94 Display Control Unit (An example of display control means) 97 Calculation Unit (Example of Calculation Means) 53 Modified section (an example of modification method) 3004a Overlap Position Management Database (An example of an overlap position management means) 3004d Overlap Position Management Database (An example of an overlap position management method) 5004 Percentage Management Database (An example of a percentage management method) 7004 Overlap Position Management Database (An example of an overlap position management method) 9004 Overlap Position Management Database (An example of an overlap position management method) [Prior art documents] [Patent Documents]

[0189] [Patent Document 1] Japanese Patent Publication No. 2012-178135 [Patent Document 2] Japanese Patent Publication No. 2011-223076 [Patent Document 3] Japanese Patent Publication No. 2012-156820

Claims

1. A communication terminal capable of displaying a predetermined region image, which is an image of a predetermined region within an overall image, on a display means, A receiving means for receiving the data of the overall image, the data of the superimposed image, and superimposed position information indicating the position where the superimposed image is superimposed within the overall image, transmitted by another communication terminal. Image processing means for superimposing the superimposed image onto the overall image based on the superimposed position information, A display control means that causes a predetermined region image in the overall image to be displayed on a display means, It has, A communication terminal characterized in that the display control means causes the display means to display the predetermined region image and the superimposed image with image quality changed according to the ratio of the predetermined region image and the superimposed image within the display area of ​​the display means.

2. A communication terminal according to claim 1, A receiving means that receives a request to change the predetermined area image displayed on the display means within the overall image, A communication terminal characterized in that, when the ratio of the predetermined region image and the superimposed image within the display area of ​​the display means is changed due to the change of the predetermined region image, the display control means causes the display means to display the predetermined region image and the superimposed image with an image quality changed according to the changed ratio of the predetermined region image and the superimposed image in the display means.

3. A communication terminal according to claim 1 or 2, further, A calculation means for calculating the ratio of the predetermined region image and the superimposed image in the display means, A transmission means for transmitting ratio information indicating the ratio calculated by the calculation means to a communication management system that relays communication with other communication terminals, A receiving means receives the data of the modified overall image and the modified predetermined region image, which have been transmitted by the transmission means, after the communication management system has received the ratio information transmitted by the transmission means and has modified the image quality of the overall image and the superimposed image according to the ratio indicated in the ratio information. It has, A communication terminal characterized in that the display control means causes the display means to display the overall image and the superimposed image, whose image quality has been changed according to the ratio of the modified predetermined region image and the superimposed image received by the receiving means.

4. The communication terminal described in claim 3, The aforementioned communication management system, An image processing system characterized by having the following features.

5. The communication terminal according to any one of claims 1 to 3, characterized in that the overall image is a full-sphere panoramic image.

6. The communication terminal according to any one of claims 1 to 3, characterized in that the communication terminal is a video conferencing terminal, a personal computer, a smartphone, a digital television, a smartwatch, or a car navigation device.

7. A display method performed by a communication terminal capable of displaying a predetermined region image, which is an image of a predetermined region within an overall image, on a display means, The aforementioned communication terminal is A receiving step of receiving the data of the overall image, the data of the superimposed image, and the superimposed position information indicating the position in which the superimposed image is superimposed on the overall image, transmitted by another communication terminal. An image processing step of superimposing the superimposed image onto the overall image based on the superimposed position information, A display control step of displaying a predetermined region image in the overall image on a display means, Execute, The communication terminal is characterized in that the display control step includes a process of causing the display means to display the predetermined region image and the superimposed image with image quality changed according to the ratio of the predetermined region image and the superimposed image within the display area of ​​the display means.

8. A program for causing a computer to perform the method described in claim 7.

Images