Image communication system, communication management apparatus, communication management method, and storage medium

Abstract

To solve the problem that it is difficult for a user to grasp when and from which viewpoint the user views a photographed image when the user wants to review the photographed image after a lapse of time after distribution in a system for distributing the photographed image while performing photographing in a conventional technology.SOLUTION: In the communication management system 5, after the transmitter / receiver 51 receives the captured-image data-item including the image-data ID and the audio data-item transmitted from the image capturing device 1a (step S301), the image conversion processor 57 performs perspective transformation based on the captured-image data-item (recording file) and the time-series information (time stamp) of the predetermined area (step S404), and the transmitter / receiver 51 transmits a new predetermined-area image data-item (converted recording file) reflecting the viewing state of each of the users (viewers) (step S405).SELECTED DRAWING: Figure 33

Description

[Technical Field]

[0001] The present invention relates to an image communication system, a communication management device, a communication management method, and a program. [Background technology]

[0002] Conference systems that enable remote conferences between remote locations via a communication network such as the Internet are becoming widespread. In these conference systems, in a conference room where one of the parties, such as attendees, is present, a communication terminal of the remote conference system is used to capture and collect images of the conference room and audio, such as speech, of the parties, and converts these into digital data and transmits them to the communication terminal of the other party. This allows a video call to be conducted by displaying images on a display and outputting audio from a speaker in the other party's conference room, thereby enabling a conference between remote locations to be held in an environment similar to a real conference (see, for example, Patent Document 1).

[0003] In addition, a camera system has been disclosed in which multiple camera units, including one capable of using a 360-degree omnidirectional camera, are installed at an event venue to broadcast images in real time, and selection information relating to images selected by users from the broadcast images is sent to a server, and the server then sells the images edited by the server based on the selection information to users (see, for example, Patent Document 2). Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional technology, in a system that distributes images captured while shooting, when a user wants to review the captured images some time after distribution, there is a problem in that it is difficult for the user to grasp when and from what viewpoint the captured image was viewed. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the invention of claim 1 provides an image communication system including a photographing device that obtains photographed image data by photographing, a communication terminal that displays a predetermined area image that is an image of a predetermined area of ​​the photographed image related to the photographed image data, and a communication management device that manages predetermined area information that represents the predetermined area, wherein the communication management device has a receiving means that receives the photographed image data photographed by the photographing device and time series information that represents the date and time associated with the predetermined area information transmitted by the communication terminal, and a transmitting means that transmits to the communication terminal the transformed predetermined area image data that has been perspectively projected based on the received photographed image data and the time series information. [Effects of the Invention]

[0006] As described above, according to the present invention, in a system that distributes images captured while shooting, even when a user looks back at the captured images some time after distribution, the user can easily understand when and from what viewpoint the user viewed the captured images. [Brief explanation of the drawings]

[0007] [Figure 1] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. 1C is a plan view of the imaging device. [Figure 2] FIG. 10 is an image diagram of the imaging device in use. [Figure 3] (a) is a hemispherical image (before) taken with the imaging device, (b) is a hemispherical image (after) taken with the imaging device, and (c) is an image represented by the Mercator projection. [Figure 4] (a) A conceptual diagram showing how a sphere is covered with a Mercator image, and (b) a diagram showing a spherical panoramic image. [Figure 5] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere panoramic image is a three-dimensional sphere. [Figure 6]6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 7] 10 is a diagram showing the relationship between predetermined area information and an image of a predetermined area T. FIG. [Figure 8] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 9] 1 is a schematic diagram of an image communication system according to an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating the hardware configuration of the imaging device. [Figure 11] FIG. 1 is a hardware configuration diagram of a video conference terminal. [Figure 12] FIG. 1 is a hardware configuration diagram of a communication management system and a PC. [Figure 13] FIG. 1 is a diagram illustrating the hardware configuration of a smartphone. [Figure 14] FIG. 2 is a functional block diagram of a part of the image communication system. [Figure 15] FIG. 2 is a functional block diagram of a part of the image communication system. [Figure 16A] FIG. 10 is a conceptual diagram illustrating an image type management table. [Figure 16B] FIG. 10 is a conceptual diagram illustrating an image type management table. [Figure 16C] FIG. 10 is a conceptual diagram illustrating a photographing device management table. [Figure 16D] FIG. 10 is a conceptual diagram illustrating a predetermined area management table. [Figure 16E] FIG. 10 is a conceptual diagram illustrating a session management table. [Figure 16F] FIG. 10 is a conceptual diagram illustrating an image type management table. [Figure 16G] FIG. 10 is a conceptual diagram illustrating a predetermined area management table. [Figure 16H] FIG. 10 is a conceptual diagram showing a predetermined area management table in the case of post-conversion recording. [Figure 16I] FIG. 10 is a conceptual diagram illustrating a recording file storage table. [Figure 17] FIG. 10 is a sequence diagram illustrating a process of joining a specific communication session. [Figure 18] FIG. 10 is a diagram showing a selection screen for a communication session (virtual conference room). [Figure 19] FIG. 10 is a sequence diagram showing a management process of image type information. [Figure 20] 1A and 1B are conceptual diagrams showing the state of a video call, in which (a) shows a case where the photographing device 1a is not used, and (b) shows a case where the photographing device 1a is used. [Figure 21] FIG. 10 is a sequence diagram showing a communication process of captured image data and sound data during a video call. [Figure 22] 10A and 10B show examples of display on a display at a viewing location B, in which (a) shows a case where image data transmitted from the image capturing devices 1a and 1b is displayed as is without generating a celestial sphere panoramic image or a predetermined area image, (b) shows a case where a celestial sphere panoramic image and a predetermined area image are generated from the image data transmitted from the image capturing devices 1a and 1b, and (c) shows a case where the predetermined area image of (b) is changed. [Figure 23] FIG. 10 is a sequence diagram showing a process of sharing predetermined area information. [Figure 24] 10 is a flowchart showing a display process of a predetermined area image. [Figure 25] 10 is a diagram showing a method for deriving the position of the gaze point of another base in a predetermined area image of the own base. FIG. [Figure 26] (a) is a diagram showing the definition of angles, and (b) is a diagram showing the definition of angle ranges. [Figure 27] 10 shows an image of the main display area, which is an example of a display of a predetermined area image including a display direction mark. [Figure 28] 10 shows an image of the main display area, which is an example of a display of a predetermined area image including a gaze point mark. [Figure 29] 10 shows an image of the main display area, which is an example of a display of a predetermined area image including a gaze point mark and a display direction mark. [Figure 30] FIG. 10 is a sequence diagram showing another process for sharing fixed region information. [Figure 31]FIG. 10 is a sequence diagram showing a communication process of captured image data and sound data during a video call. [Figure 32] 10 is a flowchart showing a recording process for selecting live recording or post-conversion recording and live recording. [Figure 33] 10 is a flowchart showing a recording process in the case of post-conversion recording. [Figure 34] An example of a file selection screen on the app for viewing post-conversion recordings. [Figure 35] This is an example of a video playback screen for a post-conversion recording selected in the app. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and if there are overlapping parts, the description thereof will be omitted.

[0009] [First embodiment] The first embodiment will be described with reference to FIGS.

[0010] <<Outline of the embodiment>> <How to generate a spherical panoramic image> A method for generating a spherical panoramic image will be described with reference to FIGS.

[0011] First, the appearance of the imaging device 1 will be described with reference to Fig. 1. Fig. 1(a) is a left side view of the imaging device, Fig. 1(b) is a front view of the imaging device, and Fig. 1(c) is a plan view of the imaging device. The imaging device 1 is a spherical imaging device capable of capturing three-dimensional spherical panoramic images (images that serve as the basis for three-dimensional spherical panoramic images). This spherical imaging device also includes a digital camera. Furthermore, the sphere refers to a space of 360° in all directions, including up, down, left, and right.

[0012] As shown in FIG. 1(a), the photographing device 1 is small enough to be held in one hand. As shown in FIGS. 1(a), 1(b), and 1(c), an image sensor 103a is provided on the front side (front side) of the upper portion of the photographing device 1, and an image sensor 103b is provided on the rear side (rear side). These image sensors (image sensors) 103a and 103b are used in conjunction with optical components (e.g., fisheye lenses 102a and 102b in FIG. 10, which will be described later) capable of capturing hemispherical images (with an angle of view of 180° or more). As shown in FIG. 1(b), an operation unit 115, such as a shutter button, is provided on the surface of the photographing device 1 opposite the front side.

[0013] Next, the usage of the image capturing device 1 will be described with reference to Fig. 2. Fig. 2 is an image diagram of the image capturing device in use. As shown in Fig. 2, the image capturing device 1 is used, for example, by being held by a user and capturing images of subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image capturing element 103a and the image capturing element 103b shown in Fig. 1, respectively.

[0014] Next, an outline of the processing up to generating a celestial sphere panoramic image from an image captured by the image capturing device 1 will be described with reference to Fig. 3 and Fig. 4. Fig. 3(a) is a diagram showing a hemispherical image (front side) captured by the image capturing device, Fig. 3(b) is a diagram showing a hemispherical image (rear side) captured by the image capturing device, and Fig. 3(c) is a diagram showing an image expressed by Mercator projection (hereinafter referred to as "Mercator image"). Fig. 4(a) is a conceptual diagram showing a state in which a sphere is covered with a Mercator image, and Fig. 4(b) is a diagram showing a celestial sphere panoramic image.

[0015] As shown in Fig. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a fisheye lens 102a (described later). Also, as shown in Fig. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a fisheye lens 102b (described later). Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are combined by the photographing device 1 to generate a Mercator image as shown in Fig. 3(c).

[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted to cover the spherical surface as shown in FIG. 4(a), and a celestial sphere panoramic image as shown in FIG. 4(b) is generated. In this way, the celestial sphere panoramic image is expressed as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere panoramic image may be a still image or a video.

[0017] As described above, a spherical panoramic image is an image pasted to cover the spherical surface, which gives a sense of incongruity to people. Therefore, by displaying a predetermined region of the spherical panoramic image (hereinafter referred to as a "predetermined region image") as a flat image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to people. This will be described with reference to FIGS. 5 and 6.

[0018] FIG. 5 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is represented as a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user viewing a celestial sphere image CE displayed as a three-dimensional sphere. FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing an image of a predetermined area when displayed on a display. In FIG. 6(a), the celestial sphere image CE shown in FIG. 4 is represented by a three-dimensional sphere CS. If the celestial sphere image CE generated in this way is a 3D sphere CS, then the virtual camera IC is located inside the celestial sphere image CE as shown in FIG. 5. The predetermined area T in the celestial sphere image CE is an imaging area of ​​the virtual camera IC, and is specified by predetermined area information indicating the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE.

[0019] The predetermined area image Q shown in FIG. 6(a) is then displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined area image represented by the initial (default) predetermined area information. Note that the predetermined area information may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the position coordinates of the virtual camera IC. The following explanation will be given using the shooting direction (rH, rV) and angle of view (α) of the virtual camera IC.

[0020] The relationship between the predetermined region information and the image of the predetermined region T will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating the relationship between the predetermined region information and the image of the predetermined region T. As shown in FIG. 7, rH represents the horizontal radius, rV represents the vertical radius, and α represents the angle of view (Angle). That is, the posture of the virtual camera IC is changed so that the gaze point of the virtual camera IC, indicated by the shooting direction (rH, rV), becomes the center point CP of the predetermined region T, which is the shooting region of the virtual camera IC. The predetermined region image Q is an image of the predetermined region T in the celestial sphere image CE. f represents the distance from the virtual camera IC to the center point CP. L represents the distance between any vertex of the predetermined region T and the center point CP (2L is the diagonal). In FIG. 7, the trigonometric function represented by the following (Equation 1) generally holds.

[0021] L / f=tan(α / 2) (Equation 1) FIG. 8 is a diagram showing points in a three-dimensional Euclidean space in spherical coordinates. The position coordinates of the center point CP when expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius vector, polar angle, and azimuth angle, respectively. The radius vector r is the distance from the origin of the three-dimensional virtual space including the spherical panoramic image to the center point CP, and is therefore equal to f. FIG. 8 is a diagram showing these relationships. Hereinafter, the explanation will be given using the position coordinates CP (r, θ, φ) of the virtual camera IC.

[0022] <Outline of the image communication system> Next, an outline of the configuration of the image communication system of this embodiment will be described with reference to Fig. 9. Fig. 9 is a schematic diagram of the configuration of the image communication system of this embodiment.

[0023] 9, the image communication system of this embodiment is configured with photographing devices 1a and 1b, video conference terminals 3a and 3d, displays 4a and 4d, a communication management system 5, a PC (Personal Computer) 7, a photographing device 8, and a smartphone 9, and is capable of communication via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

[0024] Of these, the image capturing devices 1a and 1b are special digital cameras for capturing images of subjects, scenery, etc. to obtain two hemispherical images that are the basis for a celestial sphere panoramic image, as described above, while the image capturing device 8 is a general digital camera for capturing images of subjects, scenery, etc. to obtain a general planar image.

[0025] The video conference terminals 3a and 3d are dedicated to video conferences and display images of video calls on displays 4a and 4d, respectively, via wired cables such as USB (Universal Serial Bus) cables. The video conference terminal 3a normally captures images using a camera 312 (see FIG. 11 ), which will be described later. However, when the video conference terminal 3a is connected via a wired cable to a cradle 2a that holds the image capture device 1a, the image capture device 1a is given priority, and two hemispherical images that form the basis of a celestial sphere panoramic image can be obtained. When a wired cable is used, the cradle 2a not only communicates between the image capture device 1a and the video conference terminal 3a, but also serves to supply power to and support the image capture device 1a. Here, the image capture device 1a, the cradle 2a, the video conference terminal 3a, and the display 4a are all located at the same viewing site, viewing site A. Four users A1, A2, A3, and A4 are participating in the video call at viewing site A. On the other hand, the video conference terminal 3d and the display 4d are located at the same viewing location, viewing location D. At viewing location D, three users D1, D2, and D3 are participating in a video call.

[0026] The communication management system 5 manages and controls communications between the video conference terminals 3a and 3d, the PC 7, and the smartphone 9, and manages the types of image data (general images and special images) sent and received. Therefore, the communication management system 5 is also a communication control system. Here, the special images are spherical panoramic images. The communication management system 5 is installed at a service company that provides video communication services. The communication management system 5 may be constructed by a single computer, or may be constructed by multiple computers in which each unit (function, means, or storage unit) is divided and arbitrarily assigned. In the embodiment, the communication management system 5 functions as an example of a communication management device.

[0027] The PC 7 is capable of video calls by attaching the camera device 8. In this example, the PC 7 and the camera device 8 are located at the same viewing location, viewing location C. At viewing location C, one user C1 is participating in the video call.

[0028] The smartphone 9 displays an image of the video call on a display 917 (described later) provided on the smartphone 9. The smartphone 9 normally captures images using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 (described later) provided on the smartphone 9. However, the smartphone 9 can acquire two hemispherical image data that are the basis of the omnidirectional panoramic image acquired by the image capture device 1b using short-range wireless communication technology such as NFC (registered trademark, omitted hereinafter), Bluetooth (registered trademark, omitted hereinafter), or Wi-Fi (registered trademark, omitted hereinafter). When using short-range wireless communication technology, the cradle 2b only serves to supply power to and support the image capture device 1b. Note that the image capture device 1b, the cradle 2b, and the smartphone 9 are all located at the same viewing location, viewing location B. Two users B1 and B2 are also participating in the video call at viewing location B.

[0029] The video conference terminals 3 a and 3 d, the PC 7, and the smartphone 9 are examples of communication terminals. OpenGL ES is installed in each communication terminal, and the communication terminals can generate predetermined area information indicating a partial area of ​​a celestial sphere panoramic image and generate a predetermined area image from a celestial sphere panoramic image transmitted from another communication terminal.

[0030] The arrangement of the terminals (communication terminals), devices (displays, camera devices), and users shown in FIG. 9 is one example, and other examples may be used. For example, at the viewing location C, a camera device capable of capturing a celestial sphere panoramic image may be used instead of the camera device 8. The communication terminal also includes a digital television, a smart watch, a car navigation device, and the like. Hereinafter, any of the camera devices 1a and 1b will be referred to as "camera device 1." Any of the video conference terminals 3a and 3d will be referred to as "video conference terminal 3." Any of the displays 4a and 4d will be referred to as "display 4."

[0031] Furthermore, if the photographing device 1 can be directly connected to the communication network 100 via wireless communication or the like and can distribute data (information) to other devices, the image communication system may be configured not to include communication terminals such as a video conference terminal 3 or a smartphone 9.

[0032] <<Hardware configuration of the embodiment>> 10 to 13, the hardware configurations of the image capturing device 1, the video conference terminal 3, the communication management system 5, the PC 7, and the smartphone 9 of this embodiment will be described in detail. Note that the image capturing device 8 is a general camera, and therefore a detailed description thereof will be omitted.

[0033] <Hardware configuration of the imaging device> First, the hardware configuration of the image capturing device 1 will be described using Fig. 10. Fig. 10 is a hardware configuration diagram of the image capturing device. In the following, the image capturing device 1 is assumed to be an omnidirectional (omnidirectional) image capturing device using two image capturing elements, but the number of image capturing elements may be two or more. Furthermore, the image capturing device does not necessarily have to be a device dedicated to omnidirectional image capturing; an ordinary digital camera, smartphone, or the like may be equipped with an omnidirectional image capturing unit as an add-on, so that it has substantially the same functions as the image capturing device 1.

[0034] As shown in FIG. 10, the photographing device 1 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I / F 116, a communication unit 117, an antenna 117a, and an electronic compass 118.

[0035] Of these, imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b, each with a field angle of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each wide-angle lens. Imaging elements 103a and 103b include an image sensor such as a CMOS sensor or a CCD (Charge Coupled Device) sensor that converts optical images captured by fisheye lenses 102a and 102b into image data in the form of electrical signals and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers that set various commands and parameters required for the operation of the imaging elements.

[0036] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are connected via a serial I / F bus (such as an I2C bus) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to a CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an electronic compass 118, and the like.

[0037] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then synthesizes the image data to generate Mercator image data such as that shown in Figure 3(c).

[0038] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0039] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 1 have a preview display function or a function for displaying moving images on a display (for example, the display of the video conference terminal 3a). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0040] As will be described later, the imaging control unit 105 also functions as a synchronization control means that synchronizes the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the photographing device 1 is not provided with a display unit, but may be provided with a display unit.

[0041] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I / F bus and performs predetermined processing on the sound data.

[0042] The CPU 111 controls the overall operation of the imaging device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and processed Mercator image data.

[0043] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. The user operates the operation buttons to input various shooting modes, shooting conditions, etc.

[0044] The network I / F 116 is a general term for an interface circuit (such as a USB I / F) with external media such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. The Mercator image data stored in the DRAM 114 is recorded on external media via the network I / F 116, or transmitted to an external device such as the video conference terminal 3a via the network I / F 116 as needed.

[0045] The communication unit 117 communicates with an external device such as the video conference terminal 3a by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth, or Wi-Fi (Wireless Fidelity) via an antenna 117a provided in the image capturing device 1. The communication unit 117 can also transmit Mercator image data to an external device of the video conference terminal 3a.

[0046] The electronic compass 118 calculates the orientation and tilt (roll rotation angle) of the image capturing device 1 from the Earth's magnetism and outputs the orientation and tilt information. This orientation and tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of captured images. The related information also includes data such as the date and time the image was captured and the data size of the image data.

[0047] <Videoconferencing terminal hardware configuration> Next, the hardware configuration of the video conference terminal 3 will be described with reference to Fig. 11. Fig. 11 is a diagram of the hardware configuration of the video conference terminal. As shown in Fig. 11, the video conference terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, an operation button 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an image sensor I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319.

[0048] Of these, the CPU 301 controls the overall operation of the video conference terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as an IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and sound data. The SSD (Solid State Drive) 305 controls the reading and writing of various data from and to the flash memory 304 under the control of the CPU 301. Note that an HDD may be used instead of an SSD. The media I / F 307 controls the reading and writing (storage) of data from and to a recording medium 306, such as a flash memory. The operation button 308 is a button that is operated when selecting a destination for the video conference terminal 3, etc. The power switch 309 is a switch for turning the power of the video conference terminal 3 on and off.

[0049] The network I / F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 301. The image sensor I / F 313 is a circuit that controls the driving of the camera 312. The microphone 314 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 316 is a circuit that processes input and output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I / F 317 is a circuit that transmits image data to the external display 4 under the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a short-range wireless communication circuit such as NFC, Bluetooth, or Wi-Fi.

[0050] 11. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 301 shown in FIG.

[0051] The display 4 is a type of display means configured with a liquid crystal or organic electroluminescence (EL) display that displays an image of a subject, operation icons, etc. The display 4 is connected to the display I / F 317 by a cable 4c. This cable 4c may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0052] The camera 312 includes a lens and a solid-state imaging element that converts light into an electric charge to digitize an image (video) of a subject, and a CMOS sensor, CCD sensor, or the like is used as the solid-state imaging element. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 via a USB (Universal Serial Bus) cable, or the like. When an external camera is connected, the external camera is driven under the control of the CPU 301 in preference to the built-in camera 312. Similarly, when an external microphone or an external speaker is connected, the external microphone or external speaker is driven under the control of the CPU 301 in preference to the built-in microphone 314 or built-in speaker 315, respectively.

[0053] The recording medium 306 is detachable from the video conference terminal 3. The storage medium 306 is not limited to the flash memory 304, and may be an EEPROM (Electrically Erasable and Programmable ROM) or the like, as long as it is a non-volatile memory that reads or writes data under the control of the CPU 301.

[0054] <Communication management system, PC hardware configuration> Next, the hardware configuration of the communication management system 5 and the PC 7 will be explained using Fig. 12. Fig. 12 is a diagram of the hardware configuration of the communication management system and the PC. Note that since the communication management system 5 and the PC 7 are both computers with the same configuration, the following will explain the configuration of the communication management system 5, and will omit explanation of the configuration of the PC 7.

[0055] The communication management system 5 includes a CPU 501 for controlling the overall operation of the communication management system 5, a ROM 502 storing a program used to drive the CPU 501, such as an IPL, a RAM 503 used as a work area for the CPU 501, a HD 504 for storing various data such as programs for the communication management system 5, a HDD (Hard Disk Drive) 505 for controlling the reading or writing of various data from the HD 504 under the control of the CPU 501, a media drive 507 for controlling the reading or writing (storage) of data from a recording medium 506, such as a flash memory, a display 508 for displaying various information such as a cursor, menus, windows, characters or images, a network I / F 509 for data communication using the communication network 100, a keyboard 511 having a plurality of keys for inputting characters, numbers, various instructions, etc., a mouse 512 for selecting and executing various instructions, selecting a processing target, moving the cursor, etc., a CD-RW (Compact Disc - The CD-RW drive 514 controls the reading of various data from the CD-RW drive (ReWritable) 513, and bus lines 510 such as an address bus and a data bus for electrically connecting the above-mentioned components as shown in FIG. 12.

[0056] <Smartphone hardware configuration> Next, the hardware of the smartphone will be described with reference to Fig. 13. Fig. 13 is a hardware configuration diagram of the smartphone. As shown in Fig. 13, the smartphone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration / direction sensor 906, a media I / F 908, and a GPS receiver 909.

[0057] Of these, the CPU 901 controls the overall operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901, such as an IPL. The RAM 903 is used as a work area for the CPU 901. The EEPROM 904 reads and writes various data, such as smartphone programs, under the control of the CPU 901. The CMOS sensor 905 captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 901. The acceleration / azimuth sensor 906 is a type of sensor, such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, or an acceleration sensor. The media I / F 908 controls the reading and writing (storage) of data from and to a recording medium 907, such as a flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0058] The smartphone 9 also includes a long-distance communication circuit 911, a camera 912, an image sensor I / F 913, a microphone 914, a speaker 915, an audio input / output I / F 916, a display 917, an external device connection I / F 918, a short-distance communication circuit 919, an antenna 919a of the short-distance communication circuit 919, and a touch panel 921.

[0059] Of these, the long-distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 901. The image sensor I / F 913 is a circuit that controls the operation of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs audio. The sound input / output I / F 916 is a circuit that processes the input and output of audio signals between the microphone 914 and the speaker 915 under the control of the CPU 901. The display 917 is a type of display means such as an LCD or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 918 is an interface for connecting various external devices. The short-distance communication circuit 919 is a short-distance wireless communication circuit such as NFC, Bluetooth, or Wi-Fi. The touch panel 921 is a type of input means that allows a user to operate the smartphone 9 by pressing the display 917.

[0060] The smartphone 9 also includes a bus line 910. The bus line 910 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 901.

[0061] It should be noted that recording media such as CD-ROMs on which the above-mentioned programs are stored, and HDDs on which these programs are stored, can be provided domestically or internationally as program products.

[0062] Furthermore, each of the above-mentioned programs may be recorded on a computer-readable recording medium as an installable or executable file, or may be downloaded and distributed via a network. Examples of recording media include CD-Rs (Compact Disc Recordables), DVDs, Blu-ray (registered trademark) discs, SD cards, and USB memory. Furthermore, the recording media may be provided domestically or internationally as a program product. For example, the communication management system 5 realizes the communication management method according to the present invention by executing the program according to the present invention.

[0063] <<Functional configuration of the embodiment>> Next, the functional configuration of this embodiment will be described with reference to Figures 10 to 16. Figures 14 and 15 are functional block diagrams of a portion of the image communication system.

[0064] <Functional configuration of the imaging device 1a> 14, the image capturing device 1a has a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a storage / readout unit 19a. Each of these units is a function or means realized by operating any of the components shown in FIG. 10 in response to an instruction from the CPU 111 in accordance with a program for the image capturing device that has been loaded from the SRAM 113 onto the DRAM 114.

[0065] The photographing device 1a also has a storage unit 1000a constructed from at least one of the ROM 112, SRAM 113, and DRAM 114 shown in Fig. 10. The storage unit 1000a stores the GUID (Globally Unique Identifier) ​​of the photographing device 1a. The storage unit 1000a also contains an image type management DB 1001a constructed from the image type management table shown in Fig. 16A.

[0066] The photographing device 1b has a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, a memory readout unit 19b, and a memory unit 1000b, but since these respectively realize the same functions as the reception unit 12a, the imaging unit 13a, the sound collection unit 14a, the communication unit 18a, the memory readout unit 19a, and the memory unit 1000a in the photographing device 1a, their descriptions will be omitted.

[0067] ●Image type management table● FIG. 16A is a conceptual diagram showing an image type management table. In this image type management table, image data IDs, IP addresses (which are an example of destinations of source terminals), and source names are stored and managed in association with each other. Of these, the image data ID is an example of image data identification information for identifying image data when the photographing device 1a (1b) performs video communication. The same image data ID is attached to image data transmitted from the same source terminal. This allows the destination terminal (receiving communication terminal) to identify the source terminal of the received image data. The IP address of the source terminal indicates the IP address of the communication terminal transmitting the image data indicated by the associated image data ID. The source name is a name for identifying the photographing device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name generated by each communication terminal, such as the videoconference terminal 3a, in accordance with a predetermined naming convention.

[0068] For example, it is shown that four communication terminals with IP addresses "1.2.1.3," "1.2.2.3," "1.3.1.3," and "1.3.2.3" are respectively transmitting image data indicated by image data IDs "RS001," "RS002," "RS003," and "RS004." Furthermore, the image types indicated by the source names of each communication terminal are "Video_Omni," "Video_Omni," "Video," and "Video," which indicate the image types as "special image," "special image," "general image," and "general image," respectively. Note that the special image here is a spherical panoramic image.

[0069] It should be noted that data other than image data may also be managed in association with an image data ID. Data other than image data may be, for example, sound data or material data used when sharing a screen. It should be noted that data other than image data may also be managed in association with an image data ID. Data other than image data may be, for example, sound data or material data used when sharing a screen.

[0070] The image type management table (image type management DB1001) is a data table used when the image capturing device 1 can directly communicate with the communication management system 5 via the communication network 100. For example, as in the first embodiment described below, when the image capturing device 1a and the video conference terminal 3a are connected and the video conference terminal 3a communicates with the communication management system 5 via the communication network 100, this data table does not need to be used.

[0071] (Functional configuration of the imaging device 1a) Next, each functional configuration of the photographing device 1a will be described in more detail with reference to Fig. 10 and Fig. 14. The reception unit 12a of the photographing device 1a is mainly realized by the processing of the operation unit 115 and CPU 111 shown in Fig. 10, and receives operation input from the user.

[0072] The imaging section 13a is mainly realized by the imaging unit 101, image processing unit 104, imaging control unit 105, and processing by the CPU 111 shown in FIG. 10, and captures images of scenery and the like to obtain captured image data.

[0073] The sound collection unit 14a is mainly realized by the microphone 108 and sound processing unit 109 shown in FIG. 10, and processing by the CPU 111, and collects sounds around the image capture device 1a.

[0074] The communication unit 18a is mainly realized by the processing of the CPU 111 shown in FIG. 10, and can communicate with the communication unit 38a of the video conference terminal 3a using short-range wireless communication technology such as NFC, Bluetooth, or Wi-Fi.

[0075] The storage / reading unit 19a is mainly realized by the processing of the CPU 111 shown in FIG. 10, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.

[0076] <Functional configuration of the video conference terminal 3a> 14, the video conference terminal 3a includes a transmitting / receiving unit 31a, a receiving unit 32a, an image / sound processing unit 33a, a display control unit 34a, a determining unit 35a, a generating unit 36a, a calculating unit 37a, a communication unit 38a, and a memory / reading unit 39a. Each of these units is a function or means realized by operating any of the components shown in FIG. 11 in response to an instruction from the CPU 301 in accordance with the program for the video conference terminal 3a that has been loaded from the flash memory 304 onto the RAM 303.

[0077] The video conference terminal 3a also has a storage unit 3000a constructed with at least one of the ROM 302, RAM 303, and flash memory 304 shown in Fig. 11. The storage unit 3000a contains an image type management DB 3001a, a photographing device management DB 3002a, and a predetermined area management DB 3003a. Of these, the image type management DB 3001a is constructed with the image type management table shown in Fig. 16B. The photographing device management DB 3002a is constructed with the photographing device management table shown in Fig. 16C. The predetermined area management DB 3003a is constructed with the predetermined area management table shown in Fig. 16D.

[0078] The video conference terminal 3d has a transmitting / receiving unit 31d, a receiving unit 32d, an image / sound processing unit 33d, a display control unit 34d, a judgment unit 35d, a generation unit 36d, a calculation unit 37d, a communication unit 38d, a memory reading unit 39d, and a memory unit 3000d, but since these units respectively realize the same functions as the transmitting / receiving unit 31a, the receiving unit 32a, the image / sound processing unit 33a, the display control unit 34a, the judgment unit 35a, the generation unit 36a, the calculation unit 37a, the communication unit 38a, the memory reading unit 39a, and the memory unit 3000a in the video conference terminal 3a, their descriptions will be omitted. In addition, the memory unit 3000d in the video conference terminal 3d has an image type management DB 3001d, a photographing device management DB 3002d, and a specified area management DB 3003d constructed therein. However, since these have the same data structures as the image type management DB 3001a, the photographing device management DB 3002a, and the specified area management DB 3003a in the video conference terminal 3a, their explanations will be omitted.

[0079] ●Image type management table● FIG. 16B is a conceptual diagram showing an image type management table. In this image type management table, image data IDs, IP addresses (which are an example of destinations of source terminals), and source names are stored and managed in association with each other. Of these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is attached to image data transmitted from the same source terminal. This allows the destination terminal (receiving communication terminal) to identify the source terminal of the received image data. The IP address of the source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for identifying the imaging device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name generated by each communication terminal, such as the videoconference terminal 3a, in accordance with a predetermined naming convention.

[0080] For example, it is shown that four communication terminals with IP addresses "1.2.1.3," "1.2.2.3," "1.3.1.3," and "1.3.2.3" are respectively transmitting image data indicated by image data IDs "RS001," "RS002," "RS003," and "RS004." Furthermore, the image types indicated by the source names of each communication terminal are "Video_Omni," "Video_Omni," "Video," and "Video," which indicate the image types as "special image," "special image," "general image," and "general image," respectively. Note that the special image here is a spherical panoramic image.

[0081] It should be noted that data other than image data may also be managed in association with an image data ID. Data other than image data may be, for example, sound data or material data used when sharing a screen. It should be noted that data other than image data may also be managed in association with an image data ID. Data other than image data may be, for example, sound data or material data used when sharing a screen.

[0082] ●Radiography equipment management table● 16C is a conceptual diagram showing an image capturing device management table. This image capturing device management table stores and manages the vendor ID and product ID of the GUID of an image capturing device that can obtain two hemispherical images that are the basis of a celestial sphere panoramic image. For example, a vendor ID (VID) and product ID (PID) used in a USB device can be used as the GUID. This vendor ID and product ID are stored in a communication terminal such as a videoconferencing terminal when it is shipped from the factory, but may also be additionally stored after shipping.

[0083] ●Predetermined area management table● FIG. 16D is a conceptual diagram showing a predetermined area management table. In this predetermined area management table, the IP address of the communication terminal that is the sender of the captured image data, the IP address of the communication terminal that is the destination of the captured image data, and predetermined area information indicating the predetermined area image currently being displayed on the communication terminal that is the destination of the captured image data are stored and managed in association with each other. The communication terminal that is the destination of the captured image data is also the communication terminal that is the sender of the predetermined area information. As shown in FIGS. 6 and 7, the predetermined area information is a conversion parameter for converting a captured image into an image of a predetermined area T in the captured image (predetermined area image). The IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for identifying the communication terminal, and the like. Here, the IP address is expressed as a simplified version of an IPv4 address. The IP address may also be an IPv6 address.

[0084] For example, in the predetermined area management table of FIG. 16D, when the IP address of the video conference terminal 3a is "1.2.1.3," the first to third rows of the table indicate that the captured image data transmitted from the video conference terminal 3a is transmitted via the communication management system 5 to the video conference terminal 3d having the IP address "1.2.2.3," the PC 7 having the IP address "1.3.1.3," and the smartphone 9 having the IP address "1.3.2.3." Furthermore, the video conference terminal 3d is identified as the communication terminal that transmitted the predetermined area information (r=20, θ=20, φ=30). Similarly, the PC 7 is identified as the communication terminal that transmitted the predetermined area information (r=20, θ=30, φ=40). Furthermore, the smartphone 9 is identified as the communication terminal that transmitted the predetermined area information (r=30, θ=40, φ=50).

[0085] In addition, when the transmitter / receiver unit 31a receives new specified area information that includes the same set of IP addresses as the IP address of the communication terminal that sent the captured image data that is already being managed and the IP address of the communication terminal that is the destination of the captured image data, the memory reading unit 39a rewrites the specified area information that is already being managed with the newly received specified area information.

[0086] (Functional configuration of the video conference terminal 3a) Next, the functional configuration of the video conference terminal 3a (3d) will be described in more detail with reference to Figures 11, 14, and 15. Note that the functional configuration of the video conference terminal 3d is the same as the functional configuration of the video conference terminal 3a described below, and therefore the description thereof will be omitted.

[0087] The transmission / reception unit 31a of the video conference terminal 3a is mainly realized by the processing of the network I / F 311 and CPU 301 shown in Figure 11, and transmits and receives various data (or information) to and from the communication management system 5 via the communication network 100.

[0088] The reception unit 32a is mainly realized by the operation buttons 308 shown in Fig. 11 and processing by the CPU 301, and receives various selections or inputs from the user. In addition to the operation buttons 308, a touch panel or the like may be used as another input means.

[0089] 11, and performs image processing on image data obtained by capturing an image of a subject with camera 312. After the user's voice is converted into an audio signal by microphone 314, image / sound processing unit 33a performs audio processing on audio data related to this audio signal.

[0090] Furthermore, the image / sound processing unit 33a performs image processing on image data received from another communication terminal based on image type information such as a source name, so that the display control unit 34 can display an image on the display 4. Specifically, when the image type information indicates that the image is a special image, the image / sound processing unit 33a generates omnidirectional panoramic image data as shown in FIG. 4(b) based on the image data (for example, data of each hemispherical image as shown in FIGS. 3(a) and 3(b)), and further generates a predetermined area image as shown in FIG. 6(b). Furthermore, the image / sound processing unit 33a outputs an audio signal related to audio data received from another communication terminal via the communication management system 5 to the speaker 315, and causes the speaker 315 to output audio.

[0091] The display control unit 34a is mainly realized by the processing of the display I / F 317 and CPU 301 shown in FIG. 11, and performs control for displaying various images, characters, and the like on the display 4.

[0092] The determination unit 35a is mainly realized by the processing of the CPU 301 shown in FIG. 11, and determines the image type of the image data received from the photographing device 1a, for example.

[0093] The generation unit 36a is mainly realized by the processing of the CPU 301 shown in Fig. 11, and generates a source name, which is an example of image type information, in accordance with the naming rules described above based on the determination result of the determination unit 35a as being a general image or a special image (here, a spherical panoramic image). For example, if the determination unit 35a determines that the image is a general image, the generation unit 36a generates a source name "Video" indicating that it is a general image. On the other hand, if the determination unit 35a determines that the image is a special image, the generation unit 36a generates a source name "Video_Omni" indicating that it is a special image.

[0094] The calculation unit 37a is mainly realized by the processing of the CPU 301 shown in Fig. 11, and calculates the position (position information) and direction of the predetermined area T1 relative to the predetermined area T2 in the captured image based on the predetermined area information (i2) indicating the predetermined area T2 and the predetermined area information (i1) received from another communication terminal by the transmission / reception unit 31a. Note that the predetermined area information (i1) indicates the predetermined area T1 in the captured image. An image in which the entire captured image is displayed is also referred to as an "entire image."

[0095] 11, and is capable of communicating with the communication unit 18a of the photographing device 1a via short-range wireless communication technologies such as NFC, Bluetooth, Wi-Fi, etc. Although the communication unit 38a and the transceiver unit 31a have been described as having separate communication units, they may also share a common communication unit.

[0096] The storage / reading unit 39a is mainly realized by the processing of the CPU 301 shown in FIG. 11, and stores various data (or information) in the storage unit 3000a and reads various data (or information) from the storage unit 3000a.

[0097] <Functional configuration of communication management system> Next, each functional component of the communication management system 5 will be described in detail with reference to Figures 12 and 15. The communication management system 5 has a transmitting / receiving unit 51, a recording processing unit 53, a determining unit 55, a generating unit 56, an image conversion processing unit 57, and a storage / reading unit 59. Each of these units is a function or means realized when any of the components shown in Figure 12 operates in response to an instruction from the CPU 501 in accordance with the program for the communication management system 5 loaded from the HD 504 onto the RAM 503.

[0098] The communication management system 5 also has a storage unit 5000 constructed from at least one of the ROM 502, RAM 503, and HD 504 shown in FIG. 12. The storage unit 5000 contains a session management DB 5001, an image type management DB 5002, a predetermined area management DB 5003, a predetermined area management DB 5003P, and a recorded file storage DB 5004. Of these, the session management DB 5001 is constructed from the session management table shown in FIG. 16E. The image type management DB 5002 is constructed from the image type management table shown in FIG. 16F. The predetermined area management DB 5003 and the predetermined area management DB 5003P are constructed from the predetermined area management table shown in FIG. 16G and the predetermined area management table P shown in FIG. 16H. The recorded file storage DB 5004 is constructed from the recorded file storage table shown in FIG. 16I. In addition, the image communication system of the embodiment may be configured to have a recording server equipped with a recording processing unit 53, an image conversion processing unit 57, a specified area management DB 5003, a specified area management DB 5003P, a recording file storage DB 5004, etc., independent of the communication management system 5.

[0099] ●Session Management Table● FIG. 16E is a conceptual diagram showing a session management table. In this session management table, session IDs and IP addresses of participating communication terminals are stored and managed in association with each other. Of these, the session ID is an example of session identification information for identifying a communication session that realizes a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal, such as the video conference terminal 3a, and is used when selecting a communication session in each communication terminal. The IP addresses of participating communication terminals indicate the IP addresses of communication terminals that have participated in the virtual conference room identified by the associated session ID.

[0100] ●Image type management table● FIG. 16F is a conceptual diagram illustrating an image type management table. In addition to the various pieces of information managed in the image type management table shown in FIG. 16B, this image type management table associates and manages session IDs identical to the session IDs managed in the session management table. Here, three communication terminals with IP addresses "1.2.1.3," "1.2.2.3," and "1.3.1.3" are participating in the virtual conference room identified by the same session ID "se101." The communication management system 5 manages the same image data ID, source terminal IP address, and image type information managed by communication terminals such as the video conference terminal 3a. This is because, when a new communication terminal joins the virtual conference room, image type information and the like are transmitted to communication terminals already in a video call and the newly joining communication terminal. This eliminates the need to transmit image type information and the like between communication terminals already in a video call and the newly joining communication terminal.

[0101] ●Predetermined area management table (live recording)● FIG. 16G is a conceptual diagram illustrating a predetermined area management table in the case of live recording. In live recording, the communication management system 5 performs perspective projection transformation using predetermined area information of the omnidirectional image displayed at each location (also referred to as a viewing location) in conjunction with real-time distribution (live streaming distribution), thereby generating a planar image in real time. By individually recording the video frames, the communication management system 5 generates a recording file or the like that indicates the viewing status of each viewing location. The predetermined area management table in the case of live recording basically has the same data structure as the predetermined area management table shown in FIG. 16D. However, as described below, the transmitting / receiving unit 51 transmits the latest predetermined area information to each communication terminal every certain period (e.g., every 30 seconds). Therefore, all predetermined area information received by the transmitting / receiving unit 51 is stored without being deleted until the predetermined area information is transmitted every certain period. In FIG. 16G, newer predetermined area information is managed at a higher level.

[0102] ●Predetermined area management table P (post-conversion recording)● FIG. 16H is a conceptual diagram showing a predetermined area management table for post-conversion recording. In post-conversion recording, only the original video data and the history of predetermined area information for each viewing location are recorded during real-time distribution. After real-time distribution ends, if a user requests a recording file for post-conversion recording, a recording file reflecting the viewing status at each viewing location is generated in response to the recording file request. The predetermined area management table for post-conversion recording not only includes the latest predetermined area information, but also adds a timestamp field to the predetermined area management table for live recording. This timestamp is also called time-series information and can be expressed, for example, in UNIX time. It records the date and time (time) when the predetermined area information received from each communication terminal by the transmitter / receiver 51. In this case of post-conversion recording, even for the same image source and image destination pair, a chronological history is recorded using a timestamp sequence.

[0103] ●Recording file storage table (live recording / post-conversion recording)● 16I is a conceptual diagram showing a recording file storage table. In this recording file storage table, the IP address of the image destination, a live / post flag, and a recording file name are stored and managed in association with each IP address of the image source. Among these, the live / post flag is a processing flag (information) that distinguishes whether the recording file indicated by the recording file name described below was recorded by live recording or by post-conversion recording. Furthermore, the recording file name indicates a recording file of a celestial sphere video (image) transmitted by one or more image capture devices 1 that are image sources in the image communication system, and a recording file that records a scene viewed at a viewing location where a device, communication terminal, etc. indicated by the IP address of each image destination is located.

[0104] In the case of live recording, the communication management system 5 records an independent file representing the original video (video or image during real-time distribution; hereinafter, simply referred to as "original video") for each IP address of the image sender. The communication management system 5 also distributes the video (image) captured by the image capture device 1 in real time and generates a recording file to be viewed at the communication terminal (viewing location) indicated by each IP address of the image destination. Specifically, the communication management system 5 generates, for example, a recording file (1201.mp4) from the perspective of a user at a viewing location indicated by the IP address of the image destination (e.g., 1.2.2.1). Furthermore, the communication management system 5 generates, for example, a recording file (1202.mp4) from the perspective of another user at a viewing location indicated by an IP address (e.g., 1.2.2.2). In this way, the communication management system 5 generates a recording file to be viewed at each viewing location along with the distribution of image data, which is referred to as a live recording method. However, even in this case, it is necessary to keep all of the spherical video (images) transmitted from the IP address of the image transmission source (for example, 1.2.1.3). Therefore, the communication management system 5 generates an original recording file indicated by the recording file name: 10101.mp4 without specifying the IP address of the image transmission destination ("-" in the data table). In other words, the file with the recording file name: 10101.mp4 is a file in which the spherical video (images) distributed (transmitted) by the camera device 1 installed at the viewing location of the transmission source during live streaming is recorded as is. Note that the recording file names: 10101.mp4, 1201.mp4, and 1202.mp4 are recorded files from live streaming, and therefore can be viewed at the same timing as the live streaming at each viewing location.

[0105] As described above, in the case of live recording, the communication management system 5 generates a recording file for each combination of the IP address of each video source (distribution source) and the IP address of each destination (distribution destination). That is, the state of which destination (distribution destination) location was viewing the video from which source (distribution source) location and from what viewpoint (predetermined area image) is saved. In addition, the video data from the source (distribution source) (e.g., 10101.mp4) is saved as is without conversion.

[0106] On the other hand, in the case of post-conversion recording, the communication management system 5 performs post-conversion recording when a user requests a recording file for post-conversion recording a predetermined period of time after the live broadcast. In this case, video (video files) representing the viewing situation at each viewing location are generated later in response to a user request after the real-time broadcast ends. Therefore, recording file names representing the original video are managed for each IP address of the image sender, and the later-generated recording files are added. Note that parentheses in the recording file name field indicate that the post-conversion recording file generated during post-conversion recording and the live recording file generated during live broadcast have the same content. For example, if the IP address of the image sender is 1.2.1.3 and the IP address of the image destination is 1.2.2.1, the live recording file is named 1201.mp4 and the post-conversion recording file is named 1850.mp4. However, although these file names are different for file management purposes, the contents (actual contents) of the recording files are the same, only the time of generation differs. One reason for managing live recording file names and post-conversion recording file names differently in this way is that it is useful, for example, from a business perspective of video distribution services, when managing distribution fees. Specifically, the image communication system can easily manage which recording files users have used as post-conversion recording files by using the live / post flag in combination. Thus, when generating post-conversion recording files, the communication management system 5 may change the post-conversion recording file name and the live recording file name, or may keep them the same, if the IP address of the image sender and the IP address of the image destination are the same combination.

[0107] In this embodiment, the IP address of the image sender described above is treated as an example of sender identification information or photographing device identification information. Similarly, the IP address of the image destination is treated as an example of destination identification information or communication terminal identification information. Furthermore, the photographing device identification information and communication terminal identification information described above may be, in addition to the IP address, the device identification information of the photographing device 1 or the terminal identification information of the communication terminal (smartphone 9). Furthermore, the photographing device identification information and communication terminal identification information are not limited to the device, and may also be the user identification information of the user using the photographing device 1 or the user identification information of the user using the communication terminal (smartphone 9), etc.

[0108] (Functional configuration of communication management system) Next, each functional configuration of the communication management system 5 will be described in detail with reference to Fig. 12 and Fig. 15. The transmission / reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I / F 509 and CPU 501 shown in Fig. 12, and transmits and receives various data (or information) to and from the video conference terminals 3a, 3d, or the PC 7 via the communication network 100.

[0109] The recording processing unit 53 is mainly realized by the processing of the CPU 501 shown in FIG. 12. In both the case of live recording and post-conversion recording, the recording processing unit 53 performs recording processing of original video associated with video (image) data received from each communication terminal and saves the video in an individual recording file. The recording processing unit 53 then manages each item (bibliographic information) in the recording file storage DB 5004 (see FIG. 16I). In the embodiment, the video (image) data also includes audio contained in the video. In the case of the live recording method, the recording processing unit 53 not only records the original video described above, but also records a perspective projection transformation processed image (an output image of the image transformation processing unit 57) that recreates the viewing situation of each user (viewer) at each viewing location. In the present embodiment, the live recording method is referred to as the first recording method. On the other hand, in the case of the post-conversion recording method, when a user using a communication terminal requests a recording file for post-conversion recording, the recording processing unit 53 generates the requested recording file in cooperation with the image conversion processing unit 57 based on a combination of identification information (e.g., IP address) of the source location and identification information (e.g., IP address) of the destination location included in the request. Furthermore, as another function of post-conversion recording, the recording processing unit 53 can also provide a user interface (UI) to a viewer (hereinafter referred to as a viewer requester) who requests to view a specific video on a web page. The viewer may be a user who was watching the live-streamed video at a specific viewing location at the time of the live broadcast, or another user (e.g., a site supervisor or manager related to the video being streamed).

[0110] In this embodiment, the post-conversion recording method is referred to as the second recording method. Also, in this embodiment, the recording processing unit 53 functions as an example of a recording means.

[0111] The determination unit 55 is mainly realized by the processing of the CPU 501 shown in FIG.

[0112] The generating unit 56 is mainly realized by the processing of the CPU 501 shown in FIG. 12, and generates the image data ID.

[0113] The image conversion processing unit 57 is mainly realized by the processing of the CPU 501 shown in Fig. 12, and in both the case of live recording and post-conversion recording, performs perspective projection transformation processing on the omnidirectional video (image) and predetermined area information to generate a planar image. In addition, in the case of live recording, the image conversion processing unit 57 performs perspective projection transformation processing on the omnidirectional video (image) received from each communication terminal and passes the result to the recording processing unit 53 for recording. Specifically, the image conversion processing unit 57 retrieves the predetermined area information of each viewing location where a video (image) based on the omnidirectional video (image) is viewed from the predetermined area management DB 5003 (see Fig. 16G), performs perspective projection transformation processing, and passes the result to the recording processing unit 53 for recording. On the other hand, in the case of post-conversion recording, the image conversion processing unit 57 performs perspective projection conversion processing based on each original video managed in the recording file storage DB 5004 (see FIG. 16I) specified by a user request and the history for each timestamp of the predetermined area information extracted from the predetermined area management DB 5003P (see FIG. 16H), and passes the result to the recording processing unit 53 for recording. In this embodiment, the image conversion processing unit 57 functions as an example of image conversion processing means.

[0114] Furthermore, the above-described recording processing unit 53 and image conversion processing unit 57 may be combined to function as a “recording conversion processing unit.” In this case, the recording conversion processing unit functions as an example of a recording conversion means.

[0115] 12 and processing by the CPU 501, and stores various data (or information) in the storage unit 5000, and reads various data (or information) from the storage unit 5000. In this embodiment, the storage and reading unit 59 functions as an example of a storage and reading means, and the storage unit 5000 functions as an example of a storage means.

[0116] Note that each of the above-described functional configurations shows a configuration including a module group for realizing a recording function within the communication management system 5. However, a configuration in which a recording server or the like is provided separately from the communication management system 5 may also be used.

[0117] <Functional Configuration of PC> Next, the functional configuration of the PC 7 will be described in detail with reference to FIGS. 12 and 15. The PC 7 basically has the same functions as the video conference terminal 3a. That is, as shown in FIG. 15, the PC 7 includes a transmission / reception unit 71, a reception unit 72, an image / audio processing unit 73, a display control unit 74, a determination unit 75, a generation unit 76, a calculation unit 77, a communication unit 78, and a memory readout unit 79. Each of these units is a function or means realized by any of the components shown in FIG. 12 operating according to instructions from the CPU 701 according to the program for the PC 7 developed from the HD 504 onto the RAM 503.

[0118] Further, the PC 7 has a storage unit 7000 constructed by at least one of the ROM 702, RAM 703, and HD 704 shown in FIG. 12. In this storage unit 7000, an image type management DB 7001, a photographing device management DB 7002, and a predetermined area management DB 7003 are constructed. Note that since the image type management DB 7001, the photographing device management DB 7002, and the predetermined area management DB 7003 have the same data structure as the image type management DB 3001a, the photographing device management DB 3002a, and the predetermined area management DB 3003a, respectively, the descriptions thereof are omitted.

[0119] (Functional Configurations of PC) Next, the functional configuration of the PC 7 will be described in detail with reference to FIGS. 12 and 15. The transmission / reception unit 71 of the PC 7 is mainly realized by the network I / F 709 and the processing of the CPU 701 shown in FIG. 12, and realizes the same functions as the transmission / reception unit 31a.

[0120] The reception unit 72 is mainly realized by the keyboard 711, the mouse 712, and the processing of the CPU 701 shown in FIG. 12, and realizes the same functions as the reception unit 32a.

[0121] The image / sound processing unit 73 is mainly implemented by instructions from the CPU 701 shown in FIG. 12, and realizes the same functions as the image / sound processing unit 33a.

[0122] The display control unit 74 is mainly realized by the processing of the CPU 701 shown in FIG. 12, and realizes the same functions as the display control unit 34a.

[0123] The determination unit 75 is mainly realized by the processing of the CPU 701 shown in FIG. 12, and realizes the same function as the determination unit 35a.

[0124] The generation unit 76 is mainly realized by the processing of the CPU 701 shown in FIG. 12, and realizes the same functions as the generation unit 36a.

[0125] The calculation unit 77 is mainly realized by the processing of the CPU 701 shown in FIG. 12, and realizes the same functions as the calculation unit 37a.

[0126] The communication unit 78 is mainly realized by the processing of the CPU 701 shown in FIG. 12, and realizes the same functions as the communication unit 38a.

[0127] The storage / reading unit 79a is mainly realized by the processing of the CPU 701 shown in FIG. 12, and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000.

[0128] <Smartphone functional configuration> Next, the functional configuration of the smartphone 9 will be described with reference to Figures 13 and 14. The smartphone 9 basically has the same functions as the video conference terminal 3a. That is, as shown in Figure 14, the smartphone 9 has a transmission / reception unit 91, a reception unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a generation unit 96, a calculation unit 97, a communication unit 98, and a storage / readout unit 99. Each of these units is a function or means realized when any of the components shown in Figure 13 operates in response to an instruction from the CPU 901 in accordance with a program for the smartphone 9 loaded from the EEPROM 904 onto the RAM 903.

[0129] 13. An image type management DB 9001, an imaging device management DB 9002, and a predetermined area management DB 9003 are configured in the storage unit 9000. Note that the image type management DB 9001, the imaging device management DB 9002, and the predetermined area management DB 9003 have the same data configurations as the image type management DB 3001a, the imaging device management DB 3002a, and the predetermined area management DB 3003a, respectively, and therefore descriptions thereof will be omitted.

[0130] (Smartphone functional configuration) Next, the functional configuration of the smartphone 9 will be described in detail with reference to Fig. 13 and Fig. 14. The transmitter / receiver 91 of the smartphone 9 is mainly realized by the processing of the long-distance communication circuit 911 and the CPU 901 shown in Fig. 13, and realizes the same functions as the transmitter / receiver 31a.

[0131] The reception unit 92 is mainly realized by the processing of the touch panel 921 and the CPU 901 shown in FIG. 13, and realizes the same functions as the reception unit 32a.

[0132] The image / sound processing unit 93 is mainly implemented by instructions from the CPU 901 shown in FIG. 13, and realizes the same functions as the image / sound processing unit 33a.

[0133] The display control unit 94 is mainly realized by the processing of the CPU 901 shown in FIG. 13, and realizes the same functions as the display control unit 34a.

[0134] The determination unit 95 is mainly realized by the processing of the CPU 901 shown in FIG. 13, and realizes the same function as the determination unit 35a.

[0135] The generation unit 96 is mainly realized by the processing of the CPU 901 shown in FIG. 13, and realizes the same functions as the generation unit 36a.

[0136] The calculation unit 97 is mainly realized by the processing of the CPU 901 shown in FIG. 13, and realizes the same functions as the calculation unit 37a.

[0137] The communication unit 98 is mainly realized by the processing of the CPU 901 shown in FIG. 13, and realizes the same functions as the communication unit 38a.

[0138] The storage / reading unit 99 is mainly realized by the processing of the CPU 901 shown in FIG. 13, and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000.

[0139] <<Processing or Operation of the Embodiment>> Next, the processing or operation of this embodiment will be described with reference to FIGS.

[0140] <Processing participation in a communication session> First, the process of joining a specific communication session will be described with reference to Fig. 17 and Fig. 18. Fig. 17 is a sequence diagram showing the process of joining a specific communication session. Fig. 18 is a diagram showing a selection screen for a communication session (virtual conference room).

[0141] First, when a user (e.g., user A1) at viewing site A operates the videoconference terminal 3a to display a selection screen for a communication session (virtual conference room), the reception unit 32a accepts the operation to display the selection screen, and the display control unit 34a displays the selection screen as shown in Fig. 18 on the display 4a (step S21). This selection screen displays selection buttons b1, b2, b3, etc., which indicate the virtual conference rooms R1, R2, R3, etc., to be selected. Each selection button b1, etc., is associated with a session ID.

[0142] When user A1 selects a desired selection button (here, selection button b1) in the virtual conference room, the reception unit 32a receives the selection of the communication session (step S22). Then, the transmission / reception unit 31a transmits a request to join the virtual conference room to the communication management system 5 (step S23). This participation request includes a session ID indicating the communication session whose selection was received in step S22 and the IP address of the video conference terminal 3a, which is the requesting terminal. As a result, the transmission / reception unit 51 of the communication management system 5 receives the participation request.

[0143] Next, the storage / reading unit 99 performs a process of joining the communication session by adding the IP address received in step S23 to the joining terminal IP address field of the record in the session management DB 5001 (see FIG. 16E) that has the same session ID as the session ID received in step S23 (step S24). Then, the transmitting / receiving unit 51 transmits a join request response to the video conference terminal 3a (step S25). This join request response includes the session ID received in step S23 and the result of the join processing. As a result, the transmitting / receiving unit 31a of the video conference terminal 3a receives the join request response. The following describes the case where the join processing is successful.

[0144] <Image type information management process> Next, the management process of image type information will be described with reference to Fig. 19. Fig. 19 is a sequence diagram showing the management process of image type information.

[0145] First, when a user at viewing site A (e.g., user A1) connects the USB cable of the cradle 2a with the image capturing device 1a attached to the video conference terminal 3a, the memory readout unit 19a of the image capturing device 1a reads the GUID of the image capturing device 1a stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of the image capturing device 1a to the communication unit 38a of the video conference terminal 3a (step S51). As a result, the communication unit 38a of the video conference terminal 3a receives the GUID of the image capturing device 1a.

[0146] Next, the determination unit 35a of the video conference terminal 3a determines the image type by determining whether the same vendor ID and product ID as those in the GUID received in step S51 are managed in the camera device management DB 3002a (see FIG. 16C) (step S52). Specifically, if the same vendor ID and product ID are managed in the camera device management DB 3002a, the determination unit 35a determines that the camera device 1a is a camera device that captures special images (here, omnidirectional panoramic images). On the other hand, if the same vendor ID and product ID are not managed in the camera device management DB 3002a, the determination unit 35a determines that the camera device 1a is a camera device that captures general images.

[0147] Next, the storage / readout unit 39a associates the IP address of the terminal (video conference terminal 3a) that is the source terminal with the image type information determined in step S52 and stores them in the image type management DB 3001a (see FIG. 16B) (step S53). In this state, no image data ID is associated. The image type information is, for example, a source name determined according to a predetermined naming rule or an image type (general image, special image).

[0148] Next, the transmitting / receiving unit 31a transmits a request to add image type information to the communication management system 5 (step S54). This request to add image type information includes the IP address of the terminal itself, which is the transmission source terminal stored in step S53, and the image type information. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request to add image type information.

[0149] Next, the storage / reading unit 59 of the communication management system 5 searches the session management DB 5001 (see FIG. 16E) using the IP address of the source terminal received in step S54 as a search key, and reads out the corresponding session ID (step S55).

[0150] Next, the generation unit 56 generates a unique image data ID (step S56). Then, the storage / reading unit 59 associates the session ID read in step S55, the image data ID generated in step S56, and the IP address of the transmission source terminal and image type information received in step S54 and stores them as a new record in the image type management DB 5002 (see FIG. 16F) (step S57). Then, the transmission / reception unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3a. As a result, the transmission / reception unit 31a of the video conference terminal 3a receives the image data ID (step S58).

[0151] Next, the storage / readout unit 39a of the video conference terminal 3a stores the image data ID received in step S58 in the image type management DB 3001a (see FIG. 16B) in association with the IP address and image type information of the terminal itself (video conference terminal 3a), which is the sending terminal, stored in step S53 described above (step S59).

[0152] Meanwhile, the transmitter / receiver 51 of the communication management system 5 transmits an addition notification of image type information to another communication terminal (here, the video conference terminal 3d) (step S60). This addition notification of image type information includes the image data ID generated in step S56, as well as the IP address and image type information of the terminal (video conference terminal 3a) that is the sender terminal stored in step S53. As a result, the transmitter / receiver 31d of the video conference terminal 3d receives the addition notification of image type information. Note that the destination of the transmitter / receiver 51 is another IP address that is associated in the session management DB 5001 (see FIG. 16E) with the same session ID as the IP address of the video conference terminal 3a. In other words, the destination is another communication terminal that is in the same virtual conference room as the video conference terminal 3a.

[0153] Next, the storage / readout unit 39d of the video conference terminal 3d associates the image data ID, the IP address of the transmission source terminal, and the image type information received in step S60 and stores them as a new record in the image type management DB 3001d (see FIG. 16B) (step S61). Similarly, an addition notification of the image type information is sent to the other communication terminals, the video conference terminal 3d and the PC 7, and the image type information is also stored in the image type management DBs 3001d and 7001 of the video conference terminals 3d and PC 7, respectively. As described above, the same information can be shared among the image type management DBs 3001a, 3001d, 7001, and 9001 of the communication terminals.

[0154] <Communication processing of captured image data> Next, the communication process of captured image data during a video call will be described with reference to Fig. 20 to Fig. 29. Fig. 20 is an image diagram showing the state of a video call. Of these, Fig. 20(a) shows a case where the image capturing device 1a is not used, and Fig. 20(b) shows a case where the image capturing device 1a is used.

[0155] First, as shown in FIG. 20(a), if the camera 312 (see FIG. 11) pre-installed in the video conference terminal 3a is used without using the image capture device 1a, the angle of view is 125 degrees horizontally and 70 degrees vertically, so the video conference terminal 3a must be placed at the edge of the desk where each user A1, etc. can be seen. This means that each user A1, etc. must face the video conference terminal 3a when speaking. Furthermore, because each user A1, etc. faces the video conference terminal 3a, the display 4a must also be placed next to the video conference terminal 3a. As a result, users A2 and A4, who are far from the video conference terminal 3a, are far from the microphone 314 (see FIG. 11), so they must speak relatively loudly and have difficulty seeing the content displayed on the display 4a.

[0156] 20(b), when the image capturing device 1a is used, two hemispherical images that are the basis for the omnidirectional panoramic image can be obtained, and therefore the video conference terminal 3a and the display 4a can be placed relatively close to the center of the desk. This allows each user A1, etc. to speak in a relatively low voice because they are close to the microphone 314, and the contents displayed on the display 4a can be easily seen. A whiteboard 6 is installed on the right side of the viewing site A, on which the user A1, etc. can write text, draw pictures, etc.

[0157] Next, with reference to Fig. 21, a process will be described in which the captured image data and audio data obtained at the viewing location A shown in Fig. 20(b) are transmitted to other communication terminals (smartphone 9, PC 7, video conference terminal 3d) via the communication management system 5. Fig. 21 is a sequence diagram showing the communication process of captured image data and audio data during a video call.

[0158] First, the communication unit 18a of the image capturing device 1a transmits captured image data obtained by capturing an image of a subject, scenery, or the like, and audio data obtained by collecting sound to the communication unit 38a of the video conference terminal 3a (step S101). In this case, since the image capturing device 1a is a device capable of obtaining two hemispherical images that are the basis for a celestial sphere panoramic image, the captured image data is composed of data for the two hemispherical images, as shown in Figures 3(a) and 3(b). As a result, the communication unit 38a of the video conference terminal 3a receives the captured image data and audio data.

[0159] Next, the transmitting / receiving unit 31a of the video conference terminal 3a transmits the captured image data and sound data sent from the image capturing device 1a to the communication management system 5 (step S102). This transmission includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the captured image data and image data ID.

[0160] Next, the transmitter / receiver 51 of the communication management system 5 transmits the captured image data and audio data to the communication terminals (smartphone 9, PC 7, and videoconference terminal 3d) participating in the same video call as the videoconference terminal 3a (steps S103, S104, and S105). Each of these transmission processes includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmitter / receiver 91 of the smartphone 9, the transmitter / receiver 71 of the PC 7, and the transmitter / receiver 31d of the videoconference terminal 3d receive the captured image data, image data ID, and audio data, respectively.

[0161] Next, a display example of the display 4d at the viewing location D will be described with reference to FIG. 22. FIG. 22 shows a display example of the display at the viewing location D. Of these, FIG. 22(a) shows a case where the captured image data transmitted from the camera 1a at the viewing location A via the video conference terminal 3a and the captured image data transmitted from the camera 1b at the viewing location B are displayed as is without generating a celestial sphere panoramic image or a predetermined area image. On the other hand, FIG. 22(b) shows a case where the captured image data transmitted from the camera 1a and 1b are used to generate a celestial sphere panoramic image and a predetermined area image. Note that an image of the viewing location A is displayed in the left display area (layout number "1") of the display 4d, and an image of the viewing location B is displayed in the upper right display area (layout number "2"). Furthermore, an image of the viewing location C is displayed in the middle right display area (layout number "3") of the display 4d, and an image of the viewing location D (the user's location) is displayed in the lower right display area (layout number "4"). The display area with layout number "1" is the main display area, and the display areas with layout numbers "2," "3," and "4" are sub-display areas. The images in the main display area and the sub-display area can be changed on each communication terminal. Typically, at each viewing location, an image of the viewing location where the central person in the video call is located is displayed in the main display area.

[0162] Here, when the captured image data transmitted from the imaging devices 1a and 1b capable of capturing omnidirectional panoramic images is displayed as is, as shown in FIG. 22(a), the images from viewing location A and viewing location B are displayed as front hemispherical images and rear hemispherical images, as shown in FIGS. 3(a) and (b), respectively.

[0163] In contrast, when the image / sound processing unit 93 generates a celestial sphere panoramic image from the captured image data output by the camera devices 1a and 1b that can obtain two hemispherical images that are the basis of the celestial sphere panoramic image and further generates a predetermined area image, the predetermined area image, which is a planar image, is displayed as shown in Fig. 22(b). Note that, at the viewing location C, the camera device 8 that obtains a general image takes the image, and at the viewing location D, the video conference terminal 3d that obtains a general image takes the image, so that a general image (here, a planar image) is displayed in both Figs. 22(a) and (b).

[0164] Furthermore, users at each viewing location can change the predetermined area associated with the predetermined area image in the same omnidirectional panoramic image. For example, user B1 operates the touch panel 921, etc., so that the reception unit 92 receives a request to move the predetermined area image, and the display control unit 94 can shift, rotate, reduce, or enlarge the predetermined area image. As a result, the predetermined area image shown in FIG. 22(b), which displays users A1 and A2 at the viewing location A in the initial setting (default), can be changed to the predetermined area image shown in FIG. 22(c). Specifically, FIG. 22(c) illustrates a state in which the predetermined area image including users A1 and A2 in the image captured at the viewing location A shown in FIG. 20(b) has been changed to the predetermined area image including the whiteboard 6.

[0165] 22(b) and 22(c) are examples of special image identification icons for indicating that the celestial sphere icons 191 and 192 are predetermined area images that indicate a predetermined area T that is a part of a celestial sphere panoramic image. The display positions of the celestial sphere icons 191 and 192 do not have to be the upper right, but may be anywhere, such as the upper left, lower left, or lower right. The types of the celestial sphere icons 191 and 192 are not limited to those shown in FIGS. 22(b) and 22(c). Instead of the celestial sphere icons 191 and 192, characters such as "celestial sphere image" may be used, or a combination of an icon and characters may be used.

[0166] Next, the processing in the image communication system when the predetermined area image is displayed as in Fig. 22(b) and when the predetermined area image is changed from Fig. 22(b) to Fig. 22(c) will be described using Fig. 23. Fig. 23 is a sequence diagram showing the processing for sharing predetermined area information. In Fig. 23, the video conference terminal 3a at viewing location A is the third communication terminal, the video conference terminal 3d at viewing location D is the other communication terminal, and the smartphone 9 at viewing location B is the communication terminal (own terminal).

[0167] 22(b), when a user D1 or the like at viewing location D uses a videoconference terminal 3d to display a predetermined area image of viewing location A, the transmitter / receiver 31d of the videoconference terminal 3d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S111). This predetermined area information includes the IP address of the videoconference terminal 3a that is the sender of the captured image data, and the IP address of the videoconference terminal 3d that is the destination of the captured image data (the sender of the predetermined area information). As a result, the transmitter / receiver 51 of the communication management system 5 receives the predetermined area information.

[0168] Next, the storage / readout unit 59 of the communication management system 5 associates the predetermined area information received in step S111 with the IP addresses of the sender and the destination and stores them in the predetermined area management DB 5003 (step S112). Note that the processes of steps S111 and S112 are performed every time the predetermined area image is changed on the video conference terminal 3d, as shown in Figures 22(b) to 22(c).

[0169] Next, the storage / reading unit 59 of the communication management system 5 reads out the most recent pair of predetermined area information and IP addresses (the pair stored most recently) at that time from among the pairs of predetermined area information and IP addresses stored in the predetermined area management DB 5003 at regular intervals (e.g., 30 seconds) (step S113). The transmission / reception unit 51 then distributes (transmits) the predetermined area information including the IP addresses read out in step S113 to other communication terminals (video conference terminal 3a, smartphone 9, PC 7) that are performing the same video call with the video conference terminal 3d that sent the predetermined area information (steps S114, S116, S118). As a result, the transmission / reception unit 31a of the video conference terminal 3a receives the predetermined area information. The storage / reading unit 39a then stores the predetermined area information and the IP addresses received in step S114 in the predetermined area management DB 3003a while associating them with each other (step S115). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives the predetermined area information, the storage / reading unit 99 stores the predetermined area information received in step S116 in association with each IP address in the predetermined area management DB 9003 (step S117). Furthermore, in the PC 7, after the transmitting / receiving unit 71 receives the predetermined area information, the storage / reading unit 79 stores the predetermined area information received in step S118 in association with each IP address also received in the predetermined area management DB 7003 (step S119).

[0170] As described above, the predetermined area information indicating the predetermined area image changed at viewing location D is transmitted to each communication terminal at other locations B, C, and A that are participating in the same video call, so that the predetermined area information indicating the predetermined area image being displayed at viewing location D is also shared at other locations B, C, and A. This process is also performed in the same way when the predetermined area image is changed at locations B, C, and A, and the predetermined area information indicating the predetermined area image being displayed at other locations is shared among all locations that are participating in the same video call.

[0171] Next, a method for using the predetermined area information shared among the locations will be described with reference to Figures 24 to 29. Figure 24 is a flowchart showing the display process of a predetermined area image. Note that the process is the same for each communication terminal, so here, the process of the smartphone 9 at location B will be described. Specifically, when captured image data transmitted from the video conference terminal 3a at location A is displayed as a predetermined area image by the video conference terminal 3d at the viewing location D, the video conference terminal 3d transmits predetermined area information indicating the predetermined area image to other communication terminals participating in the same video call, and the process executed by the smartphone 9 at the viewing location B will be described.

[0172] First, in the smartphone 9, the memory reading unit 99 reads out the corresponding source name (image type information) by searching the image type management DB 9001 (see FIG. 16B) using the image data ID received in step S103 shown in FIG. 21 as a search key (step S131).

[0173] Next, the determination unit 95 determines whether the image type information read out in step S131 indicates a "special image" (step S132). If the image type information indicates a "special image" (step S132; YES), the storage and reading unit 99 further searches the predetermined area management DB 9003 for predetermined area information indicating a predetermined area image displayed by a communication terminal at another base (step S133). Then, the determination unit 95 determines whether the predetermined area management DB 9003 manages predetermined area information indicating a predetermined area image displayed by a communication terminal at another base (step S134). If the predetermined area information indicating the predetermined area image displayed by the communication terminal at the other location is managed (step S134; YES), the calculation unit 97 calculates the position of the point of gaze of the predetermined area T1 relative to the predetermined area T2 in the entire image based on the predetermined area information (i2) indicating the predetermined area image of the predetermined area T2 currently displayed by the smartphone 9 (own terminal) and the predetermined area information (i1) indicating the predetermined area image of the predetermined area T1 received from the other communication terminal by the transmission / reception unit 91 and managed in the predetermined area management DB 9003. Similarly, the calculation unit 97 calculates the direction of the predetermined area T1 relative to the predetermined area T2 in the entire image (step S135). Note that, strictly speaking, the position in this case indicates the point of gaze of the predetermined area T1 relative to the point of gaze of the predetermined area T2. The point of gaze is the center point as described above, but may also be the upper left corner (or the lower left corner, upper right corner, or lower right corner) of the rectangle of each predetermined area. Furthermore, the point of gaze may also be a specific point within each predetermined area.

[0174] Here, a method for calculating the gaze point of a predetermined area T1 relative to a predetermined area T2 in an entire image will be described using Fig. 25. Fig. 25(a) is a diagram showing the definition of the angle of the virtual camera, and Fig. 25(b) is a diagram showing a method for calculating the position of the gaze point of another base in the predetermined area image of the own base by parallel projection as viewed from above.

[0175] 25(a), the calculation unit 97 acquires the radius vector r, the polar angle θ, and the azimuth angle φ from the predetermined area information indicating the predetermined area image being displayed by the display control unit 94 of the own terminal (smartphone 9), and sets this as CP1(r0, θ1, φ1). Next, the calculation unit 97 acquires the radius vector r, the polar angle θ, and the azimuth angle φ from the predetermined area information of the other base read in step S133, and sets this as CP2(r0, θ2, φ2).

[0176] Considering a predetermined area T2 centered on the gaze point CP1 of the own terminal (smartphone 9), the width w and height h of the predetermined area T2 are projected as a width of w and a height of h cosθ1 in Figure 25(b) which is parallel projection from the polar direction.

[0177] In addition, the radius of gaze point CP1 is projected onto the length of r0 sinθ1, and the radius of gaze point CP2 is projected onto the length of r0 sinθ2, so gaze point CP1 is located at the coordinates (r0 sinθ1 · r0 cosφ1, r0 sinθ1 · r0 sinφ1), and gaze point CP2 is located at the coordinates (r0 sinθ2 · r0 cosφ2, r0 sinθ2 · r0 cosφ2).

[0178] As a result of the above, the coordinates of the gaze points CP1 and CP2 can be derived in Figure 25(b), and the position of the gaze point CP2 on the plane of a specified area T2 with width ww and height hh can be derived using general coordinate transformation.

[0179] Here, a method for calculating the direction of a predetermined area T1 relative to a predetermined area T2 in an entire image will be described with reference to Fig. 26. Fig. 26(a) is a diagram showing the definition of an angle, and Fig. 26(b) is a diagram showing the definition of an angle range.

[0180] 26(a), the calculation unit 97 acquires the azimuth angle φ from the predetermined area information indicating the predetermined area image being displayed by the display control unit 94 of the own terminal (smartphone 9), and sets this as the rotation angle φ1. Next, the calculation unit 97 acquires the azimuth angle φ from the predetermined area information of the other location read out in step S133, and sets this as the rotation angle φ2. Furthermore, the calculation unit 97 sets the difference between the rotation angle φ2 and the rotation angle φ1 as the rotation angle φ3.

[0181] 26(b), if the angle range centered on the rotation angle φ of the own base is defined as α1 and the angle range centered on the angle obtained by adding 180 degrees to the horizontal angle of the own base is defined as α2, the calculation unit 97 calculates the direction of the predetermined area T1 relative to the predetermined area T2 in the entire image as follows: (1) If the rotation angle φ3 is within the angle range α1, the positional relationship is determined to be the "forward direction." (2) If the angle range α2 is within the angle range α2, the positional relationship is determined to be the "backward direction." (3) If the value obtained by subtracting half the fixed angle α from the reference horizontal angle of φ3 is within an angle range that is greater than 0 degrees and less than 180 degrees, the positional relationship is determined to be the "rightward direction." (4) If the value obtained by subtracting the fixed angle α from the reference horizontal angle of φ3 is greater than 0 degrees and less than 180 degrees, the positional relationship is determined to be the "rightward direction." (5) If the value obtained by subtracting the fixed angle α from the reference horizontal angle of φ3 is greater than 180 degrees and less than 360 degrees, the positional relationship is determined to be the "leftward direction."

[0182] Returning to Fig. 24, the image / sound processing unit 93 generates a predetermined area image including a fixation point mark indicating the fixation point calculated by the calculation unit 97 and a display direction mark indicating the direction (step S136). The display position of the fixation point mark is directly determined from the position of the predetermined area T1 relative to the predetermined area T2 in the entire image. The display position of the display direction mark is determined by the above-mentioned determination processes (1) to (5) using the position of the predetermined area T1 relative to the predetermined area T2 in the entire image.

[0183] At this time, the image / sound processing unit 93 combines celestial sphere icons 191 and 192 indicating that the image is a celestial sphere panoramic image with the predetermined area image based on the fact that the image type information indicates a "special image." Then, the display control unit 94 displays the predetermined area image generated in step S136 (step S137) as shown in FIGS. 27(a), (b), and (c), 28(a), (b), and 29. Note that FIGS. 27(a), (b), and (c) show images of the main display area and are diagrams illustrating three display examples of the predetermined area image including a display direction mark. Note that FIGS. 28(a) and (b) show images of the main display area and are diagrams illustrating two display examples of the predetermined area image including a point of interest mark. Note that FIG. 29 shows an image of the main display area and is a diagram illustrating one display example of the predetermined area image including a point of interest mark and a display direction mark. Also, as shown in Figure 22, images of all viewing locations during a video call are displayed, but in Figures 27, 28 and 29, due to drawing area limitations, only the image of viewing location A is displayed.

[0184] As shown in Fig. 27(a), in a predetermined area image that is a part of the image at viewing location A, display direction marks m11, m13, and m14 are displayed in the entire image, indicating the direction of the predetermined area image displayed at another location relative to the predetermined area image being displayed at viewing location B (own location). Note that display direction marks m21, m23, and m24 shown in Fig. 27(b) and display direction marks m31, m33, and m34 shown in Fig. 27(c) correspond to display direction marks m11, m13, and m14 shown in Fig. 27(a), respectively.

[0185] The above-described display direction marks are merely examples of direction information, and other forms of information may be used. Furthermore, direction information may be indicated not by arrows but by characters such as "right," "left," "back," and "front."

[0186] 28(a), in the predetermined area image, which is a part of the image at viewing location A, point of interest marks m41 and m42 are displayed in the entire image, indicating the point of interest of the predetermined area image displayed at another location, with the predetermined area image being displayed at viewing location B (own location) as the reference. The point of interest marks may be displayed semi-transparently so as not to obscure the predetermined area image. Note that the point of interest marks m51 and m52 shown in FIG. 28(b) correspond to the point of interest marks m41 and m42 shown in FIG. 28(a), respectively.

[0187] In FIG. 28(a), labels for a point of gaze mark m41 showing the viewing location name "C" and a point of gaze mark m42 showing the viewing location name "D" are displayed so that it is possible to identify which viewing location the point of gaze belongs to. In contrast, in FIG. 28(b), the viewing location name is not displayed, and different viewing locations are indicated by point of gaze marks with different patterns (checkered patterns, diagonal lines, etc.). In this case, each viewing location uses a different fill pattern to distinguish the points of gaze of other locations. If a table that corresponds patterns to viewing location names is prepared, users at each viewing location can identify their viewing location from the pattern of the point of gaze mark. This table may be printed on paper or stored as electronic data at each viewing location.

[0188] Note that instead of displaying a pattern to distinguish the point of interest marks, the marks may be distinguished by changing the color or the type of line. Note that the point of interest marks are an example of corresponding position information.

[0189] In Figure 29, if the point of interest of another location is included in the range of the specified area image, a point of interest mark m41 indicating the point of interest is displayed, and if it is not included in the range of the specified area image, display direction marks m11 and m14 are displayed.

[0190] 24, if it is determined in step S134 that the predetermined area management DB 9003 does not manage the predetermined area information indicating the predetermined area image displayed by the communication terminal at the other location (step S134; NO), the image / sound processing unit 93 generates the predetermined area image without including the point of interest mark and the display direction mark (step S138). After that, the process proceeds to step S137.

[0191] Furthermore, if it is determined in step S132 that the image type information does not indicate a "special image" (step S132; NO), that is, if the image type information indicates a "general image," the image / sound processing unit 93 does not generate a spherical panoramic image from the captured image data received in step S103, and the display control unit 94 displays a general image (step S139).

[0192] As described above, users B1 and B2 at viewing location B can grasp the positional relationship between the predetermined area image displayed at their own location and the predetermined area image displayed at another location, thereby preventing users B1 and B2 at viewing location B from being unable to keep up with the topic of a meeting, etc.

[0193] <<Main Effects of This Embodiment>> As described above, according to this embodiment, a communication terminal such as the video conference terminal 3a can generate a celestial sphere panoramic image and further generate a predetermined area image using corresponding image type information based on an image data ID transmitted together with image data. This has the effect of preventing the front hemispherical image and the rear hemispherical image from being displayed, as shown in Fig. 22(a).

[0194] Furthermore, a user at any viewing location can grasp which part of the entire image of the omnidirectional panoramic image is being displayed at another location, which has the effect of making it easier to follow the topic of a meeting, etc., compared to conventional methods.

[0195] 23, if the communication management system 5 were to transfer the predetermined area information to other communication terminals every time it received it from the video conference terminal 3d, users B1 and B2 would not be able to concentrate on the video call due to the flickering of the gaze point mark and display direction mark shown in Fig. 29. Therefore, as shown in steps S112 to S114 described above, the communication management system 5 periodically distributes the latest predetermined area image at that time and a set of each IP address, allowing each user to concentrate on the video call.

[0196] Second Embodiment Next, a second embodiment will be described with reference to Fig. 30. Fig. 30 is a sequence diagram showing another example of the process shown in Fig. 23, illustrating another process for sharing predetermined area information. In Fig. 30, the video conference terminal 3a at the viewing location A is the communication terminal (own terminal), and the video conference terminal 3d at the viewing location D is the other communication terminal.

[0197] In the first embodiment described above, as shown in Fig. 23, the communication management system 5 temporarily manages each piece of predetermined area information transmitted from each communication terminal (see step S112) and periodically transmits each piece of predetermined area information to each communication terminal other than the transmission source (see steps S114 to S119). In contrast, in the present embodiment, as shown in Fig. 30, the communication terminal (here, the video conference terminal 3a) that transmitted the captured image data temporarily manages the predetermined area information on behalf of the communication management system 5 (see step S213) and periodically transmits each piece of predetermined area information to each communication terminal other than the own terminal (see steps S215 to S221). That is, in the present embodiment, the communication terminal that transmitted the captured image data manages how the captured image data transmitted by the own terminal (here, the video conference terminal 3a) is displayed on the other communication terminals as a predetermined area image of the predetermined area T1.

[0198] This embodiment has the same configuration as the first embodiment, but differs from it in the processing shown in Fig. 23. Therefore, in the following, the same components are designated by the same reference numerals, and the description thereof is omitted, and processing that differs from the first embodiment will be described using Fig. 30.

[0199] First, when a user D1 or the like at viewing location D uses a videoconference terminal 3d to display a predetermined area image of viewing location A, the transmitter / receiver 31d of the videoconference terminal 3d transmits predetermined area information indicating the displayed predetermined area image to the communication management system 5 (step S211). This predetermined area information includes the IP address of the videoconference terminal 3a that is the sender of the captured image data and the IP address of the videoconference terminal 3d that is the destination of the captured image data (the sender of the predetermined area information). As a result, the transmitter / receiver 51 of the communication management system 5 receives the predetermined area information.

[0200] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits the predetermined area information including each IP address received in step S211 to the video conference terminal 3a that is the sender of the captured image data (step S212). As a result, the transmitting / receiving unit 31a of the video conference terminal 3a receives the predetermined area information.

[0201] Next, the storage / readout unit 39a of the video conference terminal 3a associates the predetermined area information received in step S212 with the IP addresses of the sender and the recipient and stores them in the predetermined area management DB 3003a (step S213). The process of step S213 is a process for managing how the captured image data sent by the terminal itself (here, the video conference terminal 3a) is displayed on other communication terminals. The processes of steps S211 to S213 are performed every time the predetermined area image is changed on the video conference terminal 3d.

[0202] Next, the storage / reading unit 39a of the video conference terminal 3a reads out the latest pair of predetermined area information and each IP address at that time (the pair stored most recently) from among the pairs of predetermined area information and each IP address stored in the predetermined area management DB 3003a at regular intervals (e.g., 30 seconds) (step S214). Then, the transmitting / receiving unit 31a transmits the predetermined area information including each IP address read in step S214 to the communication management system 5 (step S215). As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the predetermined area information including each IP address.

[0203] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits (distributes) the predetermined area information including each IP address received in step S215 to each communication terminal (videoconference terminal 3d, smartphone 9, PC 7) (steps S216, S218, S220). As a result, the transmitting / receiving unit 31d of the videoconference terminal 3d receives the predetermined area information. Then, the storage / reading unit 39d stores the predetermined area information received in step S216 in the predetermined area management DB 3003d in association with each IP address also received (step S217). Similarly, in the smartphone 9, after the transmitting / receiving unit 91 receives the predetermined area information, the storage / reading unit 99 stores the predetermined area information received in step S218 in the predetermined area management DB 9003 in association with each IP address also received (step S219). Furthermore, in PC7, after the transmitter / receiver 71 receives the specified area information, the memory / readout unit 79 stores the specified area information received in step S220 in the specified area management DB 7003 in association with each IP address also received (step S221).

[0204] <<Main Effects of This Embodiment>> As described above, according to this embodiment, the communication terminal that has transmitted the captured image data collects, based on the captured image data transmitted from its own terminal, predetermined area information indicating how the captured image data is displayed on each communication terminal, and distributes the information to each communication terminal. This has the effect of preventing the burden on the communication management system 5 from being concentrated when the same conference or the like is held at a large number of communication terminals, in addition to the effect of the first embodiment.

[0205] Third Embodiment Next, a third embodiment will be described.

[0206] In FIG. 10, the image capture device 1a is equipped with one microphone 108, but in this embodiment, the image capture device 1a is equipped with multiple directional microphones. By using multiple directional microphones in the image capture device 1a, audio data from each directional microphone is transmitted from the image capture device 1a to the video conference terminal 3a. As a result, the calculation unit 37a of the video conference terminal 3a calculates the direction of the sound source (microphone angle) from the audio data of each microphone, and the calculated direction is used to identify the position of the speaker (who transmitted the captured image). The same is true for the image capture device 1b. The image capture device 1b, which is equipped with multiple directional microphones, transmits audio data from each microphone to the smartphone 9, and the calculation unit 97 of the smartphone 9 calculates the direction of the sound source (microphone angle) from the audio data of each microphone, and the calculated angle is used to identify the position of the speaker (who transmitted the captured image).

[0207] <<Main Effects of This Embodiment>> As described above, according to the present embodiment, it is possible to more accurately display which part of the predetermined region image of the entire image of the omnidirectional panoramic image is being displayed at another location.

[0208] [Fourth embodiment] Next, a fourth embodiment will be described with reference to FIGS. 31 to 35. FIG. 31 is a sequence diagram showing the communication process of captured image data and audio data during a video call. This embodiment differs from the first embodiment in that the process of FIG. 21 shown in the first embodiment, performed by the video conference terminal 3a, is unnecessary. However, it is assumed that the predetermined area management DB 5003 (see FIG. 16G) used in the first to third embodiments is also assigned predetermined values ​​by the processes shown in FIG. 23 and the like in this embodiment. As such, in this embodiment, descriptions of the same parts as in the other embodiments will be omitted, and newly illustrated processes shown in FIG. 31 and subsequent figures will be described. Furthermore, in the fourth embodiment, assuming the use of the hardware assets and functional configurations constituting the image communication system described above, an example will be described in which an outdoor event at a construction site or the like is live-streamed and the live-streamed video (images) is viewed using one or more communication terminals located in remote locations.

[0209] 9, it is assumed that the camera 1a is a camera that transmits captured image data representing captured images to the communication management system 5 and is capable of real-time distribution (live streaming distribution) of the captured images to other communication terminals. That is, in the first embodiment, the camera 1a transmits (distributes) the captured image data representing the omnidirectional panoramic image captured by the camera 1a via the video conference terminal 3a (communication terminal). Instead, in the fourth embodiment, the camera 1a that captured the omnidirectional image is capable of transmitting (live streaming distribution) the captured image data representing the captured omnidirectional panoramic image to other communication terminals located at other locations (viewing locations B, C, D, ...) including the viewing location A.

[0210] In FIG. 31 , first, the communication unit 18a of the photographing device 1a transmits photographed image (video) data obtained by photographing a subject, scenery, or the like, and audio data obtained by collecting audio, to the transmission / reception unit 51 of the communication management system 5 (step S301). The transmission / reception unit 51 of the communication management system 5 then receives the photographed image data and audio data transmitted by the photographing device 1a. At this time, the photographed image (video) data includes an image data ID. Since the photographing device 1a can obtain two hemispherical images that are the basis of a celestial sphere panoramic image, the photographed image (video) data is composed of data for the two hemispherical images, as shown in FIGS. 3(a) and 3(b). For ease of explanation, in this embodiment, it is assumed that the photographed image (video) data includes the audio data described above, and the term "photographed image (video) data" is used to refer to the audio data. Furthermore, the photographed image (video) data may also be simply referred to as photographed image data.

[0211] Next, the communication management system 5 executes a recording process for the received captured image data (step S302). Details of step S302 will be explained in detail in the recording process flow below.

[0212] <Recording process flow> <<Live recording process flow>> Next, the live recording process executed in the communication management system 5 will be described. Fig. 32 is a flowchart showing the recording process of live recording or post-conversion recording selection and live recording. Note that each process shown in Fig. 32 is executed every time media data is received from each communication terminal. First, the determination unit 55 determines whether the received data is video data (step S302-1).

[0213] If it is determined that the received data is not video data (step S302-1; NO), the determination unit 55 exits this flow. On the other hand, if it is determined that the received data is video data (step S302-1; YES), the recording processing unit 53, in cooperation with the storage and reading unit 59, saves the video frames of the received video data as an original recording file in the recording file storage DB 5004 (see FIG. 16I) (step S302-2). In this processing step, the recording file is recorded in a video format that includes the entire celestial sphere (for example, the equirectangular format) even in the case of a spherical panoramic video.

[0214] Next, the determination unit 55 determines whether the received video frame was recorded by live recording or by post-conversion recording (step S302-3). If the received video frame was recorded by post-conversion recording (step S302-3; post), the determination unit 55 exits this flow to transition to the subsequent process indicated by circle A in Fig. 33.

[0215] On the other hand, if the received video frame was recorded by live recording (step S302-3; LIVE), the determination unit 55 further searches the image type management DB 5002 (see FIG. 16F) using the image data ID included in the received data as a search key to read out the corresponding image type, and determines whether the video related to the received data is a spherical video (step S302-4). If the video related to the received data is a flat video and not a spherical video (step S302-4; NO), the determination unit 55 exits this flow without performing any particular process.

[0216] On the other hand, if the video related to the received data is a spherical video (step S302-4; YES), the storage and reading unit 59 searches the predetermined area management DB 5003 (see FIG. 16G) using each IP address of the image transmission destination representing each viewing location as a search key, thereby reading out the latest predetermined area information corresponding to each IP address related to the received data (step S302-5).

[0217] Next, the image transformation processing unit 57 performs perspective projection transformation for each piece of the latest predetermined area information that has been read out, and generates a predetermined area image (planar image) (step S302-6).

[0218] After the specified area images (planar images) are generated, the storage and reading unit 59 stores each generated specified area image (planar image) as a video frame in the recording file storage DB 5004 (see Figure 16I) and exits this flow (step S302-7).

[0219] The processes in steps S302-5 to S302-07 described above are performed for each piece of data from each viewing location viewing the received video data, and a video file reflecting the viewing status for each viewing location is generated and managed with a recording file name assigned. As a result, in the live recording method, the communication management system 5 can generate a recording file reflecting the viewing status for each viewing client (viewing location) along with real-time distribution of captured images. In this case, there is no need to perform screen capture on each viewing client (each communication terminal), and the processing load on the viewing client side and the generation of data from the viewing client to the communication management system 5 can be reduced. In addition, the captured original video is also saved in the recording file storage DB 5004 (see FIG. 16I) with a predetermined recording file name. Therefore, for the spherical video, records remain even for areas that no viewing location is paying attention to, and users can refer to the remaining records after a predetermined time has passed, as needed.

[0220] <<Recording process flow for post-conversion recording>> Next, the post-conversion recording process executed in the communication management system 5 will be described. FIG. 33 is a flowchart showing the recording process in the case of post-conversion recording. The process in FIG. 33 is a flow chart executed when the determination unit 55 determines in step S302-3 of FIG. 32 that post-conversion recording will be executed (when transitioning to the process indicated by circle A). In post-conversion recording, when a captured image is distributed in real time, the original of each image (video) data and the history of predetermined area information for each viewing location are recorded. Then, in response to a request from a user to obtain a recording file after the real-time distribution ends, a recording file reflecting the viewing status of the specified viewing location is generated. Note that the transition to the process indicated by circle A may occur a predetermined period of time after the end of step S302-3 of FIG. 32. This predetermined period may be, for example, 10 minutes, 3 hours, or 10 days.

[0221] First, the determination unit 55 determines whether or not post-conversion recording has been requested (presence or absence of a post-conversion recording request) (step S401). This request for post-conversion recording refers to a request to acquire post-conversion predetermined area image data that will be converted by the image conversion processing unit 57, which will be described later. If post-conversion recording has not been requested (step S401; NO), the determination unit 55 remains in the process of step S401.

[0222] On the other hand, if post-conversion recording is requested (step S401; YES), the memory reading unit 59 reads the corresponding original recording file by searching the recording file storage DB 5004 (see Figure 16I) using the identification information (IP address, an example of source identification information or recording device identification information) of the camera device 1, which is the specified image sender (viewing location), as a search key (step S402).

[0223] A request for a recorded file is realized, for example, by a view requester accessing a site work history page for a construction site or other site where real-time streaming was performed using a communication terminal or the like. Specifically, the view requester accesses a web page or the like by specifying a date and time, such as when they want to obtain a record of site work from a week ago. That is, when the view requester wants to know details of the site work history at a certain location at a specified date and time, they can check details of events that occurred at the specified date and time and location. For example, the view requester accesses a specific web page and specifies the desired event history, and the recording processing unit 53 may provide the view requester with a user interface (UI) on the web page. At this time, the request includes a combination of identification information (e.g., IP address) of the source viewing location and identification information (e.g., IP address) of the destination viewing location, specifying the necessity for viewing status, such as which viewing location watched which viewing location and how. In this process, the storage / reading unit 59 retrieves the corresponding recorded file by searching the recorded file storage DB 5004 (see FIG. 16I) using the identification information (IP address) of the source viewing location included in the request as a search key. As described above, the read recording file contains the entire spherical video even if it is a spherical image. This makes it possible to obtain a recording file obtained from the original spherical video only when requested by a viewer.

[0224] Next, the storage and reading unit 59 searches the predetermined area management DB 5003P (see FIG. 16H) using the identification information (IP address, an example of source identification information or camera device identification information) of the image sender (source viewing location) and the identification information (IP address, an example of destination identification information or communication terminal identification information) of the image destination (destination viewing location) included in the user's request as search keys, thereby reading out corresponding time series information (timestamp) (step S403). Note that it is assumed that the communication terminal of the view requester has shared predetermined area information indicating the predetermined area image as performed in FIG. 23 or 30. Therefore, the storage and reading unit 59 can use the source identification information (IP address of the image sender, also an example of camera device identification information) and the destination identification information (IP address of the image destination, also an example of communication terminal identification information) specified by the communication terminal of the view requester in the processing of step S403 as search keys.

[0225] Next, the image conversion processing unit 57 combines the captured image data (recorded file) read in step S402 with the time-series information (timestamp) of the predetermined area read in step S403, and performs perspective projection transformation using the predetermined area information of the timestamp corresponding to each video frame. After performing perspective projection transformation, the image conversion processing unit 57 acquires planar image data. Then, the recording processing unit 53 generates new recorded files that reflect the viewing status of each user (viewer) at each viewing location and adds and stores them in the recorded file storage DB 5004 (see FIG. 16I) (step S404). At this time, the recording processing unit 53, in cooperation with the storage / reading unit 59, may record the new recorded file by associating a "post" processing flag (information) with the live / post flag item corresponding to the file name. This allows the storage / reading unit 59 to manage whether the saved recorded file name is a recorded file recorded and saved during live recording or post-conversion recording. The new recording file generated in step S404 is treated as an example of post-conversion predetermined area image data, and the image (video) included in the new recording file is treated as an example of post-conversion predetermined area image.

[0226] After executing the processes of steps S403 and S404 for all the video frames of the recording file read in step S402, the transmitting / receiving unit 51 transmits the converted recording file (converted predetermined area image data) that reflects the viewing status of each newly created user (viewer) to the person requesting viewing (step S405). The method of transmitting the recording file to the user may be, for example, to generate a download URL that can be used for a limited time and notify the user by a registered email address, but is not limited to this.

[0227] The processes of steps S302-1 and S302-2 described in FIG. 32 are executed every time media data transmitted from each communication terminal is received during real-time distribution.

[0228] As explained above, in the recording method using post-conversion recording, only each transmitted video is recorded during real-time distribution, so there is no duplication of the captured image data to be recorded, and processing can be performed efficiently. Also, since the time-series information of a specified area generally has a smaller data volume than video data, it is possible to reduce the memory capacity required for recording even if there are many viewing locations.

[0229] <Select between live recording and post-conversion recording> Here, an example will be given to explain when and how to switch between live recording and post-conversion recording. First, consider a case where both live recording and post-conversion recording are possible in the image communication system described in the embodiment. In this case, if the user can select between live recording and post-conversion recording for a specific conference, it is possible to set in advance whether or not live recording mode is required as part of the system settings. If live recording is set in such system settings, the communication management system 5 will execute the live recording mode processing from steps S302-5 to S302-7 in FIG. 32.

[0230] On the other hand, in the case of post-conversion recording mode, if the image communication system is to support it as a product later, it may be possible to order it if necessary (if the setting is not live recording, post-conversion recording can be set).

[0231] Furthermore, when cost is taken into consideration, there may be a selection mode in which neither the live recording mode for live recording nor the post-conversion recording mode for post-conversion recording is performed.

[0232] 31, the transmitter / receiver 51 of the communication management system 5 transmits captured image data and audio data to the communication terminals (smartphone 9, PC 7, and videoconference terminal 3d) participating in the same video call (steps S303, S304, and S305). Each of these transmission processes includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmitter / receiver 91 of the smartphone 9, the transmitter / receiver 71 of the PC 7, and the transmitter / receiver 31d of the videoconference terminal 3d each receive the captured image data, image data ID, and audio data.

[0233] ●Screen display example● Next, an example of a UI screen that a user can view when post-conversion recording is performed will be described. FIG. 34 shows an example of a file selection screen on an app for viewing post-conversion recordings. In FIG. 34, when a view requester accesses a specified site work history page on the web app, a live streaming video retransmission service screen 10001 is displayed for retransmitting live streaming video from a construction site, building site, or the like where real-time streaming was performed. Note that the communication terminal or the like used by the view requester to access the screen may be, for example, a smartphone 9. In this case, the display control unit 94 of the smartphone 9 controls the display 917 to display the live streaming video retransmission service screen 10001. This live streaming video retransmission service screen 10001 displays a video (video) file list 10002 that lists items such as date and time, location, event, video file, and selection checkbox. A thumbnail image representing a portion of the video (video) may be attached to the video (video) file item so that the view requester can identify which video (video) they wish to view again. Here, an example is shown in which the view requester has selected an original video file with video file name: 10101.mp4, whose sender identification information (IP address) is indicated by 1.2.1.3, and checked the checkbox. In this state, by operating (pressing, tapping, etc.) the "View" button 10003, the screen can transition to a video playback screen 10011 shown in FIG. 35, which will be described later. The view requester can also transition from this screen to another screen by operating the "Close" button 1004. Note that the items managed and displayed in the video (video) file list 10002 are not limited to those described above.

[0234] ●Screen display example● FIG. 35 is an example of a video playback screen for post-conversion recording selected in an app. As shown in FIG. 35, the video file (1850.mp4) selected by the viewer is played in a video playback area 10012 on a video playback screen 10011. The video playback screen 10011 also displays a file information display field 10013 for the video file being played, a "Download" button 10014, a "Close" button 10015, and a destination location selection section 10016. In the case of post-conversion recording, a list of viewing locations participating in a specific event (or at a specific location) is required in FIG. 35. Therefore, the destination location selection section 10016 is displayed to prompt the viewer to select a destination location along with the selection of the video (video) itself captured at the specific event (or specific location). This is to perform perspective projection transformation on the captured video (video) based on viewpoint information indicating who was viewing the video (video), and to allow the viewer to select whether or not to play the video (video). In other words, the destination location information corresponds to the information on the "destination location" described in step S403 of FIG. 33 (e.g., an IP address representing the destination identification information). In the example shown in FIG. 35, when a view requester selects the destination IP address: 1.2.2.1 in the destination location selection unit 10016 as the destination location information by checking the box, the video file: 1850.mp4 (recorded file storage DB 5004, see FIG. 16I) at the destination viewing location (viewpoint) becomes playable from the beginning (a circle appears at the beginning of the video playback time scale). Note that the destination location selection unit 10016 is not limited to the video playback screen 10011, and may be provided in the live streaming video retransmission service screen 10001 shown in FIG. 34. This allows the view requester to play the video file (1850.mp4) in the video playback area 10012 and check the screen, audio, and viewpoint direction on the screen at the desired date and time associated with the timestamp recorded in the video file. Although an IP address has been mentioned above as an example of the destination base information, the destination base information may also be a user ID (user identification information) of a user at the destination base.

[0235] The display control unit 94 of the smartphone 9 may add a timestamp item to the video (movie) file list 10002 shown in FIG. 34. Specifically, the display control unit 94 of the smartphone 9 and the recording / conversion processing unit of the communication management system 5 may display items identical to the timestamp items managed in the predetermined area management DB 5003P (see FIG. 16H). Thus, when a view requester selects the date and time of a timestamp he or she wishes to view from among the displayed timestamp items, video around the selected timestamp (e.g., five minutes before and after the date and time indicated by the timestamp) may be played. By incorporating such a function into the execution of post-conversion recording, it becomes possible to save only videos around the time period desired by the view requester, thereby saving memory capacity in the communication management system 5. Furthermore, if the view requester has certain authority, he or she can select a file name displayed in the file information display field 10013 and operate the “Download” button 10014 to download the selected video file (the video file currently being played) to the communication terminal and repeatedly play it in a local environment. On the other hand, by operating the "close" button 10015, the user can return to the live streaming video retransmission service screen 10001.

[0236] <<Main Effects of This Embodiment>> As described above, according to this embodiment, the communication management system 5 receives the captured image data including the image data ID and sound data obtained by collecting sound transmitted from the image capturing device 1a via the transmitting / receiving unit 51 (step S301), then performs perspective projection transformation via the image conversion processing unit 57 based on the recording file in which the captured image data is recorded and the time-series information of the predetermined area (step S404), and transmits the converted predetermined area image data (converted recording file) that reflects the viewing status of each communication terminal, newly generated by the transmitting / receiving unit 51, to the communication terminal (step S405). This provides the advantage that, even when a user reviews the captured image some time after distribution, the user can easily grasp when and from what viewpoint the captured image was viewed.

[0237] 〔supplement〕 In each of the above-described embodiments, the predetermined region T is identified by predetermined region information indicating the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE, but this is not limiting. For example, when using a constant angle of view, the predetermined region T may be identified by predetermined point information indicating the center point CP or any of the four corners of the rectangular predetermined region T in FIG. 7. Note that the "predetermined point information" includes the predetermined region information.

[0238] Furthermore, in the above-described embodiments, a three-dimensional spherical panoramic image is described as an example of a panoramic image, but the captured image (whole image) may be a two-dimensional panoramic image.

[0239] Furthermore, in each of the above-described embodiments, the communication management system 5 relays the specified area information transmitted from each communication terminal, but this is not limited to this, and each communication terminal may transmit and receive the specified area information directly to each other.

[0240] The functions of each embodiment can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C++, C#, or Java (registered trademark), or an object-oriented programming language, and the program for executing the functions of each embodiment can be distributed via telecommunications lines.

[0241] In addition, the programs for executing the functions of each embodiment can also be stored in and distributed on recording media such as ROM, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), flash memory, flexible disk, CD (Compact Disc)-ROM, CD-RW (Re-Writable), DVD-ROM, DVD-RAM, DVD-RW, Blu-ray Disc, SD card, and MO (Magneto-Optical disc).

[0242] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices designed to execute each of the above-described functions, such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), a SOC (System on a Chip), a GPU (Graphics Processing Unit), and conventional circuit modules.

[0243] Up to this point, an image communication system, a communication management device, a communication management method, and a program according to one embodiment of the present invention have been described. However, the present invention is not limited to the above-described embodiment, and other modifications, changes, deletions, and other changes can be made within the scope that can be conceived by a person skilled in the art. In any aspect, as long as the functions and effects of the present invention are achieved, the present invention is applicable. It is included in the range of. [Explanation of symbols]

[0244] 1a Imaging device 1b Imaging device 3a Video conference terminal (an example of a third communication terminal / an example of a communication terminal (own terminal)) 3D video conferencing terminal (an example of another communication terminal) 4a Display 4D display (an example of a display method) 5. Communication management system (an example of a communication management device) 7 PC 8. Imaging Device 9 Smartphone (an example of a communication terminal (own terminal)) 31d Transmitter / receiver unit (an example of a transmitting means, an example of a receiving means) 3001d Image type management DB (an example of image type management means) 3002d Imaging device management DB (an example of imaging device management means) 3003d Predetermined area management DB (an example of a predetermined area management means) 3004d Required resolution management DB (an example of required resolution management means) m11 Display direction mark (example of direction information) m14 Display direction mark (example of direction information) m41 Point of interest mark (example of corresponding position information) 51 Transmitting / receiving unit (an example of receiving means, an example of transmitting means) 53 Recording processing unit (an example of a recording means) 57 Image conversion processing unit (an example of image conversion means) 59 Memory readout unit (an example of memory readout means) [Prior art documents] [Patent documents]

[0245] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-178135 [Patent Document 2] Japanese Patent Application Laid-Open No. 2005-101980

Claims

1. An image communication system including: a photographing device that obtains photographed image data by photographing; a communication terminal that displays a predetermined area image that is an image of a predetermined area of ​​the photographed image related to the photographed image data; and a communication management device that manages predetermined area information that represents the predetermined area, The communication management device a receiving means for receiving the photographed image data photographed by the photographing device and time series information representing a date and time associated with the predetermined area information transmitted by the communication terminal; a transmitting means for transmitting to the communication terminal transformed image data of a predetermined area that has been subjected to perspective projection transformation based on the received photographed image data and the time-series information; having An image communication system comprising:

2. The receiving means further receiving photographing device identification information for identifying the photographing device and communication terminal identification information for identifying the communication terminal; The transmitting means transmitting the time-series information associated with the photographing device identification information and the communication terminal identification information, and the transformed predetermined area image data that has been subjected to perspective projection transformation based on the photographed image data, to the communication terminal; 2. The image communication system according to claim 1.

3. The transmitting means When the receiving means receives an acquisition request for the converted image data of the predetermined area from the communication terminal, the receiving means transmits the converted image data of the predetermined area to the communication terminal.

3. An image communication system according to claim 1 or 2.

4. The time series information is A date and time associated with the photographing device identification information and the communication terminal identification information.

4. The image communication system according to claim 1, wherein the image communication system is a communication system for communicating with a plurality of people.

5. 5. An image communication system according to claim 1, The communication management device further an image conversion means for converting the photographed image data into the converted predetermined area image data based on the time series information; An image communication system comprising:

6. The image conversion means converting the photographed image data into the converted predetermined area image data using predetermined area information in the latest time series information among the time series information; 6. The image communication system according to claim 5.

7. the imaging device is a spherical imaging device capable of capturing a three-dimensional spherical panoramic image, 7. The image communication system according to claim 1, wherein the image communication system is a communication system for communicating with a plurality of people.

8. A communication management device that manages predetermined area information representing a predetermined area transmitted by a communication terminal that displays a predetermined area image, which is an image of a predetermined area of ​​an image captured by an imaging device, a receiving means for receiving the photographed image data photographed by the photographing device and time series information representing a date and time associated with the predetermined area information transmitted by the communication terminal; a transmitting means for transmitting to the communication terminal transformed image data of a predetermined area that has been subjected to perspective projection transformation based on the received photographed image data and the time-series information; having A communication management device characterized by:

9. A communication management method executed by a communication management device that manages predetermined area information representing a predetermined area transmitted by a communication terminal that displays a predetermined area image, which is an image of a predetermined area of ​​an image captured by an imaging device, a receiving step of receiving the photographed image data photographed by the photographing device and time series information representing a date and time associated with the predetermined area information transmitted by the communication terminal; a transmitting step of transmitting, to the communication terminal, transformed predetermined area image data that has been subjected to perspective projection transformation based on the received photographed image data and the time-series information; Performing a process that includes A communication management method comprising:

10. a communication management device that manages predetermined area information representing a predetermined area transmitted by a communication terminal that displays a predetermined area image, which is an image of a predetermined area of ​​an image captured by an image capturing device; a receiving step of receiving the photographed image data photographed by the photographing device and time series information representing a date and time associated with the predetermined area information transmitted by the communication terminal; a transmitting step of transmitting, to the communication terminal, transformed predetermined area image data that has been subjected to perspective projection transformation based on the received photographed image data and the time-series information; A program for executing steps including:

Images