Communication terminal, image communication system, image display method, and program

Abstract

To display an image in an appropriate display direction according to the reproduction time of video data shot at another base.SOLUTION: A communication terminal 30 capable of displaying a predetermined region of part of a subject appearing in a captured image receives, from a communication management system 50 that manages a captured image distributed from another communication terminal 30, video data including the captured image, and predetermined region information indicating a predetermined region to be displayed by the communication terminal 30 for each reproduction time of the video data, and when there is predetermined region information corresponding to the reproduction time of the received video data, displays an image indicating the predetermined region indicated in the received predetermined region information.SELECTED DRAWING: Figure 30

Description

[Technical Field]

[0001] The present disclosure relates to a communication terminal, an image communication system, an image display method, and a program. [Background technology]

[0002] There are known camera devices that can capture images in all directions using multiple wide-angle or fisheye lenses. There are also known systems that distribute image data captured using such camera devices in real time, allowing users to view the status of the distribution center where the camera devices are installed in real time at a different location.

[0003] Also, a technique is known in which a communication terminal is connected to an imaging device capable of acquiring a celestial sphere panoramic image in real time, the celestial sphere panoramic image acquired from the imaging device is transmitted to each communication terminal of a counterpart, and each communication terminal of the counterpart displays a predetermined area image indicating a predetermined area that is a part of the celestial sphere panoramic image on a display, etc. This allows users at each location to independently determine and display a predetermined area image indicating a predetermined area that interests them within the entire image of the celestial sphere panoramic image.

[0004] Furthermore, in the distribution of spherical panoramic images, viewers can freely change the display direction, so viewers can view the images in different display directions. For example, Patent Document 1 discloses a configuration in which the expansion method and angle of view of omnidirectional image data are changed and recording time data at the time of the change is stored and played back, with the aim of playing back image data that reflects various user production intentions from omnidirectional image data. Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional methods, when distributing video data, there was a problem in that it was not possible to display an appropriate specified area on the communication terminal in accordance with the requests of the distributor or the viewer, such as allowing the viewer to freely operate the display direction on the communication terminal during certain time periods and synchronizing the display direction on the communication terminal during other time periods. [Means for solving the problem]

[0006] In order to solve the above-mentioned problem, the invention of claim 1 is a communication terminal capable of displaying a predetermined area of a part of a subject appearing in a captured image, and is equipped with a receiving means for receiving video data including the captured image and predetermined area information indicating a predetermined area to be displayed on the communication terminal for each playback time of the video data from a communication management system that manages the captured image distributed from another communication terminal, and a display control means for displaying an image indicating the predetermined area indicated in the received predetermined area information if predetermined area information corresponding to the playback time of the received video data exists. [Effects of the Invention]

[0007] According to the present invention, it is possible to display an image in an appropriate display direction in accordance with the playback time of video data captured at another location. [Brief explanation of the drawings]

[0008] [Figure 1] (A) is a left side view of the imaging device, (B) is a front view of the imaging device, and (C) is a plan view of the imaging device. [Figure 2] FIG. 10 is an image diagram of the imaging device in use. [Figure 3] (A) is a hemispherical image (front) taken with a camera, (B) is a hemispherical image (back) taken with a camera, and (C) is an image represented by equirectangular projection. [Figure 4] (A) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (B) is a diagram showing a spherical image. [Figure 5]FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. [Figure 6] 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a state in which an image of a predetermined area is displayed on a display. [Figure 7] 10 is a diagram showing the relationship between predetermined area information and an image of a predetermined area T. FIG. [Figure 8] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 9] FIG. 1 is a diagram illustrating an example of the overall configuration of an image communication system. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of the imaging device. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal and a communication management system. [Figure 12] FIG. 1 is a diagram illustrating an example of a functional configuration of an image communication system. [Figure 13] FIG. 1 is a diagram illustrating an example of a functional configuration of an image communication system. [Figure 14] FIG. 10 is a schematic diagram illustrating an example of a photographing device management table. [Figure 15] FIG. 10 is a conceptual diagram illustrating an example of an image type management table. [Figure 16] FIG. 10 is a conceptual diagram illustrating an example of a predetermined area management table. [Figure 17] FIG. 10 is a schematic diagram illustrating an example of a session management table. [Figure 18] FIG. 10 is a conceptual diagram illustrating an example of an image type management table. [Figure 19] FIG. 10 is a conceptual diagram illustrating an example of a video data management table. [Figure 20] FIG. 10 is a schematic diagram illustrating an example of a predetermined area management table. [Figure 21] FIG. 10 is a sequence diagram illustrating an example of a process for joining a specific communication session in the image communication system. [Figure 22] FIG. 10 is a diagram illustrating an example of a communication session selection screen. [Figure 23]FIG. 10 is a sequence diagram illustrating an example of a management process of image type information in the image communication system. [Figure 24] FIG. 10 is a sequence diagram showing an example of a communication process of captured image data and sound data in the image communication system. [Figure 25] FIG. 10 is a diagram illustrating an example of a display screen displayed on the communication terminal. [Figure 26] 10 is a flowchart illustrating an example of a recording process of video data. [Figure 27] FIG. 10 is a sequence diagram illustrating an example of a process for setting a distribution condition. [Figure 28] FIG. 10 is a diagram illustrating an example of a distribution condition setting screen. [Figure 29] FIG. 10 is a sequence diagram illustrating an example of a distribution process of recorded video data. [Figure 30] 10 is a flowchart illustrating an example of a playback process of video data in a communication terminal. [Figure 31] 10A and 10B are diagrams illustrating an example of a display screen displayed on a communication terminal when the display directions are synchronized. [Figure 32] 10 is a flowchart illustrating an example of a process for changing a display direction in a communication terminal. [Figure 33] 10A and 10B are diagrams illustrating an example of a display screen displayed on a communication terminal when the display directions are not synchronized. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted.

[0010] ●Embodiment● How to generate spherical images A method for generating a spherical image will be described with reference to FIGS.

[0011] First, the appearance of the imaging device 10 will be described using Fig. 1. The imaging device 10 is a digital camera for obtaining captured images that serve as the basis for spherical (360°) images. Fig. 1(A) is a left side view of the imaging device, Fig. 1(B) is a front view of the imaging device, and Fig. 1(C) is a plan view of the imaging device.

[0012] As shown in FIG. 1(A), the photographing device 10 is small enough to be held in one hand. As shown in FIGS. 1(A), 1(B), and 1(C), the upper portion of the photographing device 10 is provided with an image sensor 103a on the front side (front side) and an image sensor 103b on the rear side (rear side). These image sensors (image sensors) 103a and 103b are used in conjunction with optical components (e.g., lenses 102a and 102b, described below) capable of capturing hemispherical images (with a field of view of 180° or more). As shown in FIG. 1(B), an operation unit 115, such as a shutter button, is provided on the side of the photographing device 10 opposite the front side.

[0013] Next, the usage of the image capturing device 10 will be described with reference to Fig. 2. Fig. 2 is an image diagram of the image capturing device in use. As shown in Fig. 2, the image capturing device 10 is used, for example, by being held by a user and capturing images of subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image capturing element 103a and the image capturing element 103b shown in Fig. 1, respectively.

[0014] Next, an outline of the process of creating a celestial sphere image from an image captured by the image capturing device 10 will be described with reference to Fig. 3 and Fig. 4. Fig. 3(A) is a diagram showing a hemispherical image (front side) captured by the image capturing device, Fig. 3(B) is a diagram showing a hemispherical image (rear side) captured by the image capturing device, and Fig. 3(C) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 4(A) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 4(B) is a diagram showing a celestial sphere image.

[0015] As shown in Fig. 3(A), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by the lens 102a (described later). Also, as shown in Fig. 3(B), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by the lens 102b (described later). Then, the image capturing device 10 combines the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees to create an equirectangular projection image EC as shown in Fig. 3(C).

[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the image capturing device 10 pastes an equirectangular projection image EC so as to cover the spherical surface as shown in FIG. 4(A), and creates a celestial sphere image (celestial sphere panoramic image) CE as shown in FIG. 4(B). In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Also, the celestial sphere image CE may be a still image or a video.

[0017] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people. Therefore, the imaging device 10 can display a predetermined region T (hereinafter referred to as a "predetermined region image") that is a part of the spherical image CE as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to FIGS. 5 to 8.

[0018] FIG. 5 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user viewing a celestial sphere image CE displayed as a three-dimensional sphere. FIG. 6(A) is a three-dimensional perspective view of FIG. 5, and FIG. 6(B) is a diagram showing the predetermined area image when displayed on a display. FIG. 6(A) shows the celestial sphere image CE shown in FIG. 5 as a three-dimensional sphere CS. If the celestial sphere image CE generated in this manner is a three-dimensional sphere CS, the virtual camera IC is located inside the celestial sphere image CE as shown in FIG. 5. The predetermined area T in the celestial sphere image CE is the shooting area of the virtual camera IC, and is specified by predetermined area information that indicates the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE. Zooming of the predetermined area T can also be achieved by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined area image Q is an image of a predetermined area T in the spherical image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the spherical image CE (see FIG. 7).

[0019] The predetermined area image Q shown in FIG. 6(A) is then displayed on a predetermined display as an image of the shooting area of the virtual camera IC, as shown in FIG. 6(B). The image shown in FIG. 6(B) is a predetermined area image represented by the initial (default) predetermined area information. The following explanation will be given using the shooting direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

[0020] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 7. FIG. 7 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 7, "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the angle of view (Angle). That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC, indicated by the shooting direction (ea, aa), becomes the center point CP(x, y) of the predetermined area T, which is the shooting area of the virtual camera IC. As shown in FIG. 7, when the diagonal angle of view of the predetermined area T, represented by the angle of view α of the virtual camera IC, is α, the center point CP(x, y) becomes the parameter ((x, y)) of the predetermined area information. The predetermined area image Q is an image of the predetermined area T in the celestial sphere image CE. f represents the distance from the virtual camera IC to the center point CP(x, y). L represents the distance between any vertex of the predetermined area T and the center point CP(x, y) (2L is the diagonal). In FIG. 7, the trigonometric function shown in the following (Equation 1) generally holds.

[0021]

number

[0022] The image capturing device 10 described above is an example of an image capturing device capable of capturing a wide-angle image, and a spherical image is an example of a wide-angle image. Here, a wide-angle image generally refers to an image captured using a wide-angle lens, which is a lens capable of capturing a wider range than what the human eye can perceive. Furthermore, a wide-angle image generally refers to an image captured using a lens with a focal length of 35 mm or less in 35 mm film equivalent.

[0023] FIG. 8 is a diagram showing points in a three-dimensional Euclidean space in spherical coordinates. Here, the position coordinates of the center point CP when expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius vector, polar angle, and azimuth angle, respectively. The radius vector r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to the center point CP, and is therefore equal to f. FIG. 8 is a diagram showing these relationships. Hereinafter, the explanation will be given using the position coordinates (r, θ, φ) of the virtual camera IC.

[0024] ●Outline of the image communication system Next, an outline of the configuration of an image communication system according to an embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram illustrating an example of the overall configuration of an image communication system. The image communication system 1 illustrated in Fig. 9 is a system for bidirectionally transmitting and receiving captured images such as video between multiple locations, and is a system that allows a captured image such as video distributed from a location to be displayed at another location, thereby enabling a user to view a wide-range image (for example, a spherical image) captured at the other location. The image communication system 1, for example, allows an image captured at a predetermined location such as an office floor to be viewed at another location located remotely. Note that the location where the image is captured is not limited to an office floor, and may be a school, factory, warehouse, construction site, server room, store, or any other space where a user (viewer) at a viewing location needs to understand the situation at a remote location.

[0025] As shown in Figure 9, the image communication system 1 includes an imaging device 10 (10A, 10B, hereinafter referred to as imaging device 10 when there is no need to distinguish between them) located at each of multiple locations (location A, location B), a communication management system 50, and a communication terminal 30 (30A, 30C, 30D, hereinafter referred to as communication terminal 30 when there is no need to distinguish between them) located at each of multiple locations (location A, location C, location D).

[0026] The communication terminals 30 and the communication management system 50 that constitute the image communication system 1 can communicate via a communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), etc. Note that the communication network 100 may include not only wired communication networks but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0027] Of these, the photographing device 10 is a special digital camera that captures a subject, a landscape, or the like to obtain two hemispherical images that serve as the basis for a celestial sphere image, as described above. The photographed images obtained by the photographing device 10 may be moving or still images, or may be both moving and still images. Furthermore, the photographed images may be video containing both images and audio. The communication terminal 30 distributes images acquired from the photographing device 10 via a wired cable, such as a Universal Serial Bus (USB) cable, to communication terminals 30 at other locations via a communication management system 50. In this example, the photographing device 10A and the communication terminal 30A are located at location A where users A1 and A2 are located. Meanwhile, the photographing device 10B is located at location B where user B1 is located, and can directly communicate with the communication management system 50 via the communication network 100. The number of locations where the photographing devices 10 are installed is not limited to two, and may be one location or three or more locations. The connection between the photographing device 10 and the communication terminal 30 may be wireless, such as via short-range wireless communication, rather than a wired connection using a wired cable.

[0028] The communication terminal 30 is a computer such as a PC used by a user at each base. The communication terminal 30 displays an image (still image or video) distributed from another base. The communication terminal 30 acquires, for example, a spherical image, which is a captured image captured by the imaging device 10, via the communication network 100. Furthermore, the communication terminal 30 has OpenGL ES installed therein, and can generate predetermined area information indicating a partial area of the spherical image, or generate a predetermined area image from a spherical image transmitted from another communication terminal 30. That is, the communication terminal 30 can display a predetermined area of a part of a subject appearing in the spherical image, which is a captured image.

[0029] In this example, communication terminal 30A is placed at viewing location A where users A1 and A2 are located, and communication terminal 30C is placed at viewing location C where user C1 is located. Furthermore, communication terminal 30D is placed at viewing location D where users D1, D2, and D3 are located. Communication terminals 30C and 30D, which are placed at locations C and D where no image capturing device 10 is present, distribute images captured at their locations by their own cameras or the like to other locations. Note that at location A where both image capturing device 10A and communication terminal 30A are present, communication terminal 30A may distribute images captured by its own camera or the like to other locations along with images captured by image capturing device 10A (panoramic images).

[0030] The arrangement of the terminals and devices (communication terminals and image capture devices) and users shown in FIG. 9 is merely an example, and other examples may be used. Furthermore, the communication terminal 30 is not limited to a PC, and may be, for example, a tablet terminal, a smartphone, a wearable device, a PJ (Projector), an IWB (Interactive White Board: an electronic whiteboard with a blackboard function that allows intercommunication), or an autonomous robot. Furthermore, if the image capture device 10 is provided with a display unit, it may be configured to display images distributed from other locations.

[0031] The communication management system 50 manages and controls communications between the communication terminals 30 at each location, and manages the types of image data (general images and special images) that are transmitted and received. In this example, the special images are spherical images. The communication management system 50 is installed at a service company that provides image communication services.

[0032] The communication management system 50 may be implemented by a single computer, or may be implemented by multiple computers to which each unit (function or means) is divided and arbitrarily assigned. All or part of the functions of the communication management system 50 may be implemented by a server computer in a cloud environment or on-premise environment.

[0033] In a conventional system for distributing video data including spherical images, viewers can freely change the display direction of the video data, allowing viewers at different locations to view images in different display directions, whether the video data is being distributed in real time or in a recorded format. Meanwhile, in a system for distributing such spherical images, a distributor may want to synchronize the display direction with the viewer during a time period when an object the distributor wants the viewer to view is present or during a time period when an object the distributor does not want the viewer to view is present. In such cases, conventional methods have a problem in that they are unable to display an appropriate predetermined area on a communication terminal in response to the request of the distributor or the viewer, such as by allowing the viewer to freely change the display direction during certain time periods and synchronizing the display direction during other time periods.

[0034] Therefore, in the image communication system 1, the communication management system 50 stores predetermined area information corresponding to the playback time of video data including a spherical image transmitted from the communication terminal 30, and when recording and distributing the video data, transmits the predetermined area information corresponding to the playback time to the communication terminal 30 together with the video data. When playing back the received video data, the communication terminal 30 synchronizes the display direction of the video data if predetermined area information corresponding to the playback time exists, and allows the user to freely operate the display direction of the video data if predetermined area information corresponding to the playback time does not exist. In this way, when distributing video data including a recorded spherical image, the image communication system 1 reflects the synchronization of the display direction in the recorded video data, and can allow the viewer of the image to freely operate the display direction of the video data during a certain time period and synchronize the display direction during another time period.

[0035] ●Hardware configuration Next, the hardware configuration of each device or terminal constituting the image communication system according to the embodiment will be described with reference to Figures 10 and 11. Note that components may be added or deleted from the hardware configurations shown in Figures 10 and 11 as needed.

[0036] ○Hardware configuration of the imaging device○ First, the hardware configuration of the image capturing device 10 will be described using Figure 10. Figure 10 is a diagram showing an example of the hardware configuration of an image capturing device. In the following, the image capturing device 10 is assumed to be an omnidirectional (omnidirectional) image capturing device using two image capturing elements, but the number of image capturing elements may be two or more. Furthermore, the image capturing device does not necessarily have to be a device dedicated to omnidirectional image capturing; an omnidirectional image capturing unit may be attached to a regular digital camera, smartphone, or the like to have substantially the same functions as the image capturing device 10.

[0037] As shown in FIG. 10, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0038] Of these, imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) each having a field angle of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each lens. Imaging elements 103a and 103b each include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by lenses 102a and 102b into image data in the form of an electrical signal and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set.

[0039] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a network I / F 121, and the like.

[0040] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create equirectangular projection image data such as that shown in Figure 3(C).

[0041] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0042] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function for displaying moving images on a display (for example, a display of an external terminal such as a smartphone that performs short-range communication with the imaging device 10 using the short-range communication circuit 117). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0043] As will be described later, the imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b. Although the imaging device 10 is not provided with a display unit in this embodiment, a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 receives the sound data output from the microphone 108 via an I / F bus and performs predetermined processing on the sound data.

[0044] The CPU 111 controls the overall operation of the image capturing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of processed equirectangular projection images.

[0045] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. The user operates the operation unit 115 to input various shooting modes, shooting conditions, etc.

[0046] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as needed.

[0047] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the image capturing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0048] The electronic compass 118 calculates the orientation of the image capture device 10 from the Earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) conforming to the Exif standard and is used for image processing such as image correction of captured images. The related information also includes data such as the capture date and time of the image and the data size of the image data. The gyro sensor 119 detects angle changes (roll angle, pitch angle, and yaw angle) associated with the movement of the image capture device 10. The angle changes are an example of related information (metadata) conforming to the Exif standard and are used for image processing such as image correction of captured images. The acceleration sensor 120 detects acceleration in three axial directions. The image capture device 10 calculates the attitude (angle with respect to the direction of gravity) of its own device (the image capture device 10) based on the acceleration detected by the acceleration sensor 120. The inclusion of the acceleration sensor 120 in the image capture device 10 improves the accuracy of image correction. The network I / F 121 is an interface for data communication using a communication network 100 such as the Internet via a router or the like.

[0049] ○Hardware configuration of communication terminal○ Fig. 11 is a diagram showing an example of the hardware configuration of a communication terminal. Each piece of hardware configuration of communication terminal 30 is indicated by a reference number in the 300 series. Communication terminal 30 is constructed by a computer, and as shown in Fig. 11, includes CPU 301, ROM 302, RAM 303, HD (Hard Disk) 304, HDD controller 305, display 306, external device connection I / F 308, network I / F 309, bus line 310, keyboard 311, pointing device 312, DVD-RW (Digital Versatile Disk Rewritable) drive 314, media I / F 316, audio input / output I / F 317, microphone 318, speaker 319, near-field communication circuit 320, and camera 321.

[0050] Of these, the CPU 301 controls the overall operation of the communication terminal 30. The ROM 302 stores programs, such as an IPL, used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data, such as programs. The HDD controller 305 controls the reading and writing of various data from and to the HD 304 under the control of the CPU 301. The display 306 displays various information, such as a cursor, menus, windows, characters, or images. The display 306 is an example of a display unit. The display 306 may also be a touch panel display equipped with input means. The external device connection I / F 308 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory or a printer. The network I / F 309 is an interface for data communication using the communication network 100. The bus line 310 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 301, shown in FIG. 12 .

[0051] The keyboard 311 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. Note that the input means may be not only the keyboard 311 and the pointing device 312, but also a touch panel, a voice input device, etc. The DVD-RW drive 314 controls the reading and writing of various data from a DVD-RW 313, which is an example of a removable recording medium. Note that the medium is not limited to a DVD-RW, but may also be a DVD-R or a Blu-ray (registered trademark) Disc. The media I / F 316 controls the reading and writing (storage) of data from a recording medium 315, such as a flash memory. The microphone 318 is a type of built-in sound collection means for inputting audio. The audio input / output I / F 317 is a circuit that processes the input and output of audio signals between the microphone 318 and a speaker 319 under the control of the CPU 301. The short-range communication circuit 320 is a communication circuit for communicating with an external terminal (device) by short-range wireless communication technology such as NFC, Bluetooth, or Wi-Fi. The camera 321 is a type of built-in imaging means that captures an image of a subject and obtains image data. Note that the microphone 318, speaker 319, and camera 321 do not have to be built into the communication terminal 30, but may be external devices.

[0052] ○Hardware configuration of communication management system○ Fig. 11 is a diagram showing an example of the hardware configuration of a communication management system. Each piece of hardware configuration in the communication management system 50 is indicated by a reference number in the 500 range in parentheses. The communication management system 50 is constructed using a computer, and as shown in Fig. 12, has the same configuration as the communication terminal 30, so a description of each piece of hardware configuration will be omitted.

[0053] Each of the above programs may be recorded as an installable or executable file on a computer-readable recording medium and distributed. Examples of recording media include CD-Rs (Compact Disc Recordables), DVDs (Digital Versatile Disks), Blu-ray Discs, SD cards, and USB memory. The recording media may also be provided domestically or internationally as a program product. For example, the communication terminal 30 executes the program according to the present invention to realize the image display method according to the present invention.

[0054] ●Function configuration Next, the functional configuration of the image communication system according to the embodiment will be described with reference to Fig. 12 to Fig. 20. Fig. 12 and Fig. 13 are diagrams showing an example of the functional configuration of the image communication system. Fig. 12 and Fig. 13 show devices or terminals shown in Fig. 9 that are related to the processing or operation described below.

[0055] ○ Functional configuration of the imaging device ○ First, the functional configuration of the photographing device 10 will be described with reference to Fig. 12. The photographing device 10 has a communication unit 11, a reception unit 12, an imaging unit 13, a sound collection unit 14, and a storage / readout unit 19. Each of these units is a function or means realized when any of the components shown in Fig. 10 operates in response to an instruction from the CPU 111 in accordance with a program for the photographing device loaded from the SRAM 113 onto the DRAM 114. The photographing device 10 also has a storage unit 1000 constructed by the ROM 112, SRAM 113, and DRAM 114 shown in Fig. 11. The storage unit 1000 stores the GUID (Globally Unique Identifier) of the photographing device itself.

[0056] The communication unit 11 is mainly realized by the processing of the CPU 111, and communicates various data or information with other devices or terminals. The communication unit 11 performs data communication with other devices or terminals using short-range wireless communication technology, for example, using a short-range communication circuit 117. The communication unit 11 also performs data communication with other devices or terminals via various cables, for example, using an input / output I / F 116. Furthermore, the communication unit 11 performs data communication with other devices or terminals via a communication network 100, using a network I / F 121.

[0057] The reception unit 12 is mainly realized by the processing of the CPU 111 on the operation unit 115, and receives various selections or inputs from the user. The imaging unit 13 is mainly realized by the processing of the CPU 111 on the imaging unit 101, the image processing unit 104, and the imaging control unit 105, and captures an image of a subject such as a landscape and acquires captured image data. The sound collection unit 14 is mainly realized by the processing of the CPU 111 on the microphone 108 and the sound processing unit 109, and collects sounds around the imaging device 10.

[0058] The storage / readout unit 19 is mainly realized by the processing of the CPU 111 , and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000 .

[0059] ○Functional configuration of communication terminal○ Next, the functional configuration of communication terminal 30 will be described with reference to Fig. 12. Communication terminal 30 has a transmitting / receiving unit 31, a receiving unit 32, an image / sound processing unit 33, a display control unit 34, a determining unit 35, a creating unit 36, a generating unit 37, a communication unit 38, an imaging unit 41, a sound collecting unit 42, and a storage / reading unit 39. Each of these units is a function or means realized by operating any of the components shown in Fig. 11 in response to an instruction from CPU 301 in accordance with a communication terminal program loaded from HD 304 onto RAM 303. Communication terminal 30 also has a storage unit 3000 constructed by ROM 302, RAM 303, and HD 304 shown in Fig. 11.

[0060] The transmitting / receiving unit 31 is mainly realized by the processing of the CPU 301 for the network I / F 309, and transmits and receives various data or information to and from other devices or terminals via the communication network 100.

[0061] The reception unit 32 is mainly realized by the processing of the CPU 301 on the keyboard 311 or the pointing device 312, and receives various selections or inputs from the user.

[0062] The image / sound processing unit 33 is mainly realized by processing of the CPU 301, and performs image processing on captured image data acquired by the image capturing device 10 or the camera 321 capturing an image of a subject. After the user's voice is converted into an audio signal by the microphone 318, the image / sound processing unit 33 performs audio processing on audio data related to this audio signal. For example, the image / sound processing unit 33 performs image processing on the captured image data received from the image capturing device 10 or captured by the camera 321 based on image type information such as a source name so that the display control unit 34 can display an image on the display 306. Specifically, when the image type information indicates a special image, the image / sound processing unit 33 creates omnidirectional image data by converting the captured image data (for example, data of each hemispherical image as shown in FIGS. 3(A) and (B)) into omnidirectional image data as shown in FIG. 4(B).

[0063] The image / sound processor 33 also performs image processing on captured image data distributed from another communication terminal 30. Furthermore, the image / sound processor 33 outputs an audio signal related to audio data distributed from another communication terminal 30 via the communication management system 50 to the speaker 319, causing the speaker 319 to output audio.

[0064] The display control unit 34 is mainly realized by the processing of the CPU 301, and displays various images, characters, etc. on the display 306. The determination unit 35 is realized by the processing of the CPU 301, and makes various determinations. The determination unit 35 determines, for example, the type of image related to the captured image data received from the imaging device 10 or captured by the camera 321.

[0065] Creation unit 36 is mainly realized by the processing of CPU 301, and creates a source name, which is an example of image type information, in accordance with a naming rule based on the result of determination by determination unit 35 as either a general image or a special image (here, a spherical image). For example, if determination unit 35 determines that the image is a general image, creation unit 36 creates a source name "Video" indicating that it is a general image. On the other hand, if determination unit 35 determines that the image is a special image, creation unit 36 creates a source name "Video_Theta" indicating that it is a special image.

[0066] Generation unit 37 is mainly realized by processing of CPU 301, and generates a predetermined area image to be displayed on display 306. To display, for example, predetermined area information indicating a predetermined area in a captured image (spherical image) captured by imaging device 10 or an image of a predetermined area corresponding to predetermined area information received by transmission / reception unit 31 on display 306, generation unit 37 generates a predetermined area image corresponding to the predetermined area information by performing perspective projection transformation on the captured image (spherical image) using the predetermined area information.

[0067] The communication unit 38 is mainly realized by processing of the CPU 301 on the short-range communication circuit 320, and communicates with the communication unit 11 of the photographing device 10 using short-range wireless communication technology such as NFC, Bluetooth, or WiFi. Note that although the communication unit 38 and the transceiver unit 31 have been described as having separate communication units, they may also be configured to share a common communication unit.

[0068] The imaging unit 41 is mainly realized by the processing of the CPU 301 on the camera 321, and captures an image of a subject such as a landscape and acquires captured image data. The sound collection unit 42 is mainly realized by the processing of the CPU 301 on the microphone 318 and the sound input / output I / F 317, and collects sounds around the communication terminal 30.

[0069] The storage / readout unit 39 is mainly realized by the processing of the CPU 301 , and stores various data (or information) in the storage unit 3000 and reads out various data (or information) from the storage unit 3000 .

[0070] Image type management table FIG. 14 is a conceptual diagram showing an example of an image type management table. The storage unit 3000 stores an image type management DB 3001 configured with the image type management table shown in FIG. 14. This image type management table manages image data IDs, Internet Protocol (IP) addresses (which are an example of destinations of source terminals), and source names in association with each other. The image data ID is an example of image data identification information for identifying image data when distributing images. The same image data ID is attached to image data transmitted from the same source terminal. This allows the destination terminal (receiving communication terminal) to identify the source terminal of the received image data. The IP address of the source terminal indicates the IP address of the photographing device 10 or communication terminal 30 that transmits the image data indicated by the associated image data ID. The source name is a name for identifying the photographing device 10 or camera 321 that outputs the image data indicated by the associated image data ID, and is an example of image type information. The source name is a name created by the communication terminal 30 according to a predetermined naming convention.

[0071] For example, five source terminals with IP addresses "1.2.1.3," "1.2.2.3," "1.3.1.3," "1.3.2.3," and "1.2.2.5" are respectively transmitting image data indicated by image data IDs "RS001," "RS002," "RS003," "RS004," and "RS005." Furthermore, the image types indicated by the source names of the source terminals are "Video_Theta," "Video_Theta," "Video," "Video," and "Video_Theta," which indicate the image types as "special image," "special image," "general image," "general image," and "special image," respectively. In this embodiment, the special image is a spherical image. Note that the IP address is an example of destination information, and the destination information may be a Media Access Control (MAC) address, a terminal ID (identification), or the like. Although the IP address is represented as a simplified version of an IPv4 address, the IP address may also be an IPv6 address. Furthermore, data other than image data may also be managed in association with an image data ID. Data other than image data includes, for example, sound data, document data for screen sharing, and the like.

[0072] Imaging device management table Fig. 15 is a conceptual diagram showing an example of an image capturing device management table. An image capturing device management DB 3002 configured by the image capturing device management table shown in Fig. 15 is created in the storage unit 3000. This image capturing device management table stores and manages the vendor ID and product ID of the GUID of the image capturing device 10 that can obtain two hemispherical images that are the basis of a celestial sphere image. For example, the vendor ID (VID) and product ID (PID) used in a USB device can be used as the GUID. The vendor ID and product ID may be stored in the communication terminal 30 from the factory, or may be additionally stored after the factory.

[0073] Predetermined area management table Fig. 16 is a conceptual diagram showing an example of a predetermined area management table. A predetermined area management DB 3003 configured by the predetermined area management table shown in Fig. 16 is constructed in the storage unit 3000. This predetermined area management table manages, in association with each other, a timestamp indicating the playback time of video data, which is captured image data, and predetermined area information indicating a predetermined area image to be displayed on the communication terminal 30. As shown in Figs. 6 and 7, the predetermined area information is a conversion parameter for converting a captured image into an image of a predetermined area T in the captured image (predetermined area image).

[0074] For example, the first row in the predetermined area management table of Fig. 16 manages that video data having a playback time corresponding to timestamp "10001" is to be displayed in a predetermined area (r=10, θ=20, φ=30). Similarly, the second row in the predetermined area management table of Fig. 16 manages that video data having a playback time corresponding to timestamp "10002" is to be displayed in a predetermined area (r=20, θ=30, φ=40), and the second row in the predetermined area management table of Fig. 16 manages that video data having a playback time corresponding to timestamp "10004" is to be displayed in a predetermined area (r=20, θ=20, φ=30). On the other hand, the third row in the predetermined area management table of Fig. 16 does not store predetermined area information and manages that the display direction of the predetermined area image is not synchronized with the video data having a playback time corresponding to timestamp "10003." In addition, when the transmitting / receiving unit 31 receives new specified area information corresponding to the timestamp of video data that is already being managed, the storage / reading unit 39 rewrites the specified area information that is already being managed with the newly received specified area information.

[0075] ○Functional configuration of communication management system○ Next, the functional configuration of the communication management system 50 will be described with reference to Fig. 13. The communication management system 50 has a transmitting / receiving unit 51, a determining unit 52, a recording processing unit 53, a generating unit 54, a distribution condition setting unit 55, and a storing / reading unit 59. Each of these units is a function or means realized when any of the components shown in Fig. 11 operates in response to an instruction from the CPU 501 in accordance with a communication management system program loaded from the HD 504 onto the RAM 503. The communication management system 50 also has a storage unit 5000 constructed by the ROM 502, RAM 503, and HD 504 shown in Fig. 11.

[0076] The transmitting / receiving unit 51 is mainly realized by the processing of the CPU 501 on the network I / F 509, and transmits and receives various data or information to and from other devices via the communication network 100.

[0077] The determination unit 52 is realized by the processing of the CPU 501, and performs various determinations. The recording processing unit 53 is realized mainly by the processing of the CPU 501, and records video data consisting of captured image data and audio data transmitted from each communication terminal 30. For example, when the data transmitted from each communication terminal 30 is video data, the recording processing unit 53 stores the received video data individually as a recording file, and stores metadata (bibliographic information) in the video data management DB 5003.

[0078] Generation unit 54 is mainly realized by processing of CPU 501, and generates image data ID and predetermined area information. Generation unit 54 generates predetermined area information indicating, for example, a predetermined area (for example, predetermined area T shown in FIG. 5, etc.) in a captured image captured by imaging device 10. Note that an image when the entire captured image is displayed (for example, omnidirectional image CE shown in FIG. 5, etc.) is also referred to as a "whole image."

[0079] The distribution condition setting unit 55 is mainly realized by the processing of the CPU 501, and sets distribution conditions when the video data recorded by the recording processing unit 53 is distributed to the communication terminals 30 at each base. The distribution condition setting unit 55 sets predetermined area information according to the playback time, for example, in association with a timestamp indicating the playback time of the recorded video data.

[0080] The storage / readout unit 59 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000 .

[0081] Session management table FIG. 17 is a conceptual diagram showing an example of a session management table. A session management DB 5001 configured with the session management table shown in FIG. 17 is created in the storage unit 5000. This session management table manages session IDs and IP addresses of participating communication terminals in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session that realizes image communication, and is generated for each virtual room. The session ID is also managed by each communication terminal, and is used when each communication terminal selects a communication session. The IP addresses of participating communication terminals indicate the IP addresses of communication terminals that have participated in the virtual room identified by the associated session ID.

[0082] Image type management table FIG. 18 is a conceptual diagram showing an example of an image type management table. An image type management DB 5002 configured by the image type management table shown in FIG. 18 is constructed in the storage unit 5000. This image type management table manages, in addition to each piece of information managed in the image type management table shown in FIG. 14, a session ID that is the same as the session ID managed in the session management table, in association with the session ID. The reason that the communication management system 50 manages the same image data ID, source terminal IP address, and image type information as those managed by the communication terminal 30 is to transmit image type information, etc., to communication terminals already in a video call and the newly participating communication terminal when a new communication terminal enters a virtual room, for example. This eliminates the need to transmit and receive image type information, etc., between a communication terminal already in a call and the newly participating communication terminal.

[0083] Video data management table Fig. 19 is a conceptual diagram showing an example of a video data management table. A video data management DB 5003 configured by the video data management table shown in Fig. 19 is constructed in the storage unit 5000. This video data management table manages the IP address of the communication terminal 30 or the imaging device 10 that is the sender of the captured image data, in association with the recorded data file of the video data that is the captured image data.

[0084] Predetermined area management table Fig. 20 is a conceptual diagram showing an example of a predetermined area management table. A predetermined area management DB 5004 configured by the predetermined area management table shown in Fig. 20 is constructed in the storage unit 5000. This predetermined area management table manages, for each recorded data file of video data, the IP address of the terminal from which the captured image data is sent, the IP address of the communication terminal 30 to which the captured image data is sent, a timestamp indicating the playback time of the video data that is the captured image data, and predetermined area information indicating the predetermined area image to be displayed on the communication terminal 30, in association with each other. As shown in Figs. 6 and 7, the predetermined area information is a conversion parameter for converting a captured image into an image of a predetermined area T in the captured image (predetermined area image).

[0085] For example, the first row of the predetermined area management table in Fig. 20 manages that video data with a playback time corresponding to timestamp "10001" is transmitted from a communication terminal 30 with an IP address of "1.2.1.3" to a communication terminal 30 with an IP address of "1.3.1.3" via the communication management system 50, and that a predetermined area (r=10, θ=20, φ=30) is displayed on the destination communication terminal 30. Also, as shown in the second and third rows of the predetermined area management table in Fig. 20, when the IP address of the image destination is "all," the communication management system 50 transmits predetermined area information corresponding to the associated timestamp to all communication terminals 30 to which the video data is to be distributed, and causes all communication terminals 30 to display the same predetermined area.

[0086] 20, if no predetermined area information is stored, the communication management system 50 does not transmit predetermined area information corresponding to the associated timestamp to the communication terminal 30 that is the destination of the video data (image destination), and does not synchronize the display direction of the predetermined area image to be displayed on the communication terminal 30. Also, as shown in the fourth and fifth rows of the predetermined area management table in FIG. 20, if different image destinations or predetermined area information are stored for the same image sender and timestamp, the communication management system 50 transmits predetermined area information corresponding to the associated timestamp to the communication terminals 30 that are the corresponding image destinations, and causes each communication terminal 30 to display a different predetermined area.

[0087] Furthermore, when the transmitting / receiving unit 51 receives new predetermined area information corresponding to a set of the IP addresses of the communication terminals that are the sender and receiver of the captured image data and the timestamp of the video data, which are already managed, the storage / reading unit 59 rewrites the already managed predetermined area information with the newly received predetermined area information. Here, the timestamp is an example of time information indicating the playback time of the video data. The timestamp indicates the time indicating the playback position within the total playback time of the video data in the recording data file. Note that the timestamp may be in a format that represents the playback time of the video data by storing the time when the captured image data related to the video data was received from the sender of the image.

[0088] Processing or operation of the embodiment ○Session participation processing○ Next, the processing or operation of the image communication system according to the embodiment will be described with reference to Figs. 21 to 33. In the following description, an example will be described in which an image captured at site A is distributed to each site, but similar processing is performed when an image is distributed from another site. First, the processing for joining a specific communication session will be described with reference to Figs. 21 and 22. Fig. 21 is a sequence diagram showing an example of the processing for joining a specific communication session in the image communication system. Fig. 22 is a diagram showing an example of a communication session selection screen.

[0089] First, a user at site A (e.g., user A1) performs an operation to display a communication session selection screen, and the reception unit 32 receives the operation to display the selection screen, whereby the display control unit 34 of the communication terminal 30A displays the selection screen 800 shown in Fig. 22 on the display 306 (step S11). The selection screen 800 shown in Fig. 22 displays selection buttons 810a, 810b, 810c, etc., which indicate the virtual rooms A1, B1, B2, etc., to be selected. Each selection button 810a, etc. is associated with a corresponding session ID.

[0090] Here, when user A1 selects the desired selection button (here, selection button 810a) for the virtual room that is the base, the reception unit 32 of the communication terminal 30A receives the selection of the communication session (step S12). Then, the transmission / reception unit 31 of the communication terminal 30A transmits a request to join a communication session with another base to the communication management system 50 (step S13). This participation request includes a session ID indicating the communication session whose selection was received in step S12, and the IP address of the communication terminal 30A that is the requesting terminal. As a result, the transmission / reception unit 51 of the communication management system 50 receives the participation request.

[0091] Next, the storage / readout unit 59 of the communication management system 50 performs a process of joining the communication session by adding the IP address received in step S13 to the joining terminal IP address field of the record in the session management DB 5001 (see FIG. 17) that has the same session ID as the session ID received in step S13 (step S14). Then, the transmission / reception unit 51 transmits a join request response to the communication terminal 30A (step S15). This join request response includes the session ID received in step S13 and the result of the join process in step S14. As a result, the transmission / reception unit 31 of the communication terminal 30A receives the join request response. The following describes a case where the join process is successful. Note that the communication terminals 30 at each base perform a process of joining the communication session using a process similar to the process shown in FIG. 21.

[0092] ○Image type information management processing○ Next, the management process of image type information will be described with reference to Fig. 23. Fig. 23 is a sequence diagram showing the management process of image type information.

[0093] First, when a user at site A connects the photographing device 10A to the communication terminal 30A, the storage / readout unit 19 of the photographing device 10A reads out the GUID of its own device (the photographing device 10A) stored in the storage unit 1000. Then, the communication unit 11 of the photographing device 10A transmits the GUID of its own device to the communication terminal 30A (step S31). As a result, the communication unit 38 of the communication terminal 30A receives the GUID of the photographing device 10A.

[0094] Next, the determination unit 35 of the communication terminal 30A determines whether the same vendor ID and product ID as the vendor ID and product ID in the GUID received in step S31 are managed in the photographing device management DB 3002 (see FIG. 15), thereby determining the image type (step S32). Specifically, if the same vendor ID and product ID are managed in the photographing device management DB 3002, the determination unit 35 of the communication terminal 30A determines that the photographing device 10A is a photographing device that captures special images (here, spherical images). On the other hand, if the same vendor ID and product ID are not managed in the photographing device management DB 3002, the determination unit 35 of the communication terminal 30A determines that the photographing device 10A is a photographing device that captures general images.

[0095] Next, the storage / readout unit 39 of the communication terminal 30A associates the IP address of the communication terminal 30A itself (the source terminal) with the image type information determined in step S32 and stores them in the image type management DB 3001 (see FIG. 14) (step S33). In this state, no image data ID is associated. The image type information is, for example, a source name or image type (general image, special image) determined according to a predetermined naming rule.

[0096] Next, the transmitting / receiving unit 31 of the communication terminal 30A transmits a request to add image type information to the communication management system 50 (step S34). This request to add image type information includes the IP address of the communication terminal 30A (the source terminal) and the image type information stored in step S33. As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the request to add image type information.

[0097] Next, the storage / reading unit 59 of the communication management system 50 searches the session management DB 5001 (see FIG. 17) using the IP address of the source terminal received in step S34 as a search key, and reads out the corresponding session ID (step S35).

[0098] Next, the generation unit 54 generates a unique image data ID (step S36). Then, the storage / reading unit 59 associates the session ID read out in step S35, the image data ID generated in step S36, and the IP address of the transmission source terminal and image type information received in step S34 and stores them as a new record in the image type management DB 5002 (see FIG. 18) (step S37). Then, the transmission / reception unit 51 transmits the image data ID generated in step S36 to the communication terminal 30A (step S38). As a result, the transmission / reception unit 31 of the communication terminal 30A receives the image data ID.

[0099] Next, the storage / readout unit 39 of the communication terminal 30A stores the image data ID received in step S38 in the image type management DB 3001 (see FIG. 14) in association with the IP address and image type information of the communication terminal itself (communication terminal 30A), which is the sending terminal stored in step S33 (step S39).

[0100] Meanwhile, the transmitter / receiver 51 of the communication management system 50 transmits an addition notification of the image type information to another communication terminal (here, communication terminal 30D) (step S40). This addition notification of the image type information includes the image data ID generated in step S36, as well as the IP address and image type information of communication terminal 30A, which is the transmission source terminal, stored in step S37. As a result, the transmitter / receiver 31 of communication terminal 30D receives the addition notification of the image type information. Note that the transmission destination of the transmitter / receiver 51 is another IP address that is associated in the session management DB 5001 (see FIG. 17) with the same session ID as the IP address of communication terminal 30A. In other words, the transmission destination is another communication terminal that is in the same virtual room as communication terminal 30A.

[0101] Next, storage / readout unit 39 of communications terminal 30D associates the image data ID, the IP address of communications terminal 30A, which is the transmission source terminal, and the image type information received in step S40, and stores them as a new record in image type management DB 3001D (see FIG. 18) (step S41). Similarly, an additional communication of image type information is sent to communications terminal 30C, which is another communications terminal, and stored in image type management DB 3001C. As described above, each communications terminal can share the same information in each of image type management DBs 3001A, 3001C, and 3001D.

[0102] ○Communication processing of captured image data○ Next, a process in which photographed image data and sound data obtained at site A are transmitted to each communication terminal (communication terminals 30C, 30D) via the communication management system 50 will be described with reference to Figures 24 and 25. Figure 24 is a sequence diagram showing an example of a communication process for photographed image data and sound data in an image communication system. Note that while Figure 24 shows an example in which photographed image data obtained by one photographing device 10 is distributed to each communication terminal 30, the same process is also performed when multiple photographed image data obtained by other photographing devices 10 installed at the same site are distributed.

[0103] First, communication unit 11 of photographing device 10A transmits photographed image data acquired by photographing a subject, scenery, or the like, and sound data acquired by collecting sound to communication terminal 30A (step S51). In this case, photographing device 10A is a device that can acquire two hemispherical images that are the basis for a celestial sphere image, and therefore, as shown in Figures 3(A) and (B), the photographed image data is composed of data for the two hemispherical images. As a result, communication unit 38 of communication terminal 30A receives the photographed image data and sound data.

[0104] Next, the transmitter / receiver 31 of the communication terminal 30A transmits the captured image data and sound data sent from the photographing device 10A to the communication management system 50 (step S52). This transmission includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmitter / receiver 51 of the communication management system 50 receives the captured image data, sound data, and the image data ID.

[0105] Next, the transmitter / receiver 51 of the communication management system 50 transmits the captured image data and sound data to the communication terminals (communication terminals 30C and 30D) participating in the same session as communication terminal 30A (steps S53 and S54). Each of these transmissions includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmitter / receiver 51 of communication terminal 30C and communication terminal 30D each receive the captured image data, sound data, and image data ID.

[0106] Here, a display example of the communication terminal 30 at each base will be described with reference to Fig. 25. Fig. 25 is a diagram showing an example of a display screen displayed on the communication terminal. Display screen 200 displays captured image data captured at each base. For example, if the captured image captured at a predetermined base is a spherical image, display screen 200 displays an image in which the spherical image and the predetermined area image have been generated from the captured image data sent from another base.

[0107] Here, when the captured image data sent from the photographing devices 10A and 10B capable of capturing a spherical image are displayed as they are, the images at the points A and B are displayed as a front hemispherical image and a rear hemispherical image, as shown in Fig. 3(A) and (B), respectively. In contrast, when the generation unit 37 generates a spherical image and further generates a predetermined area image from the photographing image data output by the photographing devices 10A and 10B that can obtain two hemispherical images that are the basis of the spherical image, the predetermined area image, which is a planar image, is displayed, as shown in Fig. 25. Note that, at the point C, the communication terminal 30C that obtains a general image takes the image, and at the point D, the communication terminal 30D that obtains a general image takes the image, so a general image (here, a planar image) is displayed.

[0108] Furthermore, users at each location can change the predetermined area of the predetermined area image in the same celestial sphere image. For example, the user at each location can operate an input means such as the pointing device 312, causing the reception unit 32 to receive an instruction to move the predetermined area image, and the display control unit 35 can shift, rotate, reduce, or enlarge the predetermined area image.

[0109] An image of location A is displayed in the display area (layout number "1") on the left side of display screen 200, and an image of location B is displayed in the display area on the upper right side (layout number "2". Furthermore, an image of location C is displayed in the display area on the middle right side of display screen 200 (layout number "3"), and an image of location D is displayed in the display area on the lower right side (layout number "4".) The display area with layout number "1" is the main display area, and the display areas with layout numbers "2", "3", and "4" are sub-display areas. The images in the main display area and the sub-display area can be changed on each communication terminal. Normally, at each location, an image of the location where the central person in the video call is located is displayed in the main display area.

[0110] 25 are examples of special image identification icons for indicating that the celestial sphere icons 210a and 210b are predetermined area images that indicate a predetermined area T that is a part of a celestial sphere image. The display positions of the celestial sphere icons 210a and 210b do not have to be the upper right, but may be anywhere, such as the upper left, lower left, or lower right. The types of the celestial sphere icons 210a and 210b are not limited to those shown in FIG. 25. Instead of the celestial sphere icons 210a and 210b, characters such as "celestial sphere image" may be used, or a combination of an icon and characters may be used.

[0111] ○Video data recording processing Next, a process for recording video data transmitted from the communication terminals 30 at each base in the communication management system 50 will be described with reference to Fig. 26. Fig. 26 is a flowchart showing an example of the video data recording process. The video data recording process shown in Fig. 26 is executed when the communication management system 50 receives captured image data, as in step 52 of Fig. 25.

[0112] First, the determination unit 52 of the communication management system 50 determines whether the data transmitted from the communication terminal 30 (for example, the communication terminal 30A in FIG. 24) that transmits the captured image data, etc., is video data. Then, when the determination unit 52 determines that the video data has been received by the transmitting / receiving unit 51 (YES in step S71), the process proceeds to step S72.

[0113] Next, the recording processing unit 53 stores the data file of the video data received by the transmitting / receiving unit 51 in the video data management DB 5003 (see FIG. 19) (step S72). In this case, the recording processing unit 53 associates the IP address of the sender of the received video data with the recording data file (e.g., file name) of the received video data, and stores them in the video data management DB 5003. For example, the recording processing unit 53 combines the captured image data and sound data transmitted from the communication terminals 30 at the respective bases, records them as one recording data file for each base, and stores them in the video data management DB 5003.

[0114] On the other hand, if the determination unit 52 determines that the data received by the transmission / reception unit 51 is not video data (NO in step S71), the process proceeds to step S73.

[0115] Next, the determination unit 52 determines whether or not predetermined area information transmitted from a communication terminal 30 viewing captured image data distributed from another location (for example, communication terminal 30C or communication terminal 30D in FIG. 24 ) has been received. When the display direction of a celestial sphere image is synchronized with that of another communication terminal 30 in live distribution of video data, the communication terminal 30 transmits the predetermined area information to the communication management system 50 to constantly notify which celestial sphere image the communication terminal 30 is viewing. When the determination unit 52 determines that the predetermined area information transmitted from the communication terminal 30 by the transmission / reception unit 51 has been received (YES in step S73), the determination unit 52 proceeds to step S74. Then, the storage / readout unit 59 stores the predetermined area information received by the transmission / reception unit 51 in the predetermined area management DB 5004 (see FIG. 20 ) (step S74). In this case, the storage / reading unit 59 stores the received specified area information in the specified area management DB 5004 in association with the IP addresses of the sender and destination of the video data corresponding to the received specified area information, as well as a timestamp indicating the playback time of the video data corresponding to the received specified area information.

[0116] On the other hand, in step S73, if the determination unit 52 determines that the predetermined area information has not been received by the transmission / reception unit 51 (NO in step S73), the process ends. Note that, in the case of live distribution of video data, if the display direction of the omnidirectional image is not synchronized with that of other communication terminals 30, the communication terminal 30 performs control such that the predetermined area information is not transmitted or the predetermined area information is not stored in the communication management system 50.

[0117] ○Distribution condition setting process○ Next, a process for setting distribution conditions for recorded video data will be described with reference to Fig. 27 and Fig. 28. Fig. 27 is a sequence diagram showing an example of the process for setting distribution conditions. Fig. 27 describes a process in which user A1 located at site A uses communication terminal 30A to set distribution conditions for video data consisting of a celestial sphere image captured by imaging device 10A.

[0118] First, in response to a predetermined input operation by user A1, the transmitting / receiving unit 31 of the communication terminal 30A transmits a distribution condition setting request to the communication management system 50 (step S91). As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the distribution condition setting request transmitted from the communication terminal 30A.

[0119] Next, the storage / readout unit 59 of the communication management system 50 searches the video data management DB 5003 (see FIG. 19) using the IP address of the communication terminal 30A that sent the distribution condition setting request received in step S91 as a search key, and reads out the data file of the video data associated with the IP address of the communication terminal 30A (step S92). Then, the transmission / reception unit 51 transmits the video data read out in step S92 to the communication terminal 30A that sent the request (step S93). As a result, the transmission / reception unit 31 of the communication terminal 30A receives the video data sent from the communication management system 50.

[0120] Next, the display control unit 34 of the communication terminal 30A displays a distribution condition setting screen 700 on the display 306 (step S94). FIG. 28 is a diagram showing an example of the distribution condition setting screen. The distribution condition setting screen 700 shown in FIG. 28 is a screen for setting a predetermined area to be displayed on the communication terminal 30 when distributing the video data to each location while playing back recorded video data. The distribution condition setting screen 700 includes an image display area 710 that displays an image related to the video data being played back, a playback position display area 730 that displays the playback position of the video data, a predetermined area information display area 740 that indicates a predetermined area corresponding to the display direction displayed in the image display area 710, a set button 750 that is pressed when setting the distribution conditions, and an end button 755 that is pressed when ending the setting of the distribution conditions.

[0121] Of these, the image display area 710 includes display direction change buttons 720 (720a, 720b, 720c, 720d) for changing the display direction of the video data. User A1 can change the display direction of the image displayed in the image display area 710, for example, by selecting a desired display direction change button 720. The playback location display area 730 also includes a play button 731 that is pressed to play video data, a pause button 732 that is pressed to stop playback of the video data, and a slider 735 that indicates the playback history (playback location) over the entire playback time. The slider 735 constitutes a seek bar. The seek bar displays the playback location of the video data and is an operation area for specifying the playback location of the video data. User A1 can visually grasp, for example, which part of the video data is being played from the beginning to the end, based on the position of the slider 735. Furthermore, the user A1 can move the slider 735 using an input means such as the pointing device 312, thereby playing back the video data from any desired playback point.

[0122] The predetermined area information display area 740 shows predetermined area information corresponding to the predetermined area image displayed in the image display area 710. In the predetermined area information display area 740, the value of the predetermined area corresponding to the displayed predetermined area image is changed as needed, for example, by user A1 selecting and moving the display direction change button 720 to change the display direction of the image displayed in the image display area 710. Note that user A1 may manually input the numerical value of the predetermined area to be displayed in the predetermined area information display area 740 to change the display direction of the image displayed in the image display area 710.

[0123] The reception unit 32 of the communication terminal 30A receives the input of the distribution conditions when the user A1 selects the display direction change button 720 or inputs a value in the predetermined area information display area 740 and presses the set button 750 (step S95). In this case, the reception unit 32 receives the predetermined area information selected or input by the user A1 as the distribution conditions. Then, the transmission / reception unit 31 of the communication terminal 30A transmits the predetermined area information input in step S95 and time information indicating the playback time of the video data corresponding to the input predetermined area information to the communication management system 50 (step S96). As a result, the transmission / reception unit 51 of the communication management system 50 receives the predetermined area information and time information transmitted from the communication terminal 30A.

[0124] Then, the distribution condition setting unit 55 of the communication management system 50 stores the timestamp corresponding to the predetermined area information and time information received in step S96 in the predetermined area management DB 5004 (see FIG. 20) in association with the IP address of the communication terminal 30A that is the sender of the distribution condition setting request received in step S91 (step S97). In this case, the distribution condition setting unit 55 sets the IP address of the image transmission destination to be stored in association with the received predetermined area information to "all," for example. When storing predetermined area information corresponding to the pair of the sender of the captured image data and the timestamp of the video data that is already managed in the predetermined area management DB 5004, the distribution condition setting unit 55 rewrites the already managed predetermined area information with the newly received predetermined area information.

[0125] If an IP address of a specific image destination corresponding to the set of predetermined area information to be stored in the predetermined area management DB 5004 already exists, the distribution condition setting unit 55 may use the existing IP address of the specific image destination as is, without setting the IP address of the image source to "all." Also, the distribution condition setting unit 55 may be configured to have user A1 input the IP address of the image destination using the distribution condition setting screen 700 shown in Fig. 28, for example, and store the IP address of the specific image destination in association with the received predetermined area information.

[0126] ○Distribution processing of recorded data○ Next, a process for distributing video data recorded in the communication management system 50 will be described with reference to Fig. 29 to Fig. 33. Fig. 29 is a sequence diagram showing an example of the process for distributing recorded video data. The example in Fig. 29 describes a process for distributing video data recorded by the communication management system 50 to a communication terminal 30C at a location C.

[0127] First, the transmitter / receiver 31 of the communication terminal 30C transmits a video distribution request to the communication management system 50 in response to a predetermined input operation by the user C1 (step S111). This video distribution request includes information for identifying the video data requested to be distributed (for example, the IP address of the sender of the video data or the file name of the video data). As a result, the transmitter / receiver 51 of the communication management system 50 receives the video distribution request transmitted from the communication terminal 30C.

[0128] Next, the storage / reading unit 59 of the communication management system 50 reads, from the predetermined area management DB 5004 (see FIG. 20), predetermined area information corresponding to the video data that is the target of the video distribution request received in step S92 (step S112). In this case, the storage / reading unit 59 reads the target predetermined area information along with the timestamp associated with the predetermined area information. Then, the transmission / reception unit 51 transmits, to the requesting communication terminal 30C, a set of time information indicating the playback time specified by the predetermined area information read in step S112 and the timestamp associated with the predetermined area information (step S113). As a result, the transmission / reception unit 31 of the communication terminal 30C receives the set of predetermined area information and time information transmitted from the communication management system 50. Note that the communication management system 50 may perform the processes of steps S113 and S114 on all target predetermined area information stored in the predetermined area management DB 5004 at once, or may perform the processes sequentially at regular intervals (e.g., 30 seconds) according to the playback time of the video data.

[0129] Next, the storage / readout unit 39 of the communication terminal 30C associates the predetermined area information and the time information received in step S113 with a time stamp corresponding to the information and stores them in the predetermined area management DB 3003C (see FIG. 16) (step S114).

[0130] Next, the storage / readout unit 59 of the communication management system 50 reads out from the video data management DB 5003 (see FIG. 19) the recorded data file of the video data that is the target of the video distribution request received in step S92 (step S115). Then, the transmission / reception unit 51 sequentially transmits captured image data and audio data related to the video data to the requesting communication terminal 30C based on the recorded data file read out in step S115 (step S116). Each transmission includes an image data ID that identifies the captured image data to be transmitted. As a result, the transmission / reception unit 31 of the communication terminal 30C receives the captured image data and audio data transmitted from the communication management system 50.

[0131] Then, the display control unit 34 of the communication terminal 30C plays back the video data received in step S116 to display the captured image (step S117). Note that, if there are multiple recorded data files that were recorded simultaneously in the same session, the communication management system 50 may simultaneously distribute them and combine and display them on the viewing communication terminal 30C. In addition, in response to the video distribution request in step S111, the communication management system 50 may distribute data to individual communication terminals 30, or may distribute data synchronously to multiple communication terminals 30.

[0132] Here, examples of displaying recorded video data on communication terminal 30C will be described with reference to Fig. 30 to Fig. 33. First, playback of video data when the display direction of the video data is synchronized will be described with reference to Fig. 30 and Fig. 31. Fig. 30 is a flowchart showing an example of playback processing of video data on a communication terminal.

[0133] First, as shown in step S113 of Fig. 29, if the communication terminal 30C receives the predetermined area information transmitted from the communication management system 50 (YES in step S131), the communication terminal 30C proceeds to step S132. On the other hand, if the communication terminal 30C has not received the predetermined area information (NO in step S131), the communication terminal 30C ends the process.

[0134] Next, as shown in step S114 of FIG. 29, the storage / reading unit 39 of the communication terminal 30C stores the specified area information received in step S131 in the specified area management DB 3003C in association with a timestamp indicating the playback time of the video data corresponding to the specified area information (step S132).

[0135] Next, the determination unit 35 of the communication terminal 30C refers to the predetermined area management DB 3003C in which the predetermined area information is stored in step S132, and if there is predetermined area information corresponding to the timestamp of the video data transmitted from the communication management system 50 (YES in step S133), the determination unit 35 of the communication terminal 30C transitions to step S134. On the other hand, if there is predetermined area information corresponding to the timestamp of the video data transmitted from the communication management system 50 in step S132, the determination unit 35 of the communication terminal 30C refers to the predetermined area management DB 3003C in which the predetermined area information is stored in step S132, and there is no predetermined area information corresponding to the timestamp of the video data transmitted from the communication management system 50 (NO in step S133), the determination unit 35 ends the process without synchronizing the display direction of the video data.

[0136] Next, the generation unit 37 of the communication terminal 30C synchronizes the display direction of the video data and generates a predetermined area image by performing perspective projection transformation using the predetermined area information received in step S132 in order to display an image of the predetermined area identified by the predetermined area information stored in step S132 (step S134). This allows the communication terminal 30C to generate a predetermined area image corresponding to the distribution conditions set by the communication management system 50.

[0137] Then, the display control unit 35 causes the predetermined area image generated in step S134 to be displayed on the display 306 (step S135).

[0138] FIG. 31 is a diagram showing an example of a display screen displayed on a communication terminal when the display direction is synchronized. Display screen 400 shown in FIG. 31 shows an example in which video data whose display direction is synchronized for a predetermined playback time is displayed. Display screen 400 includes a predetermined area image 410 generated from the video data in step S135, a synchronized icon 415 indicating that the display direction is synchronized, and a playback point display area 430 that displays the playback point of the video data. Of these, playback point display area 430 includes a play button 431 that is pressed to play video data, a pause button 432 that is pressed to stop playback of the video data, and a slider 435 that indicates the playback history (playback points) over the entire playback time. The configurations of play button 431, pause button 432, and slider 435 are similar to the configurations of play button 731, pause button 732, and slider 735 shown in FIG. 28.

[0139] In this way, by having communication terminal 30C display a predetermined area image corresponding to the distribution conditions set by communication management system 50, communication management system 50 can synchronize the display direction of video data displayed on communication terminal 30. In this case, user C1 cannot change the display direction while viewing predetermined area image 410 displayed on display screen 400, and can understand that the display direction of the video data cannot be changed by checking synchronized icon 415 displayed on display screen 400. Furthermore, the predetermined area information transmitted from communication management system 50 to communication terminal 30C may be information transmitted from the viewing communication terminal 30 during live distribution of video data and stored in communication management system 50, as shown in step S74 of FIG. 26, or may be information stored in communication management system 50 by the process of setting distribution conditions, as shown in step S97 of FIG. 27.

[0140] Next, playback of video data when the display direction is not synchronized will be described with reference to Figures 32 and 33. Figure 32 is a flowchart showing an example of processing for changing the display direction in a communication terminal.

[0141] First, if the display direction of the video data transmitted from the communication management system 50 is synchronized (YES in step S151), the communication terminal 30C ends the process. In this case, the communication terminal 30C displays a predetermined area image corresponding to the synchronized display direction, as shown in steps S134 and S135 of FIG. 30. On the other hand, if the display direction of the video data is not synchronized (NO in step S151), the communication terminal 30C proceeds to step S152.

[0142] Next, communication terminal 30C determines whether a request to change the display direction of the video data has been received (step S152). FIG. 33 is a diagram showing an example of a display screen displayed on the communication terminal when the display direction is not synchronized. Display screen 400 shown in FIG. 33 includes a predetermined area image 410, which is an image of a predetermined area of the video data received in step S116, display direction change buttons 420 (420a, 420b, 420c, 420d) for changing the display direction of the video data, and a playback position display area 430, which displays the playback position of the video data. User C1 can change the display direction of the video data corresponding to the predetermined area image 410 by, for example, selecting the desired display direction change button 420. Note that playback position display area 430 has the same configuration as that shown in FIG. 31.

[0143] If the reception unit 32 of the communication terminal 30C receives a request to change the display orientation by the user C1 selecting the display orientation change button 420 (YES in step S152), the process proceeds to step S153. On the other hand, if the reception unit 32 of the communication terminal 30C does not receive a request to change the display orientation (NO in step S152), the process ends.

[0144] Next, the storage / reading unit 39 of the communication terminal 30C stores the predetermined area information indicating the predetermined area corresponding to the display direction accepted in step S152 in the predetermined area management DB 3003C in association with a timestamp indicating the playback time of the video data corresponding to the predetermined area information (step S153).

[0145] Next, the generation unit 37 of the communication terminal 30C generates a predetermined area image by performing perspective projection transformation using the predetermined area information stored in step S153 to display an image of the predetermined area identified by the predetermined area information stored in step S153 (step S154). Then, the display control unit 35 displays the predetermined area image generated in step S154 on the display 306 (step S155).

[0146] In this way, when the display direction of the video data is not synchronized by the communication management system 50, the communication terminal 30C can display the predetermined area image corresponding to the desired display direction through an input operation by the user C1.

[0147] Effect of the embodiment As described above, when distributing video data including a recorded spherical image, the image communication system 1 reflects synchronization of the display orientation in the recorded video data, allows the viewer of the image to freely operate the display orientation of the video data during a certain time period, and synchronizes the display orientation during another time period. This allows the image communication system 1 to display an image with an appropriate display orientation on the communication terminal 30 according to the playback time of the video data captured at another location.

[0148] In addition, the image communication system 1 can synchronize the display direction of the recorded video data on the communication terminal 30 by having the communication management system 50 set specified area information corresponding to the playback time of the video data when recording and distributing it for the recorded data file of the video data transmitted from the imaging device 10 or communication terminal 30 at each base.

[0149] In the above embodiment, the predetermined region T is identified by predetermined region information indicating the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE, but the present invention is not limited to this. For example, when using a constant angle of view, the predetermined region T may be identified by predetermined point information indicating the center point CP or any of the four corners of the rectangular predetermined region T in Fig. 7. Furthermore, in the above embodiment, a case has been described in which the captured image (whole image) is a three-dimensional panoramic image as an example of the omnidirectional image (omnidirectional panoramic image), but a two-dimensional panoramic image may also be used.

[0150] ●Additional Information● Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in the present embodiment includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices designed to perform each function described above, such as an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a system on a chip (SOC), a graphics processing unit (GPU), and a conventional circuit module.

[0151] Furthermore, the various tables in the above-described embodiments may be generated by the learning effects of machine learning, and tables may not be used by classifying data for each associated item using machine learning. Here, machine learning refers to a technology that allows a computer to acquire human-like learning capabilities, in which the computer autonomously generates algorithms necessary for judgments such as data classification from previously acquired learning data and applies these algorithms to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. Any learning method for machine learning is acceptable.

[0152] So far, we have described a communication terminal, an image communication system, an image display method, and a program according to one embodiment of the present invention, but the present invention is not limited to the above-described embodiment, and other embodiments can be added, modified, or deleted, etc., within the scope that can be conceived by a person skilled in the art, and any aspect is included in the scope of the present invention as long as it achieves the functions and effects of the present invention. [Explanation of symbols]

[0153] 1. Image communication system 10 Imaging equipment 30 Communication terminal 31 Transmitting / receiving unit (an example of receiving means) 32 Reception unit (an example of reception means) 34 Display control unit (an example of a display control means) 37 Generation unit (an example of generation means) 50 Communication Management System 51 Transmitting / receiving unit (an example of a photographed image data receiving means, an example of a video data transmitting means) 53 Recording processing unit (an example of a recording processing means) 55 Distribution condition setting unit (an example of setting means) [Prior art documents] [Patent documents]

[0154] [Patent Document 1] JP 2016-25640 A

Claims

1. A communication terminal capable of displaying a predetermined area of ​​a captured image, a receiving means for receiving predetermined area information corresponding to a playback time of the video data including the captured image transmitted from another communication terminal and the video data; a display control means for, when predetermined area information corresponding to a playback time of the received video data exists, synchronizing a display direction of the video data based on the predetermined area information corresponding to the playback time and playing back the video data; a receiving means for receiving a selection of a predetermined area in which the received video data is to be displayed when there is no predetermined area information for the playback time of the received video data; The display control means displays a predetermined area image indicating the predetermined area for which the selection has been accepted.

2. 2. The communication terminal according to claim 1, further comprising: a generating means for generating a predetermined area image showing the predetermined area shown in the received predetermined area information when predetermined area information at a playback time corresponding to the received predetermined area information exists; The display control means is a communication terminal that displays the generated predetermined area image.

3. 2. The communication terminal according to claim 1, further comprising: a generation unit that generates a predetermined area image showing the predetermined area for which the selection has been accepted; The display control means is a communication terminal that displays the generated predetermined area image.

4. The communication terminal according to claim 1 , wherein the video data is recorded data distributed by another communication terminal.

5. The communication terminal according to claim 1 , wherein the captured image is a spherical image.

6. An image communication system that transmits video data including a captured image from a first communication terminal to a second communication terminal, The first communication terminal a transmitting means for transmitting predetermined area information corresponding to a playback time of the video data including the captured image and the video data to the second communication terminal; The second communication terminal a receiving means for receiving predetermined area information corresponding to a playback time of the video data including the captured image transmitted from another communication terminal and the video data; a display control means for, when predetermined area information corresponding to a playback time of the received video data exists, synchronizing a display direction of the video data based on the predetermined area information corresponding to the playback time and playing back the video data; a receiving means for receiving a selection of a predetermined area in which the received video data is to be displayed when there is no predetermined area information for the playback time of the received video data; The display control means displays a predetermined area image showing the predetermined area for which the selection has been accepted.

7. An image display method executed by a communication terminal capable of displaying a predetermined area of ​​a captured image, comprising: a receiving step of receiving predetermined area information corresponding to a playback time of the video data including the captured image transmitted from another communication terminal and the video data; a display control step of playing back the video data by synchronizing a display direction of the video data based on the predetermined area information corresponding to the playback time when predetermined area information corresponding to the playback time of the received video data exists; If there is no predetermined area information for the playback time of the received video data, a receiving step is performed to receive a selection of a predetermined area in which the received video data is to be displayed; The display control step is an image display method for displaying a predetermined area image showing the predetermined area for which the selection has been accepted.

8. A communication terminal capable of displaying a predetermined area of ​​a photographed image, a receiving step of receiving predetermined area information corresponding to a playback time of the video data including the captured image transmitted from another communication terminal and the video data; a display control step of playing back the video data by synchronizing a display direction of the video data based on the predetermined area information corresponding to the playback time when predetermined area information corresponding to the playback time of the received video data exists; If there is no predetermined area information for the playback time of the received video data, a receiving step is executed to receive a selection of a predetermined area in which the received video data is to be displayed; The display control step is a program for displaying a predetermined area image showing the predetermined area for which the selection has been accepted.

Images