Program, shared communication terminal, display method, and image communication system

Abstract

To provide an image communication system capable of allowing an image capturing source to specify a target point in real time.SOLUTION: An image communication system 10 including an imaging apparatus 5 for capturing an image and communication terminals 1 to 4 for displaying the image on a screen includes a target point definition part 194 for defining the target point in the image, an attitude information acquisition part 15a for acquiring the attitude information of the imaging apparatus, a target point conversion part 196 for converting the target point defined by the target point definition part according to the attitude information acquired by the attitude information acquisition part, a cut-out processing part 220 for cutting out a predetermined area including the target point converted by the target point conversion part from the image, and an output part 224 for displaying the predetermined area including the target point cut out by the cut-out processing part on the screen.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus, an image communication system, an image processing method, and a program. [Background technology]

[0002] There is known an imaging device that uses multiple wide-angle lenses, such as fisheye lenses and ultra-wide-angle lenses, to capture 360 ​​degrees around or in all directions (hereinafter referred to as a sphere) at once. In this imaging device, images from each lens are projected onto each imaging element, and the resulting images are stitched together by image processing to generate a sphere image. For example, a sphere image in which a subject in the sphere is captured can be generated using two wide-angle lenses with a viewing angle of more than 180 degrees.

[0003] However, since a spherical image generated by an imaging device is expressed in a spherical coordinate system, whereas a normal display device such as a monitor is flat, it is difficult to display the entire image at once. For this reason, when displaying a spherical image, a dedicated viewer is usually used that converts a part of the angle of view of the spherical image into an image suitable for a normal display device.

[0004] A technique for determining a point of interest has been devised so that a terminal can display an arbitrary angle of view of a spherical image (see, for example, Patent Document 1). Patent Document 1 discloses a method in which a user changes the point of interest by tilting or rotating an imaging device, and the point of interest is displayed on a normal display device. Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, it is difficult for the image capture source to specify the point of interest in real time. As an example of a use case where it is desirable to specify a point of interest in real time, a real estate viewing will be described.

[0006] When the imaging device is used for viewing real estate, the imaging device can capture an image of the entire room of the property, so that a customer who wants to view the property can get an overall view of the property without having to travel to the property, thereby saving the trouble of traveling. For example, viewing can be achieved by the following mechanism. 1. A real estate agent visits the property and takes pictures of the property using an imaging device. 2. Customers who wish to view the property receive the images from the imaging device on a communication terminal and view them remotely in real time. 3. Not only video but also audio is connected, so customers can view the property while talking to the real estate agent.

[0007] However, because customers can only see a certain angle of the spherical image, they may not be looking at the part the real estate agent is explaining, and it is difficult to display that part immediately on a flat device. In other words, customers do not know where to focus on in the spherical image. For example, even if a local real estate agent says, "This room is equipped with the latest air conditioner! This is it!", a customer viewing the property remotely may not be looking in the direction of the air conditioner in the spherical image, making it difficult to communicate about the designated focus points in real time.

[0008] To address this problem, if a real estate agent displays a preview of a spherical image on a communication terminal carried by the real estate agent and performs an operation to indicate a point of interest, the communication terminal of the real estate agent can transmit the point of interest to the communication terminal of the customer who is viewing the property remotely, and the real estate agent and the customer can focus on the same point of interest. However, it is difficult for the real estate agent to operate the communication terminal while capturing images with an imaging device. This is possible if multiple real estate agents go to the site, but it is often difficult to secure enough personnel.

[0009] There is technology that can estimate points of interest by analyzing spherical images, but since this is only an estimate, the points described by the real estate agent may not necessarily be the points of interest.

[0010] In view of the above-mentioned problems, an object of the present invention is to provide an image communication system that enables the image capturer to specify a point of interest in real time. [Means for solving the problem]

[0011] In view of the above problems, the present invention is an image communication system having an imaging device that captures an image and a communication terminal that displays the image on a screen, characterized in that it has an attention point definition unit that defines an attention point in the image, an attitude information acquisition unit that acquires attitude information of the imaging device, an attention point conversion unit that converts the attention point defined by the attention point definition unit in accordance with the attitude information acquired by the attitude information acquisition unit, a cut-out processing unit that cuts out a specified area including the attention point converted by the attention point conversion unit from the image, and an output unit that displays the specified area including the attention point cut out by the cut-out processing unit on the screen. Effect of the Invention

[0012] It is possible to provide an image communication system that enables the image capturer to specify a point of interest in real time. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating a schematic operation of an image communication system that transmits a celestial sphere image in which a point of interest in an arbitrary direction is specified between a site A and a site B. [Diagram 2] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. [Diagram 3] FIG. 1 is a conceptual diagram of an imaging device in use. [Figure 4] (a) is a hemispherical image (front) captured by an imaging device, (b) is a hemispherical image (back) captured by an imaging device, and (c) is an image represented by equirectangular projection. [Diagram 5] (a) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (b) is a diagram showing a spherical panoramic image. [Figure 6] FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere panoramic image is a three-dimensional solid sphere. [Figure 7]FIG. 7(a) is a three-dimensional perspective view of FIG. 6, and FIG. 7(b) is a diagram showing a state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 8] 13 is a diagram showing the relationship between predetermined region information and an image of a predetermined region T. FIG. [Figure 9] This is a diagram showing points in three-dimensional Euclidean space in spherical coordinates. [Figure 10] 1 is a schematic diagram of a configuration of an image communication system according to an embodiment of the present invention. [Figure 11] FIG. 2 is a diagram illustrating a hardware configuration of an example of an imaging device. [Figure 12] FIG. 1 is a diagram illustrating an example of a hardware configuration of the communication terminals 3 and 4. [Figure 13] FIG. 2 is a diagram illustrating a hardware configuration of an example of a communication management system. [Figure 14] 2 is a diagram illustrating an example of a hardware configuration of communication terminals 1 and 2. FIG. [Figure 15] FIG. 1 is a functional block diagram (part 1) of an example showing functions of an image communication system in block form. [Figure 16] FIG. 2 is a functional block diagram (part 2) of an example showing functions of the image communication system in blocks. [Figure 17] FIG. 13 is a conceptual diagram illustrating an image type management table. [Figure 18] FIG. 13 is a conceptual diagram illustrating an imaging device management table. [Figure 19] FIG. 13 is a conceptual diagram showing a predetermined area information management table. [Figure 20] FIG. 13 is a conceptual diagram illustrating a session management table. [Figure 21] FIG. 13 is a conceptual diagram illustrating an image type management table. [Figure 22] FIG. 13 is a conceptual diagram showing a predetermined area information management table. [Figure 23] FIG. 11 is a sequence diagram illustrating an example of a process for joining a specific communication session. [Figure 24] FIG. 13 is a diagram showing an example of a selection screen for a communication session (virtual conference room). [Diagram 25] FIG. 2 is a diagram illustrating an example of coordinate axes of an imaging device. [Figure 26] 1 is a diagram illustrating an example of a reference attitude of an imaging device; [Figure 27] 4 is a diagram illustrating an example of values ​​detected by an acceleration / azimuth sensor. FIG. [Figure 28] (a) is a conceptual diagram of a photographer pointing at a target, (b) is an example of an equirectangular image in which a celestial sphere image is represented in equirectangular projection without zenith correction, and (c) is an example of an equirectangular image in which a celestial sphere image is represented in equirectangular projection with zenith correction. [Figure 29] 2 is a functional block diagram of an example of an image processing unit included in the imaging device. FIG. [Diagram 30] 1A and 1B are diagrams illustrating an example of a conversion table and an example of conversion from a planar coordinate system to a spherical coordinate system. [Diagram 31] 11A and 11B are diagrams illustrating an example of correction of a conversion table based on posture information. [Diagram 32] 10A to 10C are diagrams illustrating an example of image rotation processing by an image rotation unit; [Diagram 33] FIG. 11 is a sequence diagram illustrating an example of a communication process of spherical image and sound data in a video call. [Diagram 34] FIG. 11 is a flowchart illustrating an example of a process in which an imaging device generates a celestial sphere image and a process in which a communication terminal displays the celestial sphere image. [Diagram 35] FIG. 2 is a diagram showing an example of a video display screen displayed on a display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image processing system and an image processing method performed by the image processing system will be described below with reference to the drawings as an example of an embodiment of the present invention.

[0015] <Outline of the image communication system> First, a schematic operation of the image communication system 10 will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining a schematic operation of the image communication system 10 that transmits a celestial sphere image with a point of interest in an arbitrary direction specified between a site A and a site B.

[0016] (1) The imaging device 5a of this embodiment has predefined coordinates of a point of interest. The coordinates of the point of interest are, for example, x and y coordinates indicating a pixel or a certain range of an imaging element in which a subject is captured in the longitudinal direction (upper side) of the imaging device 5a. The coordinates of the point of interest are coordinates on the imaging element and are preset at the time of shipment.

[0017] (2) When the image-capturing person 8 at site A wants the user at site B to pay attention to the air conditioner 140, the image-capturing person 8 faces the longitudinal direction (upward) of the imaging device 5a toward the air conditioner 140. The imaging device 5a constantly acquires orientation information for zenith correction, which will be described later. The imaging device 5a captures a celestial sphere image and performs zenith correction based on the orientation information, so that the air conditioner 140 is not captured tilted. In addition, since the air conditioner 140 is captured in the attention point coordinates, the subject of the attention point coordinates is the air conditioner. Then, when the image-capturing person 8 tilts the imaging device 5a, the change in the attention point coordinates is the orientation information itself. Therefore, the attention point coordinates after conversion of the defined attention point coordinates can be identified by zenith correction.

[0018] (3) The imaging device 5a transmits the spherical image and the converted point of interest coordinates to the communication terminal 2 at the site B via the communication terminal 1.

[0019] (4) The communication terminal 2 cuts out a predetermined range including the coordinates of the point of interest, regardless of the predetermined area image that had been displayed up until that point, generates a predetermined area image, and displays it on a display or the like, so that the photographer 8 can draw the user's attention to an object that he or she wants the user to pay attention to. In addition, the viewer cannot change the predetermined area image generated from the photographer's view information for at least a certain period of time. In other words, the predetermined area image generated from the photographer's view information is forcibly displayed.

[0020] In this way, in the image communication system 10 of this embodiment, the focus point coordinates are predefined in the imaging device 5a, and when the photographer points to the object he or she wants to show using these focus point coordinates, the focus point coordinates are converted according to the posture information and transmitted to the user, so that the photographer 8 can direct the user at site B to the focus point in real time.

[0021] <Terminology> The defined attention point is a point or a certain range on the imaging element as described above. The subject is always captured in the same direction as seen from the imaging device 5a. In this embodiment, the pixel or a certain range of the imaging element capturing the subject on the upper side in the longitudinal direction is defined as the defined attention point, but the location of the attention point can be determined arbitrarily. For example, it may be a pixel of the imaging element capturing the subject in the direction pointed by the lower side in the longitudinal direction, the optical axis direction of the lens, or the direction pointed by the protruding part of the imaging device 5a.

[0022] The orientation information of the imaging device may be any information that can specify the direction in which the imaging device is pointing, such as information indicating how far the imaging device is tilted from an upright position or information regarding the rotation of each axis in three-dimensional space.

[0023] <How to generate spherical images> A method for generating a spherical image will be described with reference to FIGS.

[0024] First, the appearance of the imaging device 5a will be described with reference to Fig. 2. The imaging device 5a is a digital camera for obtaining a captured image that is the basis of a three-dimensional spherical (360°) image. Fig. 2(a) is a left side view of the imaging device 5a, Fig. 2(b) is a front view of the imaging device 5a, and Fig. 2(c) is a plan view of the imaging device 5a.

[0025] As shown in FIG. 2(a), the imaging device 5a is sized so that a person can hold it in one hand. Also, as shown in FIG. 2(a), FIG. 2(b), and FIG. 2(c), an imaging element 103a is provided on the front side (front side) and an imaging element 103b is provided on the back side (rear side) of the upper part of the imaging device 5a. These imaging elements (image sensors) 103a and 103b are used in conjunction with optical members (for example, fisheye lenses 102a and 102b in FIG. 11 described later) capable of capturing hemispherical images (angle of view of 180° or more). Also, as shown in FIG. 2(b), an operation unit 115 such as a shutter button is provided on the surface opposite to the front side of the imaging device 5a.

[0026] Next, the use of the imaging device 5a will be described with reference to Fig. 3. Fig. 3 is a conceptual diagram of the imaging device in use. As shown in Fig. 3, the imaging device 5a is used, for example, by a user holding it in his / her hand to capture an image of a subject around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the imaging element 103a and the imaging element 103b shown in Fig. 2, respectively.

[0027] Next, an outline of the process of creating a celestial sphere image from an image captured by an imaging device 5a will be described with reference to Fig. 4 and Fig. 5. Fig. 4(a) is a diagram showing a hemispherical image (front side) captured by an imaging device, Fig. 4(b) is a diagram showing a hemispherical image (rear side) captured by an imaging device, and Fig. 4(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular image"). Fig. 5(a) is a conceptual diagram showing a state in which a sphere is covered by an equirectangular image, and Fig. 5(b) is a diagram showing a celestial sphere image.

[0028] As shown in Fig. 4(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by the fisheye lens 102a described below. Also, as shown in Fig. 4(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by the fisheye lens 102b described below. Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are combined by the imaging device 5a to create an equirectangular image as shown in Fig. 4(c).

[0029] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular image is pasted to cover the sphere as shown in FIG. 5(a), and a spherical image as shown in FIG. 5(b) is created. In this way, the spherical image is represented as an image in which the equirectangular image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the spherical image may be a still image or a video.

[0030] As described above, since a spherical image is an image pasted to cover a spherical surface, it gives a sense of incongruity to humans. Therefore, by displaying a predetermined region of the spherical image (hereinafter, referred to as a "predetermined region image") as a flat image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to humans. This will be described with reference to Figs. 6 and 7.

[0031] FIG. 6 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional 3D sphere. The virtual camera IC corresponds to the position of the viewpoint of a user who views the omnidirectional image CE displayed as a three-dimensional 3D sphere. FIG. 7(a) is a three-dimensional perspective view of FIG. 6, and FIG. 7(b) is a diagram showing a predetermined area image when displayed on a display. In FIG. 7(a), the omnidirectional image CE shown in FIG. 6 is represented by a three-dimensional 3D sphere CS. If the omnidirectional image CE generated in this way is a 3D sphere CS, the virtual camera IC is located outside the omnidirectional image CE as shown in FIG. The predetermined area T in the omnidirectional image CE is an imaging area of ​​the virtual camera IC, and is specified by predetermined area information indicating the imaging direction and the angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE.

[0032] The predetermined area T shown in Fig. 7(a) is displayed as an image of the imaging area of ​​the virtual camera IC on a predetermined display as shown in Fig. 7(b). The image shown in Fig. 7(b) is, for example, a predetermined area image represented by initially set predetermined area information. Note that the predetermined area information may be represented by the imaging area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the position coordinates of the virtual camera IC. In the following, the imaging direction (rH, rV) and angle of view (α) of the virtual camera IC will be used for explanation.

[0033] The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 8. FIG. 8 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 8, rH indicates Horizontal Radian, rV indicates Vertical Radian, and α indicates the angle of view (Angle). That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the imaging direction (rH, rV) becomes the center point CP of the predetermined area T, which is the imaging area of ​​the virtual camera IC. The predetermined area image Q is an image of the predetermined area T in the spherical image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). In FIG. 8, a trigonometric function generally represented by the following formula (A) holds.

[0034] L / f=tan(α / 2) (A) FIG. 9 is a diagram showing points in a three-dimensional Euclidean space in spherical coordinates. The position coordinates of the center point CP when expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius, polar angle, and azimuth angle, respectively. The radius r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to the center point CP, and is therefore equal to f. FIG. 9 is a diagram showing these relationships. Hereinafter, the center point will be explained using the position coordinates (r, θ, φ).

[0035] <Outline of the image communication system> Next, an outline of the configuration of the image communication system 10 of this embodiment will be described with reference to Fig. 10. Fig. 10 is a schematic diagram of the configuration of the image communication system 10 of this embodiment.

[0036] 10, in an image communication system 10 of this embodiment, communication terminals 1 to 4 located at points A to D communicate with each other via a communication network 100 such as the Internet, and the communication terminals 1 to 4 at each point can share images. An imaging device 5a and a communication terminal 1 are located at point A, a communication terminal 2 and an imaging device 5b are located at point B, an imaging device 5c, a communication terminal 3, and a display 6c are located at point C, and a communication terminal 4 and a display 6d are located at point D.

[0037] As described above, the imaging devices 5a and 5c are special digital cameras for capturing images of subjects, scenery, etc. to obtain two hemispherical images that are the basis of a celestial sphere image. On the other hand, the imaging device 5b is a general digital camera for capturing images of subjects, scenery, etc. to obtain a general planar image.

[0038] The communication terminals 3 and 4 are video conference terminals dedicated to video conferences, and display images of video calls (video calls) on the displays 6c and 6d, respectively, via wired cables such as USB (Universal Serial Bus) cables. The communication terminals 3 and 4 normally capture images of users, etc., using a camera 312 in FIG. 12, which will be described later. However, when the communication terminals 3 and 4 are connected to a cradle 7 to which an imaging device 5c is attached via a wired cable, the imaging device 5c is given priority, and a celestial sphere image can be obtained. When a wired cable is used, the cradle 7 not only performs communication between the imaging device 5c and the communication terminal 3, but also plays a role of supplying power to the imaging device 5c and supporting the imaging device 5c.

[0039] The communication terminals 1 and 2 are general-purpose information processing devices that communicate with other locations by running video conference application software. The communication terminal 1 is, for example, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a car navigation system, a game machine, a PDA (Personal Digital Assistant), a wearable PC, or a desktop PC. However, the communication terminal 1 is not limited to these.

[0040] The communication terminal 1 displays an image of the video call on a display 917 (described later) provided in the communication terminal 1. The communication terminal 1 usually captures an image using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 (described later) provided in the communication terminal 1, but can obtain a spherical image obtained by the imaging device 5a by using a wireless communication technology such as Wi-Fi (Wireless Fidelity) or Bluetooth (registered trademark).

[0041] The communication terminal 2 displays an image of the video call on a display 917 (described later) provided in the communication terminal 2. The communication terminal 2 captures an image of the user, etc., with an external imaging device 5b.

[0042] In addition, the communication terminals 1 to 4 have OpenGL ES installed, and can create predetermined area information indicating a partial area of ​​a celestial sphere image, and can create a predetermined area image from a celestial sphere image transmitted from another communication terminal. Therefore, each of the communication terminals 1 to 4 can display a predetermined area image cut out from a celestial sphere image.

[0043] At least one image capturing person 8 is present at site A and holds the image capturing device 5a by holding it in his / her hand or by attaching it to a rod-shaped member. The image capturing person 8 can move together with the image capturing device 5a. In addition, the communication terminal 1 displays on the display 917 images captured by the image capturing devices 5a to 5c and the camera 312 of the communication terminal 4.

[0044] One user 9b is present at site B, and communication terminal 2 displays images captured by imaging devices 5a-5c and camera 312 of communication terminal 4 on display 917. User 9b at site B is one of the users focusing on the point of interest specified by image capturer 8. However, the number of users is just an example.

[0045] Two users 9c1 and 9c2 exist at site C, and communication terminal 3 displays on display 6c images captured by imaging devices 5a to 5c and camera 312 of communication terminal 4. Users 9c1 and 9c2 at site C are among the users who focus on a point of interest specified by image capturer 8. However, the number of users is just an example.

[0046] One user 9d is present at site D, and communication terminal 4 displays on display 6d images captured by imaging devices 5a-5c and camera 312 of communication terminal 4. User 9d at site D is one of the users focusing on the point of interest specified by image capturer 8. However, the number of users is just an example.

[0047] The communication management system 50 manages and controls the communications of the communication terminals 1 to 4. Therefore, the communication management system 50 is also a communication control system. The communication management system 50 is installed in a service company that provides video communication services. The communication management system 50 may be constructed by a single computer, or may be constructed by multiple computers in which each unit (function, means, or storage unit) is divided and arbitrarily assigned.

[0048] 10 are merely examples, and in this embodiment, it is sufficient to have a location A and one other location. Also, location B does not need to have the imaging device 5b, and it is sufficient if it can display the spherical image transmitted from location A.

[0049] Also, the imaging device 5a and the communication terminal 1 at site A may be integrated. In other words, if the imaging device 5a has a function of connecting to the communication network 100, the communication terminal 1 is not necessary. In this case, the imaging device 5a becomes the communication terminal 1. This is because in this embodiment, the imager 8 can specify the point of interest using only the imaging device 5a. However, with the communication terminal 1, the imager 8 can specify the point of interest by operating the communication terminal 1. Also, the communication terminal 1 may receive the omnidirectional image and the coordinates of the point of interest via a storage medium without communicating with the imaging device 5a at site A.

[0050] Hereinafter, any one of the imaging devices 5a to 5c will be referred to as "imaging device 5." Furthermore, any one of the displays 6c and 6d will be referred to as "display 6."

[0051] <<Hardware configuration of the embodiment>> Next, the hardware configurations of the imaging devices 5a and 5c, the communication terminals 1 to 4, and the communication management system of this embodiment will be described in detail with reference to Figures 11 to 14. Note that the imaging device 5c is a general camera, so a detailed description thereof will be omitted.

[0052] <Hardware configuration of the imaging devices 5a and 5c> First, the hardware configuration of the imaging devices 5a and 5c will be described with reference to Fig. 11. Fig. 11 is a hardware configuration diagram of the imaging devices 5a and 5c. In the following, the imaging devices 5a and 5c are assumed to be omnidirectional (all-directional) imaging devices using two imaging elements, but the number of imaging elements may be two or more. In addition, the imaging devices do not necessarily need to be devices dedicated to omnidirectional imaging, and a normal digital camera, smartphone, or the like may be provided with an omnidirectional imaging unit as a retrofit to have substantially the same functions as the imaging devices 5a and 5c.

[0053] As shown in FIG. 11, the imaging devices 5a and 5c are composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I / F 116, a communication unit 117, and an antenna 117a.

[0054] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to the wide-angle lenses. The imaging elements 103a and 103b include an image sensor such as a CMOS sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the fisheye lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the imaging elements are set.

[0055] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are connected to a serial I / F bus (such as an I2C bus) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to a CPU 111 via a bus 110. Furthermore, ROM 112, SRAM 113, DRAM 114, an operation unit 115, a network I / F 116, a communication unit 117, an acceleration / direction sensor 118, a gyro sensor 119, and the like are also connected to the bus 110.

[0056] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs a predetermined processing on each image data, and then synthesizes the image data to create equirectangular image data as shown in Figure 4(c).

[0057] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0058] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 5a and 5c have a preview display function or a function corresponding to video display on a display (for example, a display of the communication terminals 1 and 3). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / minute).

[0059] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the imaging devices 5a and 5c are not provided with a display unit, but may be provided with a display unit.

[0060] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via the I / F bus, and performs a predetermined process on the sound data.

[0061] The CPU 111 controls the overall operation of the imaging devices 5a and 5c and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of a processed equirectangular image.

[0062] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, etc. A user operates the operation buttons to input various imaging modes, imaging conditions, and the like.

[0063] The network I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. The data of the equirectangular image stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or transmitted to an external device such as the communication terminal 3 via the network I / F 116 as necessary.

[0064] The communication unit 117 communicates with external devices such as the communication terminals 1 and 3 by short-range wireless technology such as Wi-Fi or NFC (Near Field Communication) via the antenna 117a provided in the imaging devices 5a and 5c. The communication unit 117 can also transmit data of the equirectangular image to external devices such as the communication terminals 1 and 3.

[0065] The acceleration and orientation sensor 118 calculates the orientation and inclination (roll angle, yaw angle, pitch angle) of the imaging devices 5a, 5c from the earth's magnetism, and outputs the orientation and inclination information. This orientation and inclination information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data such as the capture date and time of the image, and the data capacity of the image data.

[0066] The gyro sensor 119 is a three-axis or six-axis sensor that detects the rotation speed around each of the X, Y, and Z axes. The rotation angle (attitude information) is obtained by accumulating (integrating) the rotation speed.

[0067] <Hardware configuration of communication terminals 3 and 4 (video conference terminals)> Next, the hardware configuration of the communication terminals 3 and 4 will be described with reference to Fig. 12. Fig. 12 is a hardware configuration diagram of the communication terminals 3 and 4. As shown in Fig. 12, the communication terminals 3 and 4 include a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, an operation button 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an image sensor I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319.

[0068] Of these, the CPU 301 controls the overall operation of the communication terminals 3 and 4. The ROM 302 stores programs used to drive the CPU 301, such as an IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and sound data. The SSD (Solid State Drive) 305 controls the reading or writing of various data from the flash memory 304 under the control of the CPU 301. Note that an HDD may be used instead of the SSD. The media I / F 307 controls the reading or writing (storing) of data from a recording medium 306, such as a flash memory. The operation button 308 is a button that is operated when selecting a destination for the communication terminals 3 and 4, etc. The power switch 309 is a switch for switching the power of the communication terminals 3 and 4 ON / OFF.

[0069] The network I / F 311 is an interface for data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means for capturing an image of a subject under the control of the CPU 301 to obtain image data. The image sensor I / F 313 is a circuit for controlling the driving of the camera 312. The microphone 314 is a type of built-in sound collection means for inputting sound. The sound input / output I / F 316 is a circuit for processing input / output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I / F 317 is a circuit for transmitting image data to the external display 6 under the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-distance communication circuit 319 is a communication circuit such as NFC (registered trademark) or Bluetooth (registered trademark).

[0070] 12. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 301 shown in FIG.

[0071] The display 6 is a type of display means configured with liquid crystal or organic EL (electro luminescence) for displaying an image of a subject, operation icons, etc. The display 6 is connected to the display I / F 317 by a cable 6y. This cable 6y may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0072] The camera 312 includes a lens and a solid-state imaging element that converts light into electric charges to digitize an image (video) of a subject, and a CMOS sensor, CCD sensor, or the like is used as the solid-state imaging element. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 via a USB (Universal Serial Bus) cable, or the like. When an external camera is connected, the external camera is driven in preference to the built-in camera 312 under the control of the CPU 301. Similarly, when an external microphone or an external speaker is connected, the external microphone and the external speaker are driven in preference to the built-in microphone 314 and the built-in speaker 315, respectively, under the control of the CPU 301.

[0073] Moreover, the recording medium 306 is configured to be detachable from the communication terminals 3 and 4. Furthermore, as long as it is a non-volatile memory that reads or writes data under the control of the CPU 301, it is not limited to the flash memory 304, and an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

[0074] <Hardware configuration of the communication management system> Next, the hardware configuration of the communication management system 50 will be described with reference to Fig. 13. Fig. 13 is a diagram showing the hardware configuration of the communication management system.

[0075] The communication management system 50 includes a CPU 501 for controlling the overall operation of the communication management system 50, a ROM 502 storing a program used to drive the CPU 501, such as an IPL, a RAM 503 used as a work area for the CPU 501, a HD 504 for storing various data such as programs for the communication management system 50, a HDD (Hard Disk Drive) 505 for controlling the reading or writing of various data from the HD 504 under the control of the CPU 501, a media drive 507 for controlling the reading or writing (storage) of data from a recording medium 506, such as a flash memory, a display 508 for displaying various information such as a cursor, a menu, a window, characters, or an image, a network I / F 509 for data communication using the communication network 100, a keyboard 511 having multiple keys for inputting characters, numbers, various instructions, etc., a mouse 512 for selecting and executing various instructions, selecting a processing target, moving the cursor, etc., a CD-RW (Compact The device is equipped with a CD-RW drive 514 that controls the reading of various data from a CD-ReWritable disc 513, and bus lines 510 such as an address bus and a data bus for electrically connecting the above components as shown in FIG. 13.

[0076] <Hardware configuration of communication terminals 1 and 2> Next, the hardware of the communication terminals 1 and 2 will be described with reference to Fig. 14. Fig. 14 is a hardware configuration diagram of the communication terminals 1 and 2. As shown in Fig. 14, the communication terminals 1 and 2 include a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration / direction sensor 906, a media I / F 908, and a GPS receiver 909.

[0077] Of these, the CPU 901 controls the overall operation of the communication terminals 1 and 2. The ROM 902 stores programs such as an IPL used to drive the CPU 901. The RAM 903 is used as a work area for the CPU 901. The EEPROM 904 reads or writes various data such as programs for the communication terminals 1 and 2 under the control of the CPU 901. The CMOS sensor 905 captures an image of a subject (mainly a self-portrait) under the control of the CPU 901 to obtain image data. The acceleration / direction sensor 906 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I / F 908 controls the reading or writing (storing) of data from or to a recording medium 907 such as a flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0078] In addition, the communication terminals 1 and 2 are equipped with a long-distance communication circuit 911, a camera 912, an image sensor I / F 913, a microphone 914, a speaker 915, an audio input / output I / F 916, a display 917, an external device connection I / F 918, a short-distance communication circuit 919, an antenna 919a of the short-distance communication circuit 919, and a touch panel 921.

[0079] Among these, the long-distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 901. The image sensor I / F 913 is a circuit that controls the driving of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 916 is a circuit that processes input and output of sound signals between the microphone 914 and the speaker 915 under the control of the CPU 901. The display 917 is a type of display means such as a liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 918 is an interface for connecting various external devices. The short-distance communication circuit 919 is a communication circuit such as NFC or Bluetooth (registered trademark). The touch panel 921 is a type of input means that allows a user to operate the communication terminals 1 and 2 by pressing the display 917.

[0080] Furthermore, the communication terminals 1 and 2 include a bus line 910. The bus line 910 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 901.

[0081] Incidentally, the recording media such as CD-ROMs on which the above-mentioned programs are stored, and the HD on which these programs are stored, can be provided domestically or internationally as program products.

[0082] <<Functional configuration of the embodiment>> Next, the functional configuration of this embodiment will be described with reference to Fig. 15 to Fig. 22. Fig. 15 and Fig. 16 are functional block diagrams showing the functions of the image communication system 10 in block form.

[0083] <Functional configuration of the imaging device 5a> As shown in Fig. 15, the imaging device 5a has a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a posture information acquisition unit 15a, an image processing unit 16a, a communication unit 18a, and a storage / readout unit 19a. Each of these units, except for the image processing unit 16a, is a function or means realized by any of the components shown in Fig. 11 operating in response to an instruction from a CPU 111 in accordance with a program for the imaging device loaded from an SRAM 113 onto a DRAM 114. The image processing unit 16a is realized by a circuit module such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), or an FPGA (field programmable gate array), but may be realized by software.

[0084] The imaging device 5a also has a storage unit 1000a that is configured by the ROM 112, the SRAM 113, and the DRAM 114 shown in Fig. 11. The storage unit 1000a stores a GUID (Globally Unique Identifier) ​​of the imaging device 5a.

[0085] (Functional configuration of the imaging device 5a) The reception unit 12a of the imaging device 5a is realized mainly by the processing of the operation unit 115 and the CPU 111 shown in FIG.

[0086] The imaging section 13a is mainly realized by the imaging unit 101, image processing unit 104, and imaging control unit 105 shown in FIG. 11, and processing by the CPU 111, and captures images of scenery and the like, and obtains captured image data (panoramic image).

[0087] The sound collection section 14a is realized by the components 108 and sound processing unit 109 shown in FIG. 11, and processing by the CPU 111, and collects sounds around the imaging device 5a.

[0088] The orientation information acquisition unit 15a acquires orientation information related to the orientation (tilt in three-dimensional space) of the imaging device. The image processing unit 16a performs a process of synthesizing two hemispherical images into one equirectangular image. During this process, a process of performing zenith correction based on the orientation information is performed, and the coordinates of the attention point indicated by the longitudinal direction (upward) can be converted to coordinates of a spherical coordinate system (equirectangular coordinates) that indicate the direction the imager is pointing.

[0089] The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 98 of the communication terminal 1 by short-range wireless communication technology such as the NFC standard, Bluetooth (registered trademark), or Wi-Fi.

[0090] The storage / reading unit 19a is realized mainly by the processing of the CPU 111 shown in FIG. 11, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.

[0091] The imaging device 5c has a reception unit 12c, an imaging unit 13c, a sound collection unit 14c, a posture information acquisition unit 15c, an image processing unit 16c, a communication unit 18c, a memory / readout unit 19c, and a memory unit 1000c; however, because these respectively realize the same functions as the reception unit 12a, the imaging unit 13a, the sound collection unit 14a, the posture information acquisition unit 15a, the image processing unit 16a, the communication unit 18a, the memory / readout unit 19a, and the memory unit 1000a in the imaging device 5a, description of these will be omitted.

[0092] <Functional configuration of communication terminal 1> 15, the communication terminal 1 has a transmission / reception unit 91, a reception unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a calculation unit 97, a communication unit 98, and a storage / readout unit 99. Each of these units is a function or means realized by any of the components shown in FIG. 14 operating in response to an instruction from the CPU 901 in accordance with the program for the communication terminal 1 loaded from the EEPROM 904 onto the RAM 903.

[0093] The communication terminal 1 also has a storage unit 9000 constructed by a ROM 902, a RAM 903, and an EEPROM 904 shown in Fig. 14. In the storage unit 9000, an image type management DB 9001, an imaging device management DB 9002, and a predetermined area management DB 9003 are constructed.

[0094] Of these, the image type management DB 9001 is configured with an image type management table shown in Fig. 17. The imaging device management DB 9002 is configured with an imaging device management table shown in Fig. 18. The predetermined area management DB 9003 is configured with a predetermined area information management table shown in Fig. 19.

[0095] (Image type management table) FIG. 17 is a conceptual diagram showing an image type management table. In this image type management table, image data ID, an IP address which is an example of the destination of the transmission source terminal, and a source name are stored and managed in association with each other. Among these, the image data ID is an example of image data identification information for identifying image data when performing video communication. The same image data ID is added to image data transmitted from the same transmission source terminal. This allows the transmission destination terminal (receiving communication terminal) to identify the transmission source terminal of the received image data. The IP address of the transmission source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for identifying an imaging device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by the communication terminal 3 or the like according to a predetermined naming rule.

[0096] For example, it is shown that four communication terminals with IP addresses "1.2.1.3", "1.2.2.3", "1.3.1.3", and "1.3.2.3" are transmitting image data indicated by image data IDs "RS001", "RS002", "RS003", and "RS004", respectively. Furthermore, the types of images indicated by the source names of the communication terminals are "Video_Wide", "Video_Wide", "Video", and "Video", which indicate that the image types are "special image", "special image", "general image", and "general image", respectively. Note that the special image here is a spherical image.

[0097] In addition, data other than image data may also be managed in association with an image data ID. Data other than image data may be, for example, sound data, or material data for screen sharing. In addition, data other than image data may also be managed in association with an image data ID. Data other than image data may be, for example, sound data, or material data for screen sharing.

[0098] (Imaging device management table) 18 is a conceptual diagram showing an imaging device management table. In this imaging device management table, the vendor ID and product ID of the GUID of an imaging device capable of obtaining two hemispherical images that are the basis of a celestial sphere image are stored and managed. As the GUID, for example, a vendor ID (VID) and product ID (PID) used in a USB device can be used. The vendor ID and product ID are stored in the communication terminals 3 and 4 at the time of shipment from the factory, but may be additionally stored after shipment from the factory.

[0099] (Prescribed area information management table) FIG. 19 is a conceptual diagram showing a predetermined area information management table. In this predetermined area information management table, the IP address of the communication terminal that is the transmission source of the captured image data, the IP address of the communication terminal that is the transmission destination of the captured image data, and predetermined area information indicating the predetermined area image being displayed on the communication terminal that is the transmission destination of the captured image data are associated and stored and managed. The communication terminal that is the transmission destination of the captured image data is also the communication terminal that is the transmission source of the predetermined area information. As shown in FIG. 6 and FIG. 7, the predetermined area information is a conversion table for converting the captured image into an image of the predetermined area T in the captured image (predetermined area image). The IP address is an example of destination information, and the destination information includes a MAC (Media Access Control) address, a terminal ID (Identification) for identifying the communication terminal, and the like. Here, the IP address is expressed by simplifying the IPv4 address. The IP address may be IPv6.

[0100] For example, in the first to third rows of the predetermined area information management table in FIG. 19, when the IP address of the communication terminal 3 is "1.2.1.3", it is managed that the captured image data transmitted from the communication terminal 3 is transmitted to the communication terminal 4 with the IP address "1.2.2.3", the communication terminal 2 with the IP address "1.3.1.3", and the communication terminal 1 with the IP address "1.3.2.3" via the communication management system 50. Furthermore, it is managed that the communication terminal 4 is the communication terminal that is the source of the predetermined area information (r=10, θ=20, φ=30). Similarly, it is managed that the communication terminal 2 is the communication terminal that is the source of the predetermined area information (r=20, θ=30, φ=40). Moreover, it is managed that the communication terminal 1 is the communication terminal that is the source of the predetermined area information (r=30, θ=40, φ=50). In other words, the predetermined area information with which the communication terminals 1, 2, and 4 are viewing the spherical image is managed.

[0101] In addition, when the transmission / reception unit 31c receives new specified area information that includes the same set of IP addresses as the IP address of the communication terminal that sent the captured image data already managed and the IP address of the communication terminal that sent the captured image data, the storage / reading unit 39c overwrites the specified area information already managed with the newly received specified area information.

[0102] (Functional configuration of communication terminal 1) The transmission / reception unit 91 of the communication terminal 1 is mainly realized by the processing of the long-distance communication circuit 911 and the CPU 901 shown in FIG. 14, and transmits and receives various data (or information) to and from the communication management system 50 via the communication network 100.

[0103] The reception unit 92 is realized mainly by processing by the touch panel 921 and the CPU 901, and receives various selections or inputs from the user. In addition to the touch panel 921, voice input or the like may be used as another input means.

[0104] 14, and performs image processing on image data obtained by capturing an image of a subject with a camera 912. In addition, after the user's voice is converted into an audio signal by a microphone 914, the image / sound processing unit 93 performs audio processing on audio data related to this audio signal.

[0105] Furthermore, the image / sound processing unit 93 performs image processing on image data received from another communication terminal based on image type information such as a source name, so that the display control unit 94 displays an image on the display 917. The image / sound processing unit 93 also outputs an audio signal related to audio data received from another communication terminal via the communication management system 50 to the speaker 915, and causes the speaker 915 to output audio.

[0106] The display control unit 94 is realized mainly by the processing of the display 917 and the CPU 901, and performs control for displaying various images, characters, and the like on the display 917.

[0107] The determination unit 95 is realized mainly by the processing of the CPU 901, and determines, for example, the image type related to the image data received from the imaging device 5a.

[0108] The creation unit 96 is mainly realized by the processing of the CPU 901, and creates a source name, which is an example of image type information, in accordance with the naming rules described above based on the result of the determination by the determination unit 95 of whether the image is a general image or a special image (here, a spherical image). For example, if the determination unit 95 determines that the image is a general image, the creation unit 96 creates a source name "Video" indicating that it is a general image. On the other hand, if the determination unit 95 determines that the image is a special image, the creation unit 96 creates a source name "Video_Wide" indicating that it is a special image.

[0109] The calculation unit 97 is mainly realized by the processing of the CPU 901, and calculates the position (position information) of the predetermined area T1 in the captured image based on the predetermined area information indicating the predetermined area T and the predetermined area information received from another communication terminal by the transmission / reception unit 91. In addition, an image in which the entire captured image is displayed is also called an "entire image."

[0110] The communication unit 98 is mainly realized by the processing of the short-range communication circuit 919, the antenna 919a, and the CPU 901, and can communicate with the communication unit 18a of the imaging device 5a by short-range wireless technology such as NFC, Bluetooth (registered trademark), Wi-Fi, etc. Note that although the communication unit 98 and the transmission / reception unit 91 have been described as having separate communication units, they may be configured to share a common communication unit.

[0111] The storage / reading unit 99 is mainly realized by the processing of the CPU 901 shown in FIG. 14, and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000.

[0112] <Functional configuration of communication terminal 2> Next, the functional configuration of the communication terminal 2 will be described in detail. The communication terminal 2 basically has the same functions as the communication terminal 1. That is, as shown in Fig. 16, the communication terminal 2 has a transmission / reception unit 71, a reception unit 72, an image / sound processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a calculation unit 77, a communication unit 78, and a storage / readout unit 79. However, since each function realizes the same function as the transmission / reception unit 91, the reception unit 92, the image / sound processing unit 93, the display control unit 94, the determination unit 95, the creation unit 96, the calculation unit 97, the communication unit 98, and the storage / readout unit 99 in the communication terminal 1, respectively, the description of these will be omitted.

[0113] 14. An image type management DB 7001, an imaging device management DB 7002, and a predetermined area management DB 7003 are constructed in this storage unit 7000. Note that the image type management DB 7001, the imaging device management DB 7002, and the predetermined area management DB 7003 have the same data structures as the image type management DB 9001, the imaging device management DB 9002, and the predetermined area management DB 9003 in the communication terminal 1, respectively, and therefore description thereof will be omitted.

[0114] <Functional configuration of communication terminal 3> Next, the functional configuration of the communication terminal 3 will be described. The communication terminal 3 basically has the same functions as the communication terminal 1. That is, as shown in Fig. 15, the communication terminal 3 has a transmission / reception unit 31c, a reception unit 32c, an image / sound processing unit 33c, a display control unit 34c, a determination unit 35c, a creation unit 36c, a calculation unit 37c, a communication unit 38c, and a storage / readout unit 39c. However, since these realize the same functions as the transmission / reception unit 91, the reception unit 92, the image / sound processing unit 93, the display control unit 94, the determination unit 95, the creation unit 96, the calculation unit 97, the communication unit 98, the storage / readout unit 99, and the storage unit 9000 in the communication terminal 1, the description of these will be omitted.

[0115] The communication terminal 3 also has a storage unit 3000c constructed by the ROM 302, RAM 303, and SSD 305 shown in Fig. 12. An image type management DB 3001c, an imaging device management DB 3002c, and a predetermined area management DB 3003c are constructed in the storage unit 3000c, but because these realize the same functions as the image type management DB 9001, the imaging device management DB 9002, and the predetermined area management DB 9003 in the communication terminal 1, their description will be omitted.

[0116] <Functional configuration of communication terminal 4> Next, the functional configuration of the communication terminal 4 will be described. The communication terminal 4 basically has the same functions as the communication terminal 1. That is, as shown in Fig. 16, the communication terminal 4 has a transmission / reception unit 31d, a reception unit 32d, an image / sound processing unit 33d, a display control unit 34d, a determination unit 35d, a creation unit 36d, a calculation unit 37d, a communication unit 38d, a storage / readout unit 39d, and a storage unit 3000d. However, since these realize the same functions as the transmission / reception unit 91, the reception unit 92, the image / sound processing unit 93, the display control unit 94, the determination unit 95, the creation unit 96, the calculation unit 97, the communication unit 98, the storage / readout unit 99, and the storage unit 9000 in the communication terminal 1, the description of these will be omitted.

[0117] The communication terminal 4 also has a storage unit 3000d constructed by the ROM 302, the RAM 303, and the SSD 305 shown in Fig. 12. An image type management DB 3001d, an imaging device management DB 3002d, and a predetermined area management DB 3003d are constructed in the storage unit 3000d, but because these realize the same functions as the image type management DB 9001, the imaging device management DB 9002, and the predetermined area management DB 9003 in the communication terminal 1, a description of these will be omitted.

[0118] <Functional configuration of the communication management system> Next, a detailed description will be given of each functional component of the communication management system 50. The communication management system 50 has a transmitting / receiving unit 51, a determining unit 55, a generating unit 56, and a storing / reading unit 59. Each of these units is a function or means realized by any of the components shown in Fig. 13 operating in response to an instruction from the CPU 501 in accordance with the program for the communication management system 50 loaded from the HD 504 onto the RAM 503.

[0119] The communication management system 50 also has a storage unit 5000 constructed by the RAM 503 and HD 504 shown in Fig. 13. A session management DB 5001, an image type management DB 5002, and a predetermined area management DB 5003 are constructed in this storage unit 5000. Of these, the session management DB 5001 is constructed by the session management table shown in Fig. 20. The image type management DB 5002 is constructed by the image type management table shown in Fig. 21. The predetermined area management DB 5003 is constructed by the predetermined area information management table shown in Fig. 22.

[0120] (Session management table) FIG. 20 is a conceptual diagram showing a session management table. In this session management table, session IDs and IP addresses of participating communication terminals are stored and managed in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session that realizes a video call, and is generated for each virtual conference room. The session ID is also managed by communication terminals 1 to 4, and is used when selecting a communication session in each communication terminal. The IP addresses of participating communication terminals indicate the IP addresses of communication terminals that have participated in the virtual conference room identified by the associated session ID.

[0121] (Image type management table) FIG. 21 is a conceptual diagram showing an image type management table. In the image type management table shown in FIG. 21, in addition to each piece of information managed in the image type management table shown in FIG. 17, a session ID that is the same as the session ID managed in the session management table is associated and managed. Here, it is shown that three communication terminals with IP addresses of "1.2.1.3", "1.2.2.3", and "1.3.1.3" are participating in the virtual conference room indicated by the same session ID "se101". In addition, in the communication management system 50, the image data ID, the IP address of the sender terminal, and the image type information managed by the communication terminals such as the communication terminals 1 to 4 (video conference terminals) are managed as the same ones in order to transmit image type information, etc. to the communication terminals already in a video call and the newly participating communication terminal when a new communication terminal enters the virtual conference room. This eliminates the need to transmit and receive image type information, etc. between the communication terminal already in a video call and the newly participating communication terminal.

[0122] (Prescribed area information management table) Fig. 22 is a conceptual diagram showing a predetermined area information management table. This predetermined area information management table basically has the same data structure as the predetermined area information management table shown in Fig. 19. However, as described later, the transmitting / receiving unit 51 transmits the latest predetermined area information to each communication terminal at regular intervals (e.g., 30 seconds), so that all the predetermined area information received by the transmitting / receiving unit 51 is stored without being deleted until the predetermined area information is transmitted at regular intervals. In Fig. 22, the newer the predetermined area information, the higher the management position.

[0123] (Functional configuration of communication management system) The transmission / reception unit 51 of the communication management system 50 is mainly realized by the processing of the network I / F 509 and CPU 501 shown in FIG.

[0124] The determination unit 55 is realized mainly by the processing of the CPU 501, and performs various determinations.

[0125] The generating unit 56 is realized mainly by the processing of the CPU 501, and generates image data ID.

[0126] The storage / reading unit 59 is mainly realized by the processing of the HDD 505 and CPU 501 shown in FIG. 13, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000.

[0127] <Participation process> First, a process of joining a specific communication session will be described with reference to Fig. 23 and Fig. 24. Fig. 23 is a sequence diagram showing a process of joining a specific communication session. Fig. 24 is a diagram showing a selection screen for a communication session (virtual conference room).

[0128] First, when the image capturing person 8 at the site A performs an operation to display a selection screen of a communication session (virtual conference room) on the communication terminal 1, the reception unit 92 receives the operation to display the selection screen, and the display control unit 94 displays the selection screen as shown in Fig. 24 on the display 917 of the communication terminal 1 (step S21). On this selection screen, selection buttons b1, b2, b3, etc., which indicate the virtual conference rooms R1, R2, R3, etc. to be selected, are displayed. Also, each selection button b1, etc. is associated with a session ID.

[0129] When the image capturing person 8 selects a desired selection button (here, selection button b1) of a virtual conference room, the reception unit 92 receives the selection of the communication session (step S22). The selected conference room is, for example, a conference room that has been determined in advance for viewing real estate.

[0130] Then, the transmitting / receiving unit 91 transmits a request to join the virtual conference room to the communication management system 50 (step S23). This request includes a session ID indicating the communication session selected and accepted in step S22, and the IP address of the communication terminal 1, which is the request source terminal. As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the request to join.

[0131] Next, the memory / read unit 59 of the communication management system 50 performs a process of joining the communication session by adding the IP address received in step S23 to the participating terminal IP address field in the record in the session management DB 5001 that has the same session ID as the session ID received in step S23 (step S24).

[0132] Then, the transmitting / receiving unit 51 transmits a participation request response to the communication terminal 1 (step S25). This participation request response includes the session ID received in step S23 and the participation processing result. As a result, the transmitting / receiving unit 91 of the communication terminal 1 receives the participation request response. Hereinafter, a case where the participation processing is successful will be described.

[0133] Note that similar participation requests are also made by the communication terminals 2 to 4 at the locations B to D, and by each selecting the same conference room, the communication terminals 1 to 4 participate in the same session and become able to make video calls to each other. The method of participating in a session shown in Figs. 23 and 24 is merely an example, and the image capturer 8 may establish a session by specifying the identification information of the destination communication terminal or user. In this case, the destination communication terminal or user specified as the destination responds to the call to establish the session.

[0134] <About posture information> Next, the attitude information of the imaging device 5a will be described with reference to Fig. 25 to Fig. 27. Fig. 25 is an example of a diagram for explaining the coordinate axes of the imaging device 5a, and Fig. 26 is an example of a diagram for explaining the attitude of the imaging device 5a that serves as a reference. The attitude of the imaging device 5a that serves as a reference is the initial state of the attitude of the imaging device.

[0135] As shown in Fig. 25, the longitudinal direction of the imaging device 5a is the Z axis, the direction passing through the two lenses from the side where the shutter button SB is located to the side where it is not is the Y axis, and the width direction of the imaging device 5a is the X axis. These axes move together with the imaging device 5a. The imaging device 5a can rotate around each of the X axis, Y axis, and Z axis. The angle of rotation around the X axis is α, the angle of rotation around the Y axis is β, and the angle of rotation around the Z axis is γ.

[0136] 26, when the person photographing 8 points at the target 180, the longitudinal direction (upward) faces the target 180. If the initial state is when the camera is upright, the state in FIG. 26 is α=-90 degrees, β=0 degrees, and γ=0 degrees.

[0137] FIG. 27 is a diagram for explaining values ​​detected by the acceleration / direction sensor 118. If the values ​​detected by the acceleration / direction sensor 118 are (ax, ay, az), the main body rotates by α around the X-axis as shown in FIG. 27(a), and the angle is α = atan 2 (ay,-az) ay^2+az^2 ≧ threshR^2 = 0 ay^2+az^2 < threshR^2 where α takes the value -π<α≦π.

[0138] Rotation about the Y axis can be expressed as shown in FIG. 27(b). Rotation about the Z axis can be expressed as shown in FIG. 27(c). However, to improve responsiveness, it is advisable to use a gyro sensor 119. The values ​​of the gyro sensor 119 (gα, gβ, gγ) output are values ​​equivalent to angular velocity [rad / sec]. The rotation angles (attitude information) α, β, and γ of the imaging device 5a are calculated using the values ​​of the gyro sensor 119, α(n+1)=α(n)+k*gα*dt β(n+1)=β(n)+k*gβ*dt γ(n+1)=γ(n)+k*gγ*dt It can be calculated as:

[0139] However, the initial orientation is (α(0), β(0), γ(0)) = (α0, 0, 0). k is a coefficient related to the sensitivity of the gyro sensor 119. The initial value is k = 1.0. To avoid the effects of camera shake, etc., k can be set to 0.5, for example, to provide a low-pass filter effect. The (α, β, γ) obtained in this way is the attitude information. Note that γ, the rotation angle around the Z axis, does not affect the pointing direction, so γ does not need to be calculated.

[0140] The rotation angle α around the X-axis can obtain a correct value even if the initial value is not zero, due to the signal from the acceleration / orientation sensor 118. Furthermore, the rotation angle γ around the Z-axis does not affect the pointing direction. In contrast, the rotation angle β around the Y-axis is simply initialized to 0, so if the initial state when the power is turned on is tilted, for example, the correct rotation angle β cannot be obtained. Therefore, when the imager 8 points to a target, it is preferable to turn on the image capture device 5a and initialize the rotation angles α, β, and γ by, for example, pressing and holding a specific button.

[0141] Alternatively, the initial value of the rotation angle β around the Y axis may be calculated from the output of the acceleration / azimuth sensor 118, in the same manner as the rotation angle α around the X axis.

[0142] <How are the coordinates of the point of interest indicated by the photographer processed in the zenith correction?> Using Fig. 28, an image of how the coordinates of the attention point pointed by the imager 8 are processed in the zenith correction will be described. Fig. 28(a) shows the imager 8 pointing at an object. The imager 8 is pointing at a doll 190 with the imaging device 5a.

[0143] For example, if a pixel or a certain range in which a subject in the longitudinal direction (upper side) of the imaging device 5a is captured is taken as the attention point coordinate, the image captured in the state of FIG. 28(a) will be as shown in FIG. 28(b). FIG. 28(b) is an equirectangular image in which a celestial sphere image is expressed by equirectangular projection. This equirectangular image is "not subjected to zenith correction." When creating an equirectangular image, the subject in the longitudinal direction (upper side) is expanded horizontally to the upper end and converted. In other words, if zenith correction is not performed, the subject in the longitudinal direction (upper side) is placed at the upper end, and since it is an equirectangular image, the subject at the upper end spreads out horizontally. If zenith correction is not performed, the attention point coordinate will always be in the same place in the equirectangular image.

[0144] The image capturing device 5a performs zenith correction on the image based on the orientation information in order to display what is at the top of the real space at the top of the image and what is at the bottom at the bottom of the image, regardless of the orientation of the image capturing device 5a. FIG. 28(c) shows a "zenith-corrected" equirectangular image. Since the direction and amount of zenith correction are determined by the orientation information, the original attention point coordinates defined in the coordinate conversion information for rotating the image for zenith correction are rotated to become the attention point coordinates in the equirectangular image. In FIG. 28(c), the attention point coordinates are shown surrounded by a circle 191, and the doll 190 in FIG. 28(a) is shown inside the circle 191. In other words, the object pointed to by the photographer 8 can be identified by the attention point coordinates.

[0145] <Functions of image processing unit 16a> 29 shows main functional blocks of the image processing unit 16a included in the imaging device 5a. The image processing unit 16a includes a captured image acquisition unit 202, a stitching processing unit 204, a zenith correction unit 206, a spherical image generation unit 208, an image compression unit 210, a focus point definition unit 194, a focus point conversion unit 196, and a focus point designation presence / absence determination unit 198.

[0146] The captured image acquisition unit 202 controls the two image sensors 130A and 130B described above and acquires captured images from each of them. In the case of a still image, two captured images for one frame are acquired at the timing when the shutter is pressed. In the case of a video, consecutive frames are captured sequentially, and two captured images are acquired for each frame. The images captured by each of the image sensors 130 are fisheye images that generally cover a hemisphere of the celestial sphere in their field of view, and constitute partial images of the celestial sphere image. Hereinafter, the images captured by each of the image sensors 130 may be referred to as partial images.

[0147] The stitching processor 204 detects a stitching position between two acquired partial images and executes a process for stitching the two partial images. In the stitching position detection process, a process is performed to detect the amount of positional shift of each of a plurality of corresponding points in an overlapping area that exists between a plurality of partial images for each frame.

[0148] The attention point definition unit 194 defines and holds the attention point coordinates (x, y) in the planar image. The attention point coordinates are not set by a user (such as the photographer 8), but are defined (fixed) in advance at the time of manufacture, design, shipping, etc. Note that there may be multiple attention point coordinates. In this case, the photographer 8 operates the imaging device 5a or the communication terminal 1 to select the attention point coordinates he or she wants to use.

[0149] The point of interest designation / non-designation determination unit 198 determines whether the photographer 8 is attempting to designate a point of interest. For example, while a button on the body of the imaging device 5a is being pressed, the point of interest designation / non-designation determination unit 198 determines that the photographer 8 is designating a point of interest. While the button is not being pressed, it determines that a point of interest is not being designated.

[0150] The zenith correction unit 206 executes a correction process for making the zenith direction of the generated omnidirectional image coincide with a predetermined reference direction based on the attitude information acquired by the attitude information acquisition unit 15a. Specifically, the correction unit 206 corrects a conversion table, which will be described later. Here, the predetermined reference direction is typically the vertical direction, which is the direction in which gravitational acceleration acts. By correcting the zenith direction of the omnidirectional image to coincide with the vertical direction (heavenly direction), it becomes possible to prevent the user from feeling uncomfortable, such as three-dimensional motion sickness, even when the field of view is changed during viewing, particularly in moving images. The conversion table will be described with reference to FIG. 30.

[0151] Based on the attention point definition unit 194, the attention point designation presence / absence determination unit 198, and the corrected conversion table, the attention point conversion unit 196 converts the attention point coordinates (x, y) on the planar coordinates into the attention point coordinates (θ, φ) on the spherical coordinates. In other words, it determines the attention point coordinates after zenith correction based on the attitude information. The attention point conversion unit 196 sets the attention point coordinates (θ, φ) on the spherical coordinates in the attention point information.

[0152] [Table 1] Table 1 shows an example of attention point information. The attention point information has the attention point coordinates (θ, φ) in a spherical coordinate system and whether or not the attention point is designated. If the attention point is designated, the attention point designation determination unit 198 sets the attention point designation presence / absence to "Yes"; if not, it sets the attention point designation presence / absence to "No." Note that the attention point itself is always included in the attention point information.

[0153] The omnidirectional image generating unit 208 executes a process of generating a omnidirectional image from the captured two partial images in a state where the processing result of the attention point conversion unit 196 is reflected. In the embodiment to be described, the conversion table is also used to generate the omnidirectional image from the two partial images. The omnidirectional image generating unit 208 generates the omnidirectional image from the two partial images using the corrected conversion table. By performing the process in this manner, it is possible to reduce the processing load for obtaining the final omnidirectional image.

[0154] However, the present invention is not limited to the above-described embodiment, and may be configured to generate a zenith-corrected celestial sphere image by joining two partial images together and performing zenith correction processing on the generated celestial sphere image.

[0155] The image compression unit 210 includes a still image compression block, and when capturing a still image, the image compression unit 210 compresses the captured image into image data in a predetermined still image format such as JPEG (Joint Photographic Experts Group). When capturing a moving image, the image compression unit 210 compresses captured continuous image frames into image data in a predetermined moving image format. The moving image compression format is not particularly limited, and may include various moving image compression formats such as H.264 / MPEG-4 AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), Motion JPEG, and Motion JPEG2000. The generated image data is transmitted to other locations B to D by the transmission unit 211. The transmission unit 211 corresponds to the communication unit 18a of the imaging device 5a and the transmission / reception unit 91 of the communication terminal 1.

[0156] <Functions of the display control unit of each communication device> When the communication terminals 1 to 4 receive the spherical image transmitted from the other locations, they display the spherical image.

[0157] 29, the display control unit 74 of the communication terminal 2 has an image development unit 212, a focus point determination unit 214, an image rotation unit 216, a cropping processing unit 220, an enlargement / letterbox processing unit 222, and an output unit 224. Although the function of the display control unit 74 will be described in Fig. 29, the display control unit 34c of the communication terminal 3, the display control unit 34d of the communication terminal 4, and the display control unit 94 of the communication terminal 1 also have the same functions.

[0158] The image development unit 212 reads out the omnidirectional image transmitted from the imaging device 5a and acquires the omnidirectional image. The acquired omnidirectional image is developed in a memory.

[0159] The image rotation unit 216 rotates the spherical image in accordance with the attention point determined by the attention point conversion unit 196. As a result, the attention point coordinates move to the center of the equirectangular image. The rotation process will be described with reference to FIG.

[0160] The cutout processing unit 220 cuts out a part (center) of the rotated omnidirectional image to generate a cutout image. Cutting out means taking out a certain part. It may also be called trimming. In a preferred embodiment, the cutout processing unit 220 cuts out the central part of the converted omnidirectional image, thereby cutting out an image corresponding to a part of a certain size centered on a point of interest in the omnidirectional image.

[0161] In the embodiment described below, the cutout processing unit 220 is described as having a function of cutting out a part of an image to generate a cutout image. However, in other embodiments, the function of the cutout processing unit 220 may be a function of reducing the resolution in addition to the function of cutting out a part of an image to generate a cutout image.

[0162] The enlargement / letterbox processing unit 222 performs enlargement processing on the image cut out by the cut-out processing unit 220 in accordance with the resolution and aspect ratio of the video output device of the output destination, such as a display or projector, and performs processing to add black bands above and below the cut-out image portion, thereby generating a display image. The output unit 224 outputs (displays) the display image generated through processing by the enlargement / letterbox processing unit 222 via the video output interface 129. Note that the processing by the enlargement / letterbox processing unit 222 can be omitted if the cut-out image corresponds to the resolution and aspect ratio of the video output device.

[0163] In the case of a still image, at least every time the attention point changes, typically at every predetermined interval, the video output process by the above functional units (image rotation unit 216, cropping processing unit 220, enlargement / letterbox processing unit 222, and output unit 224) is repeatedly executed on the same omnidirectional image, and the display image is updated according to the attention point at that time. In the case of a moving image, typically, the video output process by the above functional units is repeatedly executed for each frame, and the display image is updated.

[0164] The photographer 8 changes the focus point by tilting the imaging device 5a forward, backward, left, right, or rotating it based on a state in which the imaging device 5a is facing directly upward, and the user can view a display image of the omnidirectional image according to the changed focus point.

[0165] <About conversion tables> FIG. 30 is a diagram for explaining a conversion table and conversion from a planar coordinate system to a spherical coordinate system. FIG. 30(a) is a diagram for explaining a conversion table used by the imaging device 5a according to this embodiment. The conversion table specifies the projection of a partial image expressed in a planar coordinate system (x, y) of the imaging element onto an equirectangular image (hereinafter referred to as a corrected image) in a spherical coordinate system (θ, φ). The conversion table holds, for each fisheye lens, information that associates the coordinate values ​​(θ, φ) of the corrected image with the coordinate values ​​(x, y) of the partial image before correction that are mapped to the coordinate values ​​(θ, φ), for all coordinate values ​​(θ, φ). In the example of FIG. 30, the angle that one pixel is responsible for is 1 / 10 degrees in both the φ direction and the θ direction, and the conversion table has information indicating a correspondence relationship of 3600×1800 for each fisheye lens. The original conversion table can be calculated in advance by the manufacturer or the like after correcting distortion from an ideal lens model, and can be used as a table.

[0166] As shown in Fig. 30(b), the conversion table projects the attention point coordinates 199A expressed in the planar coordinate system (x, y) onto the spherical coordinate system (θ, φ). The attention point coordinates 199A expressed in the planar coordinate system (x, y) are always in a constant direction as seen from the imaging device 5a. Therefore, if the conversion table is fixed, the attention point coordinates 199B in the spherical coordinate system (θ, φ) are also constant.

[0167] However, since the posture of the imaging device 5a changes depending on how the user holds it, the orientation of the real space and the orientation of the corrected image will not match unless the conversion table is corrected according to the posture information. For this reason, in the zenith correction, this conversion table is corrected according to the posture information.

[0168] <Conversion table correction> 31 is a diagram for explaining the correction of the conversion table based on the posture information. Here, the three-dimensional orthogonal coordinates before the coordinate conversion are expressed as (x1, y1, z1), the spherical coordinates of (x1, y1, z1) are expressed as (θ1, φ1), the three-dimensional orthogonal coordinates after the coordinate conversion are expressed as (x2, y2, z2), and the spherical coordinates of (x2, y2, z2) are expressed as (θ2, φ2).

[0169] In the conversion table correction process, conversion from spherical coordinates (θ1, φ1) to spherical coordinates (θ2, φ2) is performed using equations (1) to (6).

[0170]

number

[0171] Next, using the posture information α, β, and γ of the imaging device 5a pointing to the target, the three-dimensional orthogonal coordinate system (x1, y1, z1) is transformed into a three-dimensional orthogonal coordinate system (x2, y2, z2) according to equation (4). Equation (4) means that the original coordinate system is rotated by α around the x-axis, by β around the y-axis, and by γ around the z-axis to obtain the transformed coordinate system.

[0172] Finally, using equations (5) and (6), the converted three-dimensional orthogonal coordinates (x2, y2, z2) are converted back to spherical coordinates (θ2, φ2). If (θ1, φ1) are the coordinates of the spherical coordinate system before the conversion table is corrected, the spherical coordinates of the conversion table can be corrected to (θ2, φ2) according to the attitude information of the image capture device 5a.

[0173] When the attention point conversion unit 196 converts the attention point coordinates in the planar coordinate system into the attention point coordinates in the spherical coordinate system using the corrected conversion table, these become the attention point coordinates in the spherical coordinate system after zenith correction.

[0174] <Image rotation processing by the image rotation unit> Next, the rotation process by the image rotation unit 216 will be described with reference to Fig. 32. Fig. 32 is a diagram showing a schematic diagram of the image rotation process by the image rotation unit 216. Image rotation refers to a process of moving the coordinates of the point of interest to the center of the image on the spherical coordinate system. The new coordinates after rotation are (θ N ,φ N ) · The coordinates of the point of interest are (θ0,φ0) Let any coordinate to be rotated be (θ, φ) Let us assume that.

[0175] The coordinates of the attention point can be moved to the center of the image using equations (7) and (8). θ N =θ-θ0+ 180° (θ-θ0≦ 180°) θ-θ0- 180° (θ-θ0> 180°) … (7) φ N =φ-φ0+ 90° (φ-φ0≦ 90°) φ-φ0- 90° (φ-φ0> 90°) … (8) Equations (7) and (8) rotate (move) any pixel by the difference between the coordinates of the center point and the point of interest in the θ and φ directions.

[0176] <How to send spherical images> Next, a process of transmitting the omnidirectional image and audio data obtained at the point A to each of the other communication terminals 2 to 4 via the communication management system 50 will be described with reference to Fig. 33. Fig. 33 is a sequence diagram showing a communication process of the omnidirectional image and audio data in a video call.

[0177] First, the communication unit 18a of the imaging device 5a transmits a spherical image obtained by capturing an image of a subject, scenery, or the like, and sound data obtained by collecting sound to the communication unit 98 of the communication terminal 1 (step S101). The imaging device 5a attaches attention point information regardless of the determination result of the attention point designation presence / absence determination unit 198. As a result, the communication unit 98 of the communication terminal 1 receives the spherical image and sound data.

[0178] Next, the transmitting / receiving unit 91 of the communication terminal 1 transmits the omnidirectional image, the sound data, and the attention point information transmitted from the imaging device 5a to the communication management system 50 (step S102). This transmission includes an image data ID for identifying the captured image data to be transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the omnidirectional image (image data ID), the sound data, and the attention point information.

[0179] Next, the transmitting / receiving unit 51 of the communication management system 50 transmits the omnidirectional image (image data ID), sound data, and attention point information to the communication terminals 2 to 4 participating in the same video call as the communication terminal 1 (steps S103, S104, and S105). Each of these transmissions includes an image data ID for identifying the omnidirectional image to be transmitted. As a result, the transmitting / receiving unit 71 of the communication terminal 2, the transmitting / receiving unit 31c of the communication terminal 3, and the transmitting / receiving unit 31d of the communication terminal 4 receive the omnidirectional image (image data ID), sound data, and attention point information, respectively. Note that the communication terminal 1 also generally receives and displays the omnidirectional image of its own location from the communication management system 50, but this is omitted in FIG. 33.

[0180] Next, a process in which the imaging device 5a generates a celestial sphere image and a process in which the communication terminals 2 to 4 display the celestial sphere image will be described in detail with reference to Fig. 34. Fig. 34 is an example of a flowchart illustrating a process in which the imaging device 5a generates a celestial sphere image and a process in which the communication terminals 2 to 4 display the celestial sphere image.

[0181] 34(a) shows the process of the imaging device 5a. Note that the process of the communication terminal 1 is omitted.

[0182] The coordinates of the point of interest in the plane coordinate system are defined in the point of interest definition unit 194 (S200). The definition is statically set in advance as described with reference to FIG.

[0183] When capturing an image, the attention point designation presence / absence determination unit 198 determines whether the photographer 8 has designated a point of interest based on sensor information (S201). The sensor information is information that detects whether a button on the imaging device 5a is pressed. If pressed, it is determined that a point of interest has been designated. If a point of interest has been designated, the attention point information in Table 1 is set to "designated."

[0184] The captured image acquisition unit 202 acquires captured images from the image sensors 130A and 130B (S202).

[0185] Next, the stitching processor 204 detects a stitch position in the overlapping area between the two acquired partial images, and reflects the result of the stitch position detection in the conversion table (S203). By reflecting the result of the stitch position detection, the conversion table shown in Fig. 30(a) is modified so that the coordinate values ​​(θ, φ) of the corrected image correspond to the coordinate values ​​(x, y) of the partial image reflecting the correction of the stitch position.

[0186] The zenith correction unit 206 corrects the conversion table based on the attitude information (S204). That is, the zenith correction is performed.

[0187] Next, the attention point conversion unit 196 uses the conversion table to convert the attention point coordinates defined by the attention point definition unit 194 into attention point coordinates in the spherical coordinate system (S205). Also, the attention point coordinates in the spherical coordinate system obtained by the conversion are set in the attention point information in Table 1. Note that even if it is determined in step S201 that the attention point is not specified, the attention point coordinates are set, but if it is determined that the attention point is not specified, the attention point coordinates do not have to be set.

[0188] The spherical image and attention point information generated by the above process are transmitted to the communication terminals 2 to 4 as described with reference to FIG.

[0189] FIG. 34(b) is a flowchart showing the process of the display control unit 34 of the communication terminals 2 to 4.

[0190] First, the image rotation unit 216 determines whether or not a point of interest is specified in the point of interest information attached to the spherical image (S210).

[0191] If a point of interest is specified, the image rotation unit 216 rotates the image so that the point of interest is located at the center of the equirectangular image (S211). The point of interest is forcibly displayed regardless of the predetermined area image that has been displayed up until that point.

[0192] Next, the cropping processor 220 crops out the center part of the spherical image to generate a cropped image (S212). It is assumed that the cropping area is determined in advance. In the example of FIG. 8, the center point CP is the coordinates of the point of interest, and the angle of view α and the distance f are determined in advance.

[0193] If no attention point is specified, the user can rotate the omnidirectional image as desired. In the case of a still image, the predetermined region T that the user last rotated the omnidirectional image is displayed, and in the case of a video, the display of the predetermined region T that the user last rotated is maintained. The cutout processing unit 220 cuts out the predetermined region T that the user determined by operation (S213).

[0194] The enlargement / letterbox processing unit 222 enlarges the cut-out image and adds black bands in accordance with the resolution and aspect ratio of the output destination, and generates a display image (S214).

[0195] The output unit 224 outputs the generated display image to the display I / F 317 and the display 917 (S215).

[0196] <Example of communication terminal display> FIG. 35 is an example of the video display screen 250 displayed on the display 917 of the communication terminal 2 at the location B. The left display area (layout number "1") of the video display screen 250 displays a panoramic image of the location A, and the upper right display area (layout number "2") displays a panoramic image of the location C. Furthermore, the middle right display area (layout number "3") of the video display screen 250 displays an image of the location D, and the lower right display area (layout number "4") displays an image of the location B (the user's location). The display area with layout number "1" is the main display area, and the display areas with layout numbers "2", "3" and "4" are sub-display areas. The image in the main display area and the image in the sub-display area can be changed by each communication terminal. Usually, at each location, the main display area displays an image of the location where the central person in the video call is located.

[0197] Note that a celestial sphere icon 192 is displayed in the display areas of layout number 1 and layout number 2. This allows the user to know that the video image displayed in the display area is a celestial sphere image, and the user can change the predetermined area T. Also, a focus point icon 193 is displayed in the display area of ​​layout number 1. The focus point icon 193 is displayed by the display control unit 74 when a focus point is specified in the focus point information. The focus point icon 193 indicates that a focus point has been specified. This allows the user to know that a focus point is currently displayed, and that the predetermined area T cannot be changed.

[0198] The predetermined area T cannot be changed means that the reception unit 72 does not accept a change of the display area, or that the reception unit 72 accepts a change of the display area but when the user 9b stops the operation to change the display area, the cut-out processing unit 220 cuts out the attention point converted by the attention point conversion unit 196 from the image and displays it again. For example, the user 9b can display any display area only while dragging (while the mouse is clicked) or while swiping and keeping the finger touching.

[0199] The user 9b may be able to stop the display of the attention point. In this case, for example, a predetermined button is prepared. When this button is pressed once, the display of the attention point is stopped, and the user 9b can display any predetermined area T. When this button is pressed once again, the attention point is automatically displayed.

[0200] As shown in Fig. 35, video communication is possible at multiple locations, so even if users 9b and 9d know each other but cannot go to a real estate store together, they can view the property with a sense of realism. For example, if users 9b and 9d are a married couple and one of them can go to the store but the other cannot, they can view the property from their respective locations and exchange their impressions on the spot.

[0201] <Summary> As described above, in the image communication system 10 of this embodiment, the focus point coordinates are defined in advance in the imaging device 5a, and the photographer 8 can point to the object he or she wants to show using these focus point coordinates, allowing users at other locations to focus on the focus point in real time using only the imaging device.

[0202] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0203] For example, although the present embodiment has been described taking real estate previews as an example, the application of the image communication system 10 is not limited to this. For example, the image communication system 10 can be applied to cases where local objects are pointed out, such as exhibitions, exhibitions, factory tours, sightseeing, and inspections.

[0204] In this embodiment, a human being points at the target using the imaging device 5a, but a machine, a robot, or an animal may point at the target. For example, if the imaging device 5a is fixed in the direction of travel of a mobile machine or the like, the target in the direction of travel can be displayed at all times, and if necessary, the image can be rotated to check the surrounding situation. In this case, it is preferable that the side receiving the omnidirectional image can switch whether or not to display the coordinates of the point of interest.

[0205] In the present embodiment, a spherical image is used as an example to display a part of the attention point, but the spherical image does not necessarily need to capture 360 ​​degrees of the surroundings. For example, it may be possible to capture only a hemisphere, or to capture 360 ​​degrees only in the horizontal direction. In addition, it may be a planar image with a large number of pixels that cannot be displayed on the display.

[0206] 15, 16, 29, etc. are divided according to main functions to facilitate understanding of the processing by the image communication system 10. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing of the image communication system 10 can be divided into more processing units depending on the processing content. Also, it can be divided so that one processing unit includes more processes.

[0207] There may be a plurality of communication management systems 50, or the functions of the communication management systems 50 may be distributed among a plurality of servers. There may be a relay device that relays image data and sound data.

[0208] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above. [Explanation of symbols]

[0209] 1~4 Communication terminal 5. Imaging device 8. Photographer 10. Image communication system [Prior art documents] [Patent documents]

[0210] [Patent Document 1] JP 2016-167739 A

Claims

1. A shared communication terminal that communicates with other communication terminals using spherical images via a communication management system, a determination means for determining whether coordinates in a spherical coordinate system are specified in the omnidirectional image received from the communication management system; a program that functions as a display control unit that displays a partial area of ​​the omnidirectional image based on the coordinates when the determination unit determines that the coordinates have been specified.

2. The program according to claim 1, wherein the display control means displays a partial area of ​​the omnidirectional image centered on the coordinates in the omnidirectional image.

3. The program described in claim 1, wherein the display control means displays a point of interest icon based on the coordinates on the spherical image.

4. The program according to claim 1, wherein the display control means prevents a part of the spherical image from being changed when the determination means determines that the coordinates have been specified.

5. The program described in claim 1, wherein the coordinates are specified by a user operating the other communication terminal.

6. The program as described in claim 1, wherein, when the determination means determines that the coordinates have not been specified, the display control means displays a partial area of ​​the spherical image that has been arbitrarily changed by a user.

7. A shared communication terminal that communicates with another communication terminal using a spherical image via a communication management system, a determination means for determining whether coordinates in a spherical coordinate system are specified in the omnidirectional image received from the communication management system; a display control means for displaying a partial area of ​​the omnidirectional image based on the coordinates when the determination means determines that the coordinates have been designated; A shared communication terminal having the above configuration.

8. A display method in which a shared communication terminal communicates with another communication terminal using a spherical image via a communication management system, comprising: A process of determining whether coordinates in a spherical coordinate system are specified for the omnidirectional image received from the communication management system; when it is determined that the coordinates have been designated, a process of displaying a partial area of ​​the omnidirectional image based on the coordinates; How to display.

9. An image communication system in which a shared communication terminal communicates with another communication terminal using a spherical image via a communication management system, a determination means for determining whether coordinates in a spherical coordinate system are specified in the omnidirectional image received from the communication management system; a display control means for displaying a partial area of ​​the omnidirectional image based on the coordinates when the determination means determines that the coordinates have been designated; An image communication system having the above configuration.

Images