Programs, communication terminals, display methods, image communication systems

Abstract

To provide an image communication system that allows the image source to specify a point of interest in real time. [Solution] An image communication system 10 having an imaging device 5 for capturing images and communication terminals 1 to 4 for displaying the images on a screen, characterized in that it includes: a point of interest definition unit 194 for defining points of interest in the image; an orientation information acquisition unit 15a for acquiring orientation information of the imaging device; a point of interest conversion unit 196 for converting the points of interest defined by the point of interest definition unit according to the orientation information acquired by the orientation information acquisition unit; an extraction processing unit 220 for extracting a predetermined region including the points of interest converted by the point of interest conversion unit from the image; and an output unit 224 for displaying the predetermined region including the points of interest extracted by the extraction processing unit on the screen.

Description

[Technical Field]

[0001] This invention relates to an imaging device, an image communication system, an image processing method, and a program. [Background technology]

[0002] An imaging device is known that uses multiple wide-angle lenses, such as fisheye lenses and ultra-wide-angle lenses, to capture a 360-degree or omnidirectional (hereinafter referred to as the 360-degree sphere) image at once. In this imaging device, the images from each lens are projected onto each image sensor, and the resulting images are stitched together by image processing to generate a 360-degree sphere image. For example, a 360-degree sphere image capturing a subject in the 360-degree sphere can be generated using two wide-angle lenses with a field of view exceeding 180 degrees.

[0003] However, while the 360-degree images generated by imaging devices are represented in a spherical coordinate system, conventional display devices such as screens are two-dimensional, making it difficult to display the entire image at once. For this reason, when displaying 360-degree images, a dedicated viewer is usually used to convert a portion of the 360-degree image's field of view into an image compatible with conventional display devices.

[0004] A technique has been devised to determine a point of interest so that the terminal can display any angle of view from a 360-degree image (see, for example, Patent Document 1). Patent Document 1 discloses a method in which the user changes the point of interest by tilting or rotating the imaging device and displays the point of interest on a normal display device. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, conventional technology had the problem that it was difficult for the image source to specify a point of interest in real time. One use case where specifying a point of interest in real time is desired will be explained using the example of a real estate viewing.

[0006] When the above imaging device is used for property viewings, it can capture images of the entire room, allowing prospective buyers to get an overview of the property without having to travel to it, thus saving them the trouble of moving. For example, a viewing can be realized through the following mechanism. 1. A real estate agent visits the property and takes images of it using an imaging device. 2. Customers who wish to view the property receive the video feed from the imaging device via a communication terminal and view it remotely in real time. 3. Since both video and audio are connected, customers can view properties while conversing with the real estate agent.

[0007] However, since customers can only see a limited field of view in a 360-degree image, they may not be looking at the exact spot the real estate agent is describing, and it is difficult to immediately display that spot on a flat-screen device. In other words, customers don't know where to focus their attention in the 360-degree image. For example, even if a local real estate agent says, "This room has the latest air conditioning! Look at this!", a customer viewing the property remotely may not be looking in the direction of the air conditioning in the 360-degree image, making it difficult to communicate in real time about the designated point of focus.

[0008] To address these inconveniences, a real estate agent could display a preview of a 360-degree image on a communication terminal they carry, allowing them to indicate points of interest. This would then transmit those points of interest to the customer's terminal during a remote viewing, ensuring both the agent and the customer focus on the same points. However, it is difficult for a real estate agent to operate a communication terminal while simultaneously capturing images with an imaging device. While this would be possible if multiple agents were present at the site, securing sufficient personnel is often challenging.

[0009] While there are technologies that analyze 360-degree images to estimate points of interest, these are merely estimations and may not necessarily correspond to the points of interest that real estate agents describe.

[0010] In view of the above problems, the present invention aims to provide an image communication system that allows the image source to specify a point of interest in real time. [Means for solving the problem]

[0011] In view of the above problems, the present invention provides an image communication system comprising an imaging device for capturing images and a communication terminal for displaying the images on a screen, characterized in that it comprises: a point of interest definition unit for defining points of interest in the image; an orientation information acquisition unit for acquiring orientation information of the imaging device; a point of interest conversion unit for converting the points of interest defined by the point of interest definition unit according to the orientation information acquired by the orientation information acquisition unit; an extraction processing unit for extracting a predetermined region including the points of interest converted by the point of interest conversion unit from the image; and an output unit for displaying the predetermined region including the points of interest extracted by the extraction processing unit on the screen. [Effects of the Invention]

[0012] This system provides an image communication system that allows the image source to specify a point of interest in real time. [Brief explanation of the drawing]

[0013] [Figure 1] This diagram illustrates the schematic operation of an image communication system that transmits a 360-degree spherical image with a specified point of interest in any direction between location A and location B. [Figure 2] (a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a top view of the imaging device. [Figure 3] This is a diagram illustrating the use of the imaging device. [Figure 4] (a) is a hemispherical image (front) captured by the imaging device, (b) is a hemispherical image (back) captured by the imaging device, and (c) is an image represented by equirectangular projection. [Figure 5] (a) A conceptual diagram showing the state of covering the sphere with an equirectangular projection image, and (b) A diagram showing a panoramic image of the entire sphere. [Figure 6] This diagram shows the positions of a virtual camera and a predetermined area when a full-spherical panoramic image is treated as a three-dimensional sphere. [Figure 7](a) is a three-dimensional perspective view of FIG. 6, and (b) is a diagram showing a state where an image of a predetermined area is displayed on the display of the communication terminal. [Figure 8] It is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. [Figure 9] It is a diagram showing points in a three-dimensional Euclidean space using spherical coordinates. [Figure 10] It is a schematic diagram of the configuration of the image communication system of this embodiment. [Figure 11] It is a hardware configuration diagram of an example of the imaging device. [Figure 12] It is a hardware configuration diagram of an example of the communication terminals 3 and 4 [Figure 13] It is a hardware configuration diagram of an example of the communication management system. [Figure 14] It is a hardware configuration diagram of an example of the communication terminals 1 and 2. [Figure 15] It is an example of a functional block diagram showing the functions of the image communication system in a block form (Part 1). [Figure 16] It is an example of a functional block diagram showing the functions of the image communication system in a block form (Part 2). [Figure 17] It is a conceptual diagram showing the image type management table. [Figure 18] It is a conceptual diagram showing the imaging device management table. [Figure 19] It is a conceptual diagram showing the predetermined area information management table. [Figure 20] It is a conceptual diagram showing the session management table. [Figure 21] It is a conceptual diagram showing the image type management table. [Figure 22] It is a conceptual diagram showing the predetermined area information management table. [Figure 23] It is an example of a sequence diagram showing the participation process in a specific communication session. [Figure 24] It is a diagram showing an example of a selection screen for a communication session (virtual conference room). [Figure 25] It is a diagram for explaining the coordinate axes of an example of the imaging device. [Figure 26] This diagram illustrates an example of a standard imaging device orientation. [Figure 27] This diagram illustrates an example of values ​​detected by an acceleration and orientation sensor. [Figure 28] (a) is a conceptual diagram showing how the imager points to the subject, (b) is an example of an equirectangular image where the entire sphere image is represented in equirectangular projection without zenith correction, and (c) is an example of an equirectangular image where the entire sphere image is represented in equirectangular projection with zenith correction. [Figure 29] This is a functional block diagram of an example of an image processing unit in an imaging device. [Figure 30] This figure illustrates an example of a conversion table and an example of a conversion from a planar coordinate system to a spherical coordinate system. [Figure 31] This diagram illustrates an example of correcting a conversion table based on posture information. [Figure 32] This diagram schematically illustrates an example of image rotation processing performed by the image rotation unit. [Figure 33] This is a sequence diagram illustrating an example of the communication processing of 360-degree images and audio data during a video call. [Figure 34] This flowchart illustrates an example of the process by which an imaging device generates a 360-degree image and the process by which a communication terminal displays the 360-degree image. [Figure 35] This figure shows an example of a video display screen shown on a display. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment for carrying out the present invention will be described with reference to the drawings, including an image processing system and an image processing method performed by the image processing system.

[0015] <Overview of the image communication system> First, we will explain the general operation of the image communication system 10 using Figure 1. Figure 1 is a diagram illustrating the general operation of the image communication system 10, which transmits a 360-degree spherical image with a specified point of interest in an arbitrary direction between base A and base B.

[0016] (1) The imaging device 5a of this embodiment has predefined coordinates of the point of interest. The coordinates of the point of interest are, for example, the x and y coordinates of the pixels or a certain range of the image sensor in the longitudinal direction (upper side) of the imaging device 5a where the subject is captured. Note that the coordinates of the point of interest are coordinates on the image sensor and are pre-set at the time of shipment.

[0017] (2) If the imager 8 at base A wants the user at base B to focus on the air conditioner 140, the imager 8 points the longitudinal side (upper side) of the imaging device 5a towards the air conditioner 140. The imaging device 5a is constantly acquiring attitude information for the zenith correction described later. The imaging device 5a captures a full-sphere image and performs zenith correction based on the attitude information, so the air conditioner 140 will not be captured at an angle. Also, since the air conditioner 140 is captured in the point of interest coordinates, the subject of the point of interest coordinates is the air conditioner. Furthermore, when the imager 8 tilts the imaging device 5a, the change in the point of interest coordinates is the attitude information itself.Therefore, the transformed point of interest coordinates can be identified by zenith correction.

[0018] (3) The imaging device 5a transmits the 360-degree image and the converted coordinates of the point of interest to the communication terminal 2 at base B via the communication terminal 1.

[0019] (4) Regardless of the previously displayed predetermined region image, the communication terminal 2 extracts a predetermined range including the coordinates of the point of interest, generates a predetermined region image, and displays it on the display or the like. This allows the imager 8 to draw the user's attention to the object they want them to focus on. Furthermore, the predetermined region image generated by the imager's view information cannot be changed by the viewer for at least a certain period of time. In other words, the predetermined region image generated by the imager's view information is forcibly displayed.

[0020] As described above, in this embodiment, the image communication system 10 has a predefined coordinate of the point of interest in the imaging device 5a. When the imager points to the object they want to show using this coordinate of the point of interest, the coordinate of the point of interest is converted according to the orientation information and transmitted to the user. This allows the imager 8 to make the user at base B focus on the point of interest in real time.

[0021] <About Terminology> As described above, the defined point of interest is a single point or a certain area on the image sensor. The subject always appears in the same direction as viewed from the imaging device 5a. In this embodiment, the defined point of interest is a pixel or a certain area of ​​the image sensor that captures a subject on the upper side in the longitudinal direction, but the location of the point of interest can be determined arbitrarily. For example, it could be a pixel of the image sensor that captures a subject in the direction pointed to by the lower side in the longitudinal direction, in the direction of the optical axis of the lens, or in the direction pointed to by the protruding part of the imaging device 5a.

[0022] The orientation information of the imaging device only needs to be information that can identify the direction that any direction the imaging device is pointing in. For example, this could be information indicating how much it is tilted from an upright position, or information regarding the rotation of each axis in three-dimensional space.

[0023] <Method for generating a 360-degree image> The method for generating a 360-degree image will be explained using Figures 2 to 9.

[0024] First, the external appearance of the imaging device 5a will be described using Figure 2. The imaging device 5a is a digital camera used to obtain the image that will be the basis for a three-dimensional 360° spherical image. Figure 2(a) is a left side view of the imaging device 5a, Figure 2(b) is a front view of the imaging device 5a, and Figure 2(c) is a top view of the imaging device 5a.

[0025] As shown in Figure 2(a), the imaging device 5a is small enough to be held in one hand. Also, as shown in Figures 2(a), 2(b), and 2(c), the upper part of the imaging device 5a is equipped with an image sensor 103a on the front side and an image sensor 103b on the rear side. These image sensors 103a and 103b are used in conjunction with an optical component (for example, the fisheye lenses 102a and 102b shown in Figure 11, which will be described later) that can capture hemispherical images (angle of view of 180° or more). Furthermore, as shown in Figure 2(b), an operating section 115, such as a shutter button, is provided on the side of the imaging device 5a opposite to the front side.

[0026] Next, we will explain the usage of the imaging device 5a using Figure 3. Figure 3 is an illustrative diagram of how the imaging device is used. As shown in Figure 3, the imaging device 5a is used, for example, by a user holding it in their hand to image subjects around the user. In this case, two hemispherical images can be obtained by imaging subjects around the user using the image sensors 103a and 103b shown in Figure 2.

[0027] Next, using Figures 4 and 5, we will outline the process from the image captured by the imaging device 5a to the creation of a full-sphere image. Figure 4(a) shows the hemispherical image (front) captured by the imaging device, Figure 4(b) shows the hemispherical image (back) captured by the imaging device, and Figure 4(c) shows the image represented by equirectangular projection (hereinafter referred to as the "equirectangular image"). Figure 5(a) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular image, and Figure 5(b) shows the full-sphere image.

[0028] As shown in Figure 4(a), the image obtained by the image sensor 103a becomes a curved hemispherical image (front side) by the fisheye lens 102a described later. Similarly, as shown in Figure 4(b), the image obtained by the image sensor 103b becomes a curved hemispherical image (rear side) by the fisheye lens 102b described later. The hemispherical image (front side) and the hemispherical image (rear side) which is inverted by 180 degrees are then combined by the imaging device 5a to create an equirectangular image, as shown in Figure 4(c).

[0029] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular image is superimposed to cover the sphere, as shown in Figure 5(a), creating a 360-degree spherical image as shown in Figure 5(b). In this way, the 360-degree spherical image is represented as an image where the equirectangular image is facing the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. The 360-degree spherical image can be a still image or a video.

[0030] As described above, a 360-degree spherical image is an image pasted over a sphere, which can cause discomfort to the human eye. Therefore, by displaying a predetermined area of ​​the 360-degree spherical image (hereinafter referred to as the "predetermined area image") as a flat image with less curvature, it is possible to display it in a way that does not cause discomfort to the human eye. This will be explained using Figures 6 and 7.

[0031] Figure 6 shows the positions of the virtual camera and the predetermined region when the 360-degree image is treated as a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of the user viewing the 360-degree image CE, which is displayed as a three-dimensional sphere. Figure 7(a) is a stereoscopic perspective view of Figure 6, and Figure 7(b) shows the predetermined region image as it appears on a display. In Figure 7(a), the 360-degree image CE shown in Figure 6 is represented as a three-dimensional sphere CS. If the generated 360-degree image CE is a sphere CS, then, as shown in Figure 6, the virtual camera IC is located outside the 360-degree image CE. The predetermined region T in the 360-degree image CE is the imaging region of the virtual camera IC and is identified by predetermined region information indicating the imaging direction and field of view of the virtual camera IC in the three-dimensional virtual space including the 360-degree image CE.

[0032] Then, the predetermined region T shown in Figure 7(a) is displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in Figure 7(b). The image shown in Figure 7(b) is, for example, a predetermined region image represented by the initially set predetermined region information. Note that the predetermined region information may be represented not by the position coordinates of the virtual camera IC, but by the imaging area (X,Y,Z) of the virtual camera IC, which is the predetermined region T. In the following explanation, we will use the imaging direction (rH,rV) and field of view (α) of the virtual camera IC.

[0033] The relationship between the predetermined region information and the image of the predetermined region T will be explained using Figure 8. Figure 8 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 8, rH is the Horizontal Radian, rV is the Vertical Radian, and α is the field of view (Angle). That is, the orientation of the virtual camera IC will be changed so that the point of fixation of the virtual camera IC, indicated by the imaging direction (rH,rV), becomes the center point CP of the predetermined region T, which is the imaging area of ​​the virtual camera IC. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined region T and the center point CP (2L is the diagonal). And, in Figure 8, the trigonometric function shown by the following equation (A) generally holds true.

[0034] L / f = tan(α / 2)···(A) Figure 9 shows a point in three-dimensional Euclidean space using spherical coordinates. Let (r, θ, φ) be the position coordinates of the center point CP when expressed in spherical polar coordinates. (r, θ, φ) are the radial, polar angle, and azimuth angle, respectively. The radial r is equal to f because it is the distance from the origin of the three-dimensional virtual space containing the panoramic image to the center point CP. Figure 9 illustrates these relationships. Hereafter, the center point will be explained using the position coordinates (r, θ, φ).

[0035] <Overview of the image communication system> Next, the general configuration of the image communication system 10 of this embodiment will be explained using Figure 10. Figure 10 is a schematic diagram of the configuration of the image communication system 10 of this embodiment.

[0036] As shown in Figure 10, in this embodiment, the image communication system 10 allows communication terminals 1 to 4 located at bases A to D to communicate via a communication network 100 such as the Internet, enabling the communication terminals 1 to 4 at each base to share video. Base A is equipped with an imaging device 5a and a communication terminal 1, base B is equipped with a communication terminal 2 and an imaging device 5b, base C is equipped with an imaging device 5c, a communication terminal 3, and a display 6c, and base D is equipped with a communication terminal 4 and a display 6d.

[0037] As described above, imaging devices 5a and 5c are special digital cameras used to capture subjects, landscapes, etc., and obtain two hemispherical images that form the basis of a 360-degree spherical image. On the other hand, imaging device 5b is a general-purpose digital camera used to capture subjects, landscapes, etc., and obtain a general planar image.

[0038] Communication terminals 3 and 4 are dedicated video conferencing terminals and display video call images on displays 6c and 6d, respectively, via wired cables such as USB (Universal Serial Bus) cables. Normally, communication terminals 3 and 4 capture images of users, etc., using the camera 312 shown in Figure 12 below. However, when connected via a wired cable to the cradle 7 to which the imaging device 5c is mounted, the imaging device 5c takes priority, and a 360-degree image can be obtained. When using a wired cable, the cradle 7 not only facilitates communication between the imaging device 5c and communication terminal 3, but also supplies power to the imaging device 5c and supports the imaging device 5c.

[0039] Communication terminals 1 and 2 are general-purpose information processing devices that communicate with other locations by running video conferencing application software. Communication terminal 1 may include, for example, a notebook PC (Personal Computer), mobile phone, smartphone, tablet device, car navigation system, game console, PDA (Personal Digital Assistant), wearable PC, or desktop PC. However, it is not limited to these.

[0040] The communication terminal 1 displays the video call image on the display 917, which will be described later, provided on the device. Normally, the communication terminal 1 takes images using the CMOS (Complementary Metal Oxide Semiconductor) sensor 905, which will be described later, provided on the device, but it can also acquire a 360-degree spherical image obtained by the imaging device 5a using wireless communication technologies such as Wi-Fi (Wireless Fidelity) or Bluetooth (registered trademark).

[0041] The communication terminal 2 displays the video call image on the display 917, which will be described later and is located on its own device. The communication terminal 2 captures images of the user, etc., with an externally attached imaging device 5b.

[0042] Furthermore, communication terminals 1-4 have OpenGL ES installed, allowing them to create predetermined region information that indicates a portion of a 360-degree image, or to create predetermined region images from 360-degree images sent from other communication terminals. Therefore, each communication terminal 1-4 can display predetermined region images extracted from a 360-degree image.

[0043] At base A, there is at least one imager 8 who holds the imaging device 5a either in their hand or attached to a rod-shaped member. The imager 8 can move together with the imaging device 5a. In addition, the communication terminal 1 displays the images captured by the imaging devices 5a to 5c and the camera 312 of the communication terminal 4 on the display 917.

[0044] At site B, there is one user 9b, and communication terminal 2 displays the images captured by imaging devices 5a-5c and the camera 312 of communication terminal 4 on display 917. User 9b at site B is one of the users who focuses on the point of interest designated by the imager 8. However, the number of users is just an example.

[0045] At site C, there are two users, 9c1 and 9c2. Communication terminal 3 displays the images captured by imaging devices 5a-5c and the camera 312 of communication terminal 4 on display 6c. Users 9c1 and 9c2 at site C are among the users who focus on the points of interest designated by the imager 8. However, the number of users is just an example.

[0046] At site D, there is one user 9d, and the communication terminal 4 displays the images captured by the imaging devices 5a-5c and the camera 312 of the communication terminal 4 on the display 6d. User 9d at site D is one of the users who focuses on the point of interest designated by the imager 8. However, the number of users is just an example.

[0047] The communication management system 50 manages and controls the communication of communication terminals 1 to 4. Therefore, the communication management system 50 is also a communication control system. The communication management system 50 is installed in service companies that provide video communication services, etc. Furthermore, the communication management system 50 may be constructed by a single computer, or it may be constructed by multiple computers in which each part (function, means, or storage unit) is divided and arbitrarily assigned.

[0048] Note that the number of locations, the types of communication terminals 1-4 placed at each location, the types of imaging devices 5a-5c, and the number of users shown in Figure 10 are examples only. In this embodiment, location A and one other location are sufficient. Furthermore, location B does not need to have imaging device 5b; it is sufficient that it can display the 360-degree image transmitted from location A.

[0049] Furthermore, the imaging device 5a and the communication terminal 1 at base A may be integrated. In other words, if the imaging device 5a has the function of connecting to the communication network 100, the communication terminal 1 is not necessary. In this case, the imaging device 5a becomes the communication terminal 1. In this embodiment, the imager 8 can specify the point of interest using only the imaging device 5a. However, with the presence of the communication terminal 1, the imager 8 can specify the point of interest by operating the communication terminal 1. Alternatively, the communication terminal 1 may receive the 360-degree image and the coordinates of the point of interest via a storage medium without communicating with the imaging device 5a at base A.

[0050] Hereafter, any imaging device among imaging devices 5a to 5c will be referred to as "imaging device 5". Furthermore, any display among displays 6c and 6d will be referred to as "display 6".

[0051] <<Hardware configuration of the embodiment>> Next, the hardware configuration of the imaging devices 5a and 5c, communication terminals 1 to 4, and communication management system of this embodiment will be described in detail using Figures 11 to 14. Note that since the imaging device 5c is a general-purpose camera, a detailed explanation will be omitted.

[0052] <Hardware configuration of imaging devices 5a and 5c> First, the hardware configuration of the imaging devices 5a and 5c will be explained using Figure 11. Figure 11 is a hardware configuration diagram of the imaging devices 5a and 5c. In the following, imaging devices 5a and 5c will be assumed to be omnidirectional imaging devices using two image sensors, but there may be more than two image sensors. Furthermore, it is not necessary for them to be devices specifically for omnidirectional imaging; an omnidirectional imaging unit can be attached to a regular digital camera or smartphone, etc., to effectively provide the same functionality as imaging devices 5a and 5c.

[0053] As shown in Figure 11, the imaging devices 5a and 5c consist of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network interface 116, a communication unit 117, and an antenna 117a.

[0054] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b, each having a field of view of 180° or more for forming hemispherical images, and two image sensors 103a and 103b, each corresponding to the wide-angle lens. The image sensors 103a and 103b include an image sensor such as a CMOS sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lenses 102a and 102b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the image sensors are set.

[0055] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are connected separately from the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via bus 110. Furthermore, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, acceleration / direction sensor 118, and gyro sensor 119 are also connected to bus 110.

[0056] The image processing unit 104 receives image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create equirectangular image data as shown in Figure 4(c).

[0057] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 acting as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0058] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 5a and 5c, there may also be functions to display a preview or video on a display (for example, the display on the communication terminals 1 and 3). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0059] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging devices 5a and 5c are not provided with displays, but they may be provided.

[0060] Microphone 108 converts sound into sound (signal) data. Sound processing unit 109 receives the sound data output from microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0061] The CPU 111 controls the overall operation of the imaging devices 5a and 5c and performs necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress and processed equirectangular image data from the image processing unit 104.

[0062] The control unit 115 is a collective term for various operation buttons, a power switch, a shutter button, and a touch panel that combines display and operation functions. The user inputs various imaging modes and imaging conditions by operating the operation buttons.

[0063] The network interface 116 is a general term for interface circuits (such as USB interfaces) that connect to external media such as SD cards or personal computers. The network interface 116 can be wireless or wired. The equirectangular image data stored in the DRAM 114 is recorded to external media via this network interface 116, or transmitted to external devices such as the communication terminal 3 via the network interface 116 as needed.

[0064] The communication unit 117 communicates with external devices such as communication terminals 1 and 3 via antennas 117a provided on imaging devices 5a and 5c using short-range wireless technologies such as Wi-Fi and NFC (Near Field Communication). This communication unit 117 can also transmit equirectangular image data to external devices such as communication terminals 1 and 3.

[0065] The acceleration / orientation sensor 118 calculates the orientation and tilt (roll angle, yaw angle, and pitch angle) of the imaging devices 5a and 5c from the Earth's magnetic field and outputs orientation / tilt information. This orientation / tilt information is an example of related information (metadata) in accordance with Exif and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data.

[0066] The gyro sensor 119 is a 3-axis or 6-axis sensor that detects rotational speed around the X, Y, and Z axes. By accumulating (integrating) the rotational speed, the rotation angle (attitude information) can be obtained.

[0067] <Hardware configuration of communication terminals 3 and 4 (video conferencing terminals)> Next, the hardware configuration of communication terminals 3 and 4 will be explained using Figure 12. Figure 12 is a hardware configuration diagram of communication terminals 3 and 4. As shown in Figure 12, communication terminals 3 and 4 are equipped with a CPU 301, ROM 302, RAM 303, flash memory 304, SSD 305, media I / F 307, operation buttons 308, power switch 309, bus line 310, network I / F 311, camera 312, image sensor I / F 313, microphone 314, speaker 315, sound input / output I / F 316, display I / F 317, external device connection I / F 318, short-range communication circuit 319, and antenna 319a of the short-range communication circuit 319.

[0068] Of these components, the CPU 301 controls the overall operation of the communication terminals 3 and 4. The ROM 302 stores programs used to drive the CPU 301, such as the IPL (Initial Program Loader). The RAM 303 is used as the work area for the CPU 301. The flash memory 304 stores various data such as communication programs, image data, and sound data. The SSD (Solid State Drive) 305 controls the reading or writing of various data to the flash memory 304 according to the control of the CPU 301. Note that an HDD may be used instead of the SSD. The media I / F 307 controls the reading or writing (storage) of data to the recording media 306, such as the flash memory. The operation button 308 is used to select the destination of the communication terminals 3 and 4, etc. The power switch 309 is a switch for turning the power of the communication terminals 3 and 4 ON / OFF.

[0069] Furthermore, the network I / F 311 is an interface for data communication using a communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 301. The image sensor I / F 313 is a circuit that controls the driving of the camera 312. The microphone 314 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 316 is a circuit that processes the input and output of sound signals between the microphone 314 and the speaker 315 according to the control of the CPU 301. The display I / F 317 is a circuit that transmits image data to an external display 6 according to the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) or Bluetooth (registered trademark).

[0070] Furthermore, the bus line 310 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 301 shown in Figure 12.

[0071] Display 6 is a type of display means composed of liquid crystal or organic EL (electroluminescence) that displays images of the subject, operation icons, etc. Display 6 is connected to display I / F 317 by cable 6y. This cable 6y may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0072] The camera 312 includes a lens and a solid-state image sensor that converts light into electric charge to create an image (video) of the subject. CMOS sensors and CCD sensors are used as the solid-state image sensor. External devices such as external cameras, external microphones, and external speakers can be connected to the external device connection I / F 318 via USB (Universal Serial Bus) cables, etc. When an external camera is connected, it is driven in priority over the built-in camera 312, according to the control of the CPU 301. Similarly, when an external microphone or external speaker is connected, it is driven in priority over the built-in microphone 314 and built-in speaker 315, respectively, according to the control of the CPU 301.

[0073] Furthermore, the recording medium 306 is configured to be detachable from the communication terminals 3 and 4. In addition, any non-volatile memory that reads or writes data according to the control of the CPU 301 may be used, not limited to flash memory 304, but also including EEPROM (Electrically Erasable and Programmable ROM), etc.

[0074] <Hardware configuration of the communication management system> Next, we will explain the hardware configuration of the communication management system 50 using Figure 13. Figure 13 is a hardware configuration diagram of the communication management system.

[0075] The communication management system 50 includes a CPU 501 that controls the overall operation of the communication management system 50, a ROM 502 that stores programs used to drive the CPU 501 such as an IPL, a RAM 503 used as the work area of ​​the CPU 501, an HD 504 that stores various data such as programs for the communication management system 50, an HDD (Hard Disk Drive) 505 that controls the reading or writing of various data to the HD 504 according to the control of the CPU 501, a media drive 507 that controls the reading or writing (storage) of data to a recording medium 506 such as flash memory, a display 508 that displays various information such as cursors, menus, windows, characters, or images, a network I / F 509 for data communication using the communication network 100, a keyboard 511 with multiple keys for inputting characters, numbers, and various instructions, a mouse 512 for selecting and executing various instructions, selecting processing targets, moving the cursor, etc., and a CD-RW (Compact) as an example of a removable recording medium. The system includes a CD-RW drive 514 that controls the reading of various data from the Disc-ReWritable) 513, and bus lines 510 such as an address bus and a data bus for electrically connecting the above components as shown in Figure 13.

[0076] <Hardware configuration of communication terminals 1 and 2> Next, the hardware of communication terminals 1 and 2 will be described using Figure 14. Figure 14 is a hardware configuration diagram of communication terminals 1 and 2. As shown in Figure 14, communication terminals 1 and 2 are equipped with a CPU 901, ROM 902, RAM 903, EEPROM 904, CMOS sensor 905, acceleration / direction sensor 906, media I / F 908, and GPS receiver 909.

[0077] Of these, the CPU 901 controls the overall operation of communication terminals 1 and 2. The ROM 902 stores programs used to drive the CPU 901, such as IPL. The RAM 903 is used as the work area for the CPU 901. The EEPROM 904 reads or writes various data, such as programs for communication terminals 1 and 2, according to the control of the CPU 901. The CMOS sensor 905 captures images of a subject (mainly a self-portrait) and obtains image data according to the control of the CPU 901. The acceleration / direction sensor 906 is a variety of sensors, such as an electronic magnetic compass, gyrocompass, and acceleration sensor that detect the Earth's magnetic field. The media I / F 908 controls the reading or writing (storage) of data to or from the recording medium 907, such as flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0078] Furthermore, the communication terminals 1 and 2 are equipped with a long-range communication circuit 911, a camera 912, an image sensor interface 913, a microphone 914, a speaker 915, an audio input / output interface 916, a display 917, an external device connection interface 918, a short-range communication circuit 919, an antenna 919a for the short-range communication circuit 919, and a touch panel 921.

[0079] Of these, the long-range communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 901. The image sensor interface 913 is a circuit that controls the driving of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs sound. The sound input / output interface 916 is a circuit that processes the input and output of sound signals between the microphone 914 and the speaker 915 according to the control of the CPU 901. The display 917 is a type of display means such as a liquid crystal or organic EL that displays images of the subject and various icons. The external device connection interface 918 is an interface for connecting various external devices. The short-range communication circuit 919 is a communication circuit such as NFC or Bluetooth (registered trademark). The touch panel 921 is a type of input means that allows the user to operate the communication terminals 1 and 2 by pressing the display 917.

[0080] Furthermore, communication terminals 1 and 2 are equipped with a bus line 910. The bus line 910 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 901.

[0081] Furthermore, recording media such as CD-ROMs on which the above programs are stored, as well as hard drives on which these programs are stored, may be provided domestically or internationally as program products.

[0082] <<Embodiment Functional Configuration>> Next, the functional configuration of this embodiment will be described using Figures 15 to 22. Figures 15 and 16 are functional block diagrams that show the functions of the image communication system 10 in a block-like manner.

[0083] <Functional configuration of imaging device 5a> As shown in Figure 15, the imaging device 5a includes a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a posture information acquisition unit 15a, an image processing unit 16a, a communication unit 18a, and a storage / reading unit 19a. Each of these units, except for the image processing unit 16a, is a function or means realized by one of the components shown in Figure 11 operating according to instructions from the CPU 111 that follow a program for the imaging device deployed on SRAM 113 to DRAM 114. The image processing unit 16a is assumed to be realized by a circuit module such as an ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), or FPGA (field programmable gate array), but it may also be realized in software.

[0084] Furthermore, the imaging device 5a has a storage unit 1000a constructed from ROM 112, SRAM 113, and DRAM 114 as shown in Figure 11. The storage unit 1000a stores the GUID (Globally Unique Identifier) ​​of the device.

[0085] (Functional configuration of each part of the imaging device 5a) The reception unit 12a of the imaging device 5a is mainly implemented by the operation unit 115 and the CPU 111 shown in Figure 11, and receives operation input from the imager 8.

[0086] The imaging unit 13a is mainly realized by the imaging unit 101, image processing unit 104, and imaging control unit 105 shown in Figure 11, as well as the processing of the CPU 111, to capture landscapes and the like and obtain captured image data (spherical image).

[0087] The sound collection unit 14a is realized by the processing of the 108 shown in Figure 11, the sound processing unit 109, and the CPU 111, and collects sound from the surrounding area of ​​the imaging device 5a.

[0088] The attitude information acquisition unit 15a acquires attitude information regarding the attitude of the imaging device (tilt in three-dimensional space). The image processing unit 16a performs a process to combine two hemispherical images into one equirectangular image. During this process, there is a step to perform zenith correction based on the attitude information, which allows the coordinates of the point of interest indicated by the longitudinal direction (upward) to be converted into coordinates in a spherical coordinate system (equirectangular coordinates) that represent the direction indicated by the imager.

[0089] The communication unit 18a is mainly implemented by the processing of the CPU 111 and can communicate with the communication unit 98 of the communication terminal 1 using short-range wireless communication technologies such as NFC, Bluetooth (registered trademark), and Wi-Fi.

[0090] The storage and reading unit 19a is mainly implemented by the processing of the CPU 111 shown in Figure 11, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.

[0091] The imaging device 5c includes a reception unit 12c, an imaging unit 13c, a sound collection unit 14c, a posture information acquisition unit 15c, an image processing unit 16c, a communication unit 18c, a storage / reading unit 19c, and a storage unit 1000c. However, since these units perform the same functions as the reception unit 12a, imaging unit 13a, sound collection unit 14a, posture information acquisition unit 15a, image processing unit 16a, communication unit 18a, storage / reading unit 19a, and storage unit 1000a in the imaging device 5a, their descriptions are omitted.

[0092] <Functional configuration of communication terminal 1> As shown in Figure 15, the communication terminal 1 includes a transmitting / receiving unit 91, a receiving unit 92, an image / sound processing unit 93, a display control unit 94, a determination unit 95, a creation unit 96, a calculation unit 97, a communication unit 98, and a storage / reading unit 99. Each of these units is a function or means realized by any of the components shown in Figure 14 operating according to instructions from the CPU 901 that follow the program for the communication terminal 1 deployed from the EEPROM 904 onto the RAM 903.

[0093] Furthermore, the communication terminal 1 has a storage unit 9000 constructed from ROM 902, RAM 903, and EEPROM 904 as shown in Figure 14. This storage unit 9000 contains an image type management DB 9001, an imaging device management DB 9002, and a predetermined area management DB 9003.

[0094] Of these, the image type management DB9001 is composed of the image type management table shown in Figure 17. The imaging device management DB9002 is composed of the imaging device management table shown in Figure 18. The predetermined area management DB9003 is composed of the predetermined area information management table shown in Figure 19.

[0095] (Image type management table) Figure 17 is a conceptual diagram showing the image type management table. In this image type management table, the image data ID, the IP address (an example of the destination of the sending terminal), and the source name are stored and managed in association with each other. Of these, the image data ID is an example of image data identification information used to identify image data when performing video communication. Image data sent from the same sending terminal is assigned the same image data ID. This allows the receiving terminal (the receiving communication terminal) to identify the sending terminal of the received image data. The IP address of the sending terminal indicates the IP address of the communication terminal that sends the image data indicated by the associated image data ID. The source name is a name used to identify the imaging device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by the communication terminal 3, etc., according to a predetermined naming convention.

[0096] For example, the four communication terminals with IP addresses "1.2.1.3", "1.2.2.3", "1.3.1.3", and "1.3.2.3" respectively are shown to be transmitting image data indicated by image data IDs "RS001", "RS002", "RS003", and "RS004", respectively. Furthermore, the image types indicated by the source names of each communication terminal are "Video_Wide", "Video_Wide", "Video", and "Video", respectively indicating that the image types are "Special Image", "Special Image", "General Image", and "General Image". In this case, the special image is a 360-degree image.

[0097] Furthermore, data other than image data may also be managed in association with image data IDs. Examples of data other than image data include audio data and document data used during screen sharing.

[0098] (Imaging device management table) Figure 18 is a conceptual diagram showing the imaging device management table. This imaging device management table stores and manages the vendor ID and product ID of the GUID of the imaging device that can obtain the two hemispherical images that form the basis of the 360-degree spherical image. As GUIDs, for example, the vendor ID (VID) and product ID (PID) used for USB devices can be used. These vendor IDs and product IDs are stored in the communication terminals 3 and 4 from the time of factory shipment, but they may be added and stored after factory shipment.

[0099] (Designated area information management table) Figure 19 is a conceptual diagram showing a predetermined region information management table. In this predetermined region information management table, the IP address of the communication terminal that sends the captured image data, the IP address of the communication terminal to which the captured image data is sent, and predetermined region information indicating the predetermined region image currently displayed on the communication terminal to which the captured image data is sent are stored and managed in association. Note that the communication terminal to which the captured image data is sent is also the communication terminal that sends the predetermined region information. As shown in Figures 6 and 7, the predetermined region information is a conversion table for converting the captured image into an image of the predetermined region T in that captured image (predetermined region image). Note that the IP address is an example of destination information, and destination information includes MAC (Media Access Control) address, terminal ID (Identification) to identify the communication terminal, etc. Also, here the IP address is represented in a simplified form of IPv4 address. The IP address may also be IPv6.

[0100] For example, in the first to third rows of the predetermined area information management table in Figure 19, it is managed that if the IP address of communication terminal 3 is "1.2.1.3", the captured image data transmitted from communication terminal 3 was sent via the communication management system 50 to communication terminal 4 with IP address "1.2.2.3", communication terminal 2 with IP address "1.3.1.3", and communication terminal 1 with IP address "1.3.2.3". Furthermore, it is managed that communication terminal 4 is the source communication terminal for predetermined area information (r=10, θ=20, φ=30). Similarly, it is managed that communication terminal 2 is the source communication terminal for predetermined area information (r=20, θ=30, φ=40). Also, it is managed that communication terminal 1 is the source communication terminal for predetermined area information (r=30, θ=40, φ=50). In other words, it is managed how communication terminals 1, 2, and 4 are viewing the 360-degree image using predetermined area information.

[0101] Furthermore, if the transmitting / receiving unit 31c receives new predetermined area information that includes the same set of IP addresses as the IP address of the communication terminal that is the source of the captured image data and the IP address of the communication terminal to which the captured image data is sent, the storage / reading unit 39c overwrites the predetermined area information that is already being managed with the newly received predetermined area information.

[0102] (Functional configuration of communication terminal 1) The transmitting and receiving unit 91 of the communication terminal 1 is mainly realized by the processing of the long-distance communication circuit 911 and CPU 901 shown in Figure 14, and transmits and receives various data (or information) with the communication management system 50 via the communication network 100.

[0103] The reception unit 92 is primarily implemented by processing via the touch panel 921 and the CPU 901, and accepts various selections or inputs from the user. In addition to the touch panel 921, other input means such as voice input may also be used.

[0104] The image and sound processing unit 93 is implemented by instructions from the CPU 901 shown in Figure 14 and performs image processing on the image data obtained when the camera 912 captures a subject. In addition, after the user's voice is converted into an audio signal by the microphone 914, the image and sound processing unit 93 performs audio processing on the audio data related to this audio signal.

[0105] Furthermore, the image / sound processing unit 93 performs image processing on image data received from other communication terminals based on image type information such as source name, in order for the display control unit 94 to display an image on the display 917. The image / sound processing unit 93 also outputs audio signals related to sound data received from other communication terminals via the communication management system 50 to the speaker 915, causing the speaker 915 to output sound.

[0106] The display control unit 94 is mainly implemented by the processing of the display 917 and the CPU 901, and controls the display 917 to display various images, characters, etc.

[0107] The determination unit 95 is mainly implemented by the processing of the CPU 901 and, for example, determines the type of image related to the image data received from the imaging device 5a.

[0108] The creation unit 96 is mainly implemented by the CPU 901 and, based on the determination by the judgment unit 95 that it is a general image or a special image (in this case, a 360-degree image), creates a source name, which is an example of image type information, according to the naming rules described above. For example, if the judgment unit 95 determines that it is a general image, the creation unit 96 creates a source name "Video" that indicates it is a general image. On the other hand, if the judgment unit 95 determines that it is a special image, the creation unit 96 creates a source name "Video_Wide" that indicates it is a special image.

[0109] The calculation unit 97 is mainly implemented by the CPU 901 and calculates the position (position information) of a predetermined region T1 in the captured image based on predetermined region information indicating a predetermined region T and predetermined region information received from other communication terminals by the transmitting / receiving unit 91. The image when the entire captured image is displayed is also called the "whole image".

[0110] The communication unit 98 is mainly realized by the processing of the short-range communication circuit 919, antenna 919a, and CPU 901, and can communicate with the communication unit 18a of the imaging device 5a using short-range wireless technologies such as NFC, Bluetooth (registered trademark), and Wi-Fi. Although the communication unit 98 and the transceiver unit 91 have been described as having separate communication units, they may also be in a shared configuration.

[0111] The memory and read unit 99 is mainly implemented by the processing of the CPU 901 shown in Figure 14, and stores various data (or information) in the memory unit 9000 and reads various data (or information) from the memory unit 9000.

[0112] <Functional Configuration of Communication Terminal 2> Next, the functional configuration of communication terminal 2 will be described in detail. Communication terminal 2 basically has the same functions as communication terminal 1. That is, as shown in Figure 16, communication terminal 2 has a transmitting / receiving unit 71, a receiving unit 72, an image / sound processing unit 73, a display control unit 74, a judgment unit 75, a creation unit 76, a calculation unit 77, a communication unit 78, and a storage / reading unit 79. However, since each of these functions achieves the same functions as the transmitting / receiving unit 91, receiving unit 92, image / sound processing unit 93, display control unit 94, judgment unit 95, creation unit 96, calculation unit 97, communication unit 98, and storage / reading unit 99 in communication terminal 1, their descriptions will be omitted.

[0113] Furthermore, the communication terminal 2 has a storage unit 7000 constructed from ROM 902, RAM 903, and EEPROM 904 as shown in Figure 14. This storage unit 7000 contains an image type management DB 7001, an imaging device management DB 7002, and a predetermined area management DB 7003. Note that the image type management DB 7001, imaging device management DB 7002, and predetermined area management DB 7003 have the same data structure as the image type management DB 9001, imaging device management DB 9002, and predetermined area management DB 9003 in the communication terminal 1, respectively, so their explanations are omitted.

[0114] <Functional configuration of communication terminal 3> Next, the functional configuration of communication terminal 3 will be described. Communication terminal 3 basically has the same functions as communication terminal 1. That is, as shown in Figure 15, communication terminal 3 has a transmitting / receiving unit 31c, a receiving unit 32c, an image / sound processing unit 33c, a display control unit 34c, a judgment unit 35c, a creation unit 36c, a calculation unit 37c, a communication unit 38c, and a storage / reading unit 39c. However, since these achieve the same functions as the transmitting / receiving unit 91, receiving unit 92, image / sound processing unit 93, display control unit 94, judgment unit 95, creation unit 96, calculation unit 97, communication unit 98, storage / reading unit 99, and storage unit 9000 in communication terminal 1, their explanation will be omitted.

[0115] Furthermore, the communication terminal 3 has a storage unit 3000c constructed from the ROM 302, RAM 303, and SSD 305 shown in Figure 12. The storage unit 3000c contains an image type management DB 3001c, an imaging device management DB 3002c, and a predetermined area management DB 3003c. However, since these perform the same functions as the image type management DB 9001, imaging device management DB 9002, and predetermined area management DB 9003 in the communication terminal 1, their explanation will be omitted.

[0116] <Functional configuration of communication terminal 4> Next, the functional configuration of communication terminal 4 will be described. Communication terminal 4 basically has the same functions as communication terminal 1. That is, as shown in Figure 16, communication terminal 4 has a transmitting / receiving unit 31d, a receiving unit 32d, an image / sound processing unit 33d, a display control unit 34d, a judgment unit 35d, a creation unit 36d, a calculation unit 37d, a communication unit 38d, a storage / reading unit 39d, and a storage unit 3000d. However, since these achieve the same functions as the transmitting / receiving unit 91, receiving unit 92, image / sound processing unit 93, display control unit 94, judgment unit 95, creation unit 96, calculation unit 97, communication unit 98, storage / reading unit 99, and storage unit 9000 in communication terminal 1, their explanation will be omitted.

[0117] Furthermore, the communication terminal 4 has a storage unit 3000d constructed from a ROM 302, RAM 303, and SSD 305 as shown in Figure 12. The storage unit 3000d contains an image type management DB 3001d, an imaging device management DB 3002d, and a predetermined area management DB 3003d. However, since these perform the same functions as the image type management DB 9001, imaging device management DB 9002, and predetermined area management DB 9003 in the communication terminal 1, their explanation will be omitted.

[0118] <Functional Configuration of Communication Management System> Next, the functional configuration of the communication management system 50 will be described in detail. The communication management system 50 includes a transmitting / receiving unit 51, a determination unit 55, a generation unit 56, and a storage / reading unit 59. Each of these units is a function or means realized by any of the components shown in Figure 13 operating according to instructions from the CPU 501 that follow the program for the communication management system 50 deployed from the HD 504 onto the RAM 503.

[0119] Furthermore, the communication management system 50 has a storage unit 5000 constructed from the RAM 503 and HD 504 shown in Figure 13. This storage unit 5000 contains a session management DB 5001, an image type management DB 5002, and a predetermined area management DB 5003. Of these, the session management DB 5001 is composed of the session management table shown in Figure 20. The image type management DB 5002 is composed of the image type management table shown in Figure 21. The predetermined area management DB 5003 is composed of the predetermined area information management table shown in Figure 22.

[0120] (Session management table) Figure 20 is a conceptual diagram showing the session management table. In this session management table, the session ID and the IP address of the participating communication terminal are stored and managed in association. The session ID is an example of session identification information used to identify a communication session that enables video calls, and is generated for each virtual conference room. The session ID is also managed by communication terminals 1 to 4 and is used when selecting a communication session for each communication terminal. The IP address of the participating communication terminal indicates the IP address of the communication terminal that joined the virtual conference room indicated by the associated session ID.

[0121] (Image type management table) Figure 21 is a conceptual diagram showing the image type management table. The image type management table shown in Figure 21 manages the same session IDs as those managed in the image type management table shown in Figure 17, in addition to the information managed in the session management table. Here, it is shown that three communication terminals with IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1.3" are participating in the virtual conference room indicated by the same session ID "se101". Note that the communication management system 50 manages the same image data ID, source terminal IP address, and image type information for communication terminals such as communication terminals 1-4 (video conferencing terminals) in order to send image type information etc. to the communication terminal already in a video call and the newly joined communication terminal when a new communication terminal enters the virtual conference room. This eliminates the need to send and receive image type information etc. between the communication terminal already in a video call and the newly joined communication terminal.

[0122] (Designated area information management table) Figure 22 is a conceptual diagram showing the predetermined area information management table. This predetermined area information management table basically has the same data structure as the predetermined area information management table shown in Figure 19. However, as will be described later, the transmitting / receiving unit 51 transmits the latest predetermined area information to each communication terminal at regular intervals (e.g., 30 seconds), so all predetermined area information received by the transmitting / receiving unit 51 is stored without being deleted until the predetermined area information is transmitted at regular intervals. In Figure 22, newer predetermined area information is managed at a higher level.

[0123] (Functional configuration of each function in the communication management system) The transmitting and receiving unit 51 of the communication management system 50 is mainly realized by the processing of the network I / F 509 and CPU 501 shown in Figure 13, and transmits and receives various data (or information) with communication terminals 1 to 4 via the communication network 100.

[0124] The decision unit 55 is mainly implemented by the CPU 501 and performs various decisions.

[0125] The generation unit 56 is mainly implemented by the CPU 501 and generates image data IDs.

[0126] The storage and reading unit 59 is mainly implemented by the HDD 505 and CPU 501 shown in Figure 13, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000.

[0127] <Processing of participation> First, we will explain the process of joining a specific communication session using Figures 23 and 24. Figure 23 is a sequence diagram showing the process of joining a specific communication session. Figure 24 is a diagram showing the screen for selecting a communication session (virtual meeting room).

[0128] First, when the imager 8 at base A performs an operation on the communication terminal 1 to display the selection screen for communication sessions (virtual meeting rooms), the reception unit 92 receives the operation to display the selection screen, and the display control unit 94 displays the selection screen shown in Figure 24 on the display 917 of the communication terminal 1 (step S21). This selection screen displays selection buttons b1, b2, b3, etc., which indicate the virtual meeting rooms R1, R2, R3, etc. that are to be selected. In addition, each selection button b1, etc., is associated with each session ID.

[0129] Here, when the imager 8 selects the desired selection button for the virtual conference room (in this case, selection button b1), the reception unit 92 accepts the selection of the communication session (step S22). The selected conference room is, for example, a conference room predetermined for conducting a property viewing.

[0130] The transmitting / receiving unit 91 then sends a request to join the virtual conference room to the communication management system 50 (step S23). This request includes a session ID indicating the communication session selected in step S22, and the IP address of the requesting terminal, communication terminal 1. As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the request to join.

[0131] Next, the storage / reading unit 59 of the communication management system 50 performs the process of joining the communication session by adding the IP address received in step S23 to the field of the participating terminal IP address in the record with the same session ID as the session ID received in step S23 in the session management DB 5001 (step S24).

[0132] The transmitting / receiving unit 51 then sends a participation request response to the communication terminal 1 (step S25). This participation request response includes the session ID received in step S23 and the participation processing result. As a result, the transmitting / receiving unit 91 of the communication terminal 1 receives the participation request response. The following describes the case where the participation process is successful.

[0133] Furthermore, if communication terminals 2-4 at locations B-D also make a similar participation request and each selects the same conference room, communication terminals 1-4 will join the same session and be able to make video calls with each other. Also, the method of joining a session shown in Figures 23 and 24 is just one example; the imager 8 may also establish a session by specifying the identification information of the destination communication terminal or user. In this case, the session is established when the destination communication terminal or user responds to the call.

[0134] <About posture information> Next, the attitude information of the imaging device 5a will be explained using Figures 25 to 27. Figure 25 is an example of a diagram illustrating the coordinate axes of the imaging device 5a, and Figure 26 is an example of a diagram illustrating the reference attitude of the imaging device 5a. The reference attitude of the imaging device 5a is the initial state of the imaging device's attitude.

[0135] As shown in Figure 25, the longitudinal direction of the imaging device 5a is defined as the Z-axis, the direction passing through the two lenses from the plane with the shutter button SB to the plane without it is defined as the Y-axis, and the width direction of the imaging device 5a is defined as the X-axis. These axes move together with the imaging device 5a. The imaging device 5a can rotate around the X-axis, Y-axis, and Z-axis, respectively. Let α be the rotation angle around the X-axis, β be the rotation angle around the Y-axis, and γ be the rotation angle around the Z-axis.

[0136] As shown in Figure 26, when the imager 8 points to the object 180, the longitudinal direction (upper side) is directed towards the object 180. If the upright position is considered the initial state, then the state in Figure 26 is α=-90 degrees, β=0 degrees, γ=0 degrees.

[0137] Figure 27 is a diagram illustrating the values ​​detected by the acceleration / direction sensor 118. If the values ​​detected by the acceleration / direction sensor 118 are (ax, ay, az), then as shown in Figure 27(a), the main body rotates α around the X axis, and the angle is α = atan 2 (ay,-az) ay 2 +az 2 ≧ threshR 2 = 0 ay 2 +az 2 < threshR 2 It can be expressed as follows: α takes values ​​of -π < α ≤ π.

[0138] Rotation around the Y-axis can be represented as shown in Figure 27(b). Rotation around the Z-axis can be represented as shown in Figure 27(c). However, to improve responsiveness, it is recommended to use the gyro sensor 119. The values ​​(gα, gβ, gγ) of the gyro sensor 119 output are equivalent to angular velocity [rad / sec]. The rotation angles (attitude information) α, β, γ of the imaging device 5a are obtained using the values ​​of the gyro sensor 119. α(n+1) = α(n) + k*gα*dt β(n+1) = β(n) + k*gβ*dt γ(n+1)=γ(n)+k*gγ*dt It can be calculated as follows.

[0139] However, the initial orientation (α(0), β(0), γ(0)) = (α0, 0, 0). k is a coefficient related to the sensitivity of the gyro sensor 119. The initial value is k = 1.0. If you want to avoid the effects of hand shake, you can set k = 0.5 to give the effect of a low-pass filter. The (α, β, γ) obtained in this way is the attitude information. Note that γ, the rotation angle around the Z axis, does not affect the direction it points, so it does not need to be calculated.

[0140] The rotation angle α around the X axis can be obtained correctly even if its initial value is not zero, thanks to the signal from the acceleration / direction sensor 118. Furthermore, the rotation angle γ around the Z axis does not affect the direction being pointed to. In contrast, the rotation angle β around the Y axis is simply set to 0 initially; therefore, if the initial state when the power is turned on is tilted, for example, the correct rotation angle β cannot be obtained. For this reason, when the imager 8 points to an object, it is preferable to activate the imaging device 5a and initialize the rotation angles α, β, and γ by pressing and holding a predetermined button.

[0141] Alternatively, the rotation angle β around the Y-axis can be initially determined from the output of the acceleration / direction sensor 118, similar to the rotation angle α around the X-axis.

[0142] <How the coordinates of the point of interest indicated by the imager are processed by zenith correction> Figure 28 illustrates how the coordinates of the point of interest pointed out by the imager 8 are processed by zenith correction. Figure 28(a) shows the imager 8 pointing to the object. The imager 8 is pointing to the doll 190 with the imaging device 5a.

[0143] For example, if we define the coordinates of the point of interest as the pixels or a certain range in which the subject in the longitudinal direction (upper side) of the imaging device 5a is captured, the image captured in the state shown in Figure 28(a) will look like Figure 28(b). Figure 28(b) is an equirectangular image in which the entire spherical image is represented using an equirectangular projection. This equirectangular image is "not zenith corrected". When creating an equirectangular image, the subject in the longitudinal direction (upper side) is transformed by being expanded horizontally at the upper end. That is, without zenith correction, the subject in the longitudinal direction (upper side) is placed at the upper end, and because it is an equirectangular image, the subject at the upper end spreads horizontally. When zenith correction is not performed, the coordinates of the point of interest will always be in the same location in the equirectangular image.

[0144] Regardless of the orientation of the imaging device 5a, the imaging device 5a displays objects above in real space on the upper side of the image and objects below on the lower side of the image, thus correcting the image to the zenith based on the orientation information. Figure 28(c) shows a "zenith-corrected" equirectangular image. Since the direction and amount of zenith correction are determined by the orientation information, the original point of interest coordinates defined by the coordinate transformation information used to rotate the image for zenith correction will be rotated to obtain the point of interest coordinates in the equirectangular image. In Figure 28(c), the point of interest coordinates are enclosed in a circle 191, and the doll 190 in Figure 28(a) is visible within the circle 191. In other words, the object pointed to by the imager 8 can be identified by the point of interest coordinates.

[0145] <About the functions of the image processing unit 16a> Figure 29 shows the main functional blocks of the image processing unit 16a of the imaging device 5a. The image processing unit 16a includes an image acquisition unit 202, an image stitching unit 204, a zenith correction unit 206, a full-sphere image generation unit 208, an image compression unit 210, a point of interest definition unit 194, a point of interest conversion unit 196, and a point of interest designation determination unit 198.

[0146] The image acquisition unit 202 controls the two image sensors 130A and 130B described above and acquires images from each of them. In the case of still images, two images for one frame are acquired at the moment the shutter is pressed. In the case of video, consecutive frames are captured sequentially, and two images are acquired for each frame. The images captured by each image sensor 130 are generally fisheye images that capture a hemisphere of the entire sphere and constitute a partial image of the full sphere image. Hereafter, the images captured by each image sensor 130 may be referred to as partial images.

[0147] The stitching processing unit 204 detects the joining position between the two acquired partial images and executes a process to stitch the two partial images together. In the joining position detection process, for each frame, a process is performed to detect the amount of positional shift for each of the multiple corresponding points in the overlapping region that exists between the multiple partial images.

[0148] The point of interest definition unit 194 defines and stores the coordinates (x,y) of the point of interest within the planar image. The point of interest coordinates are not set by the user (e.g., the imager 8), but are predetermined (fixed) during manufacturing, design, or shipment. There may be multiple point of interest coordinates. In this case, the imager 8 selects the point of interest coordinates they wish to use by operating the imaging device 5a or the communication terminal 1.

[0149] The point of focus designation determination unit 198 determines whether the imager 8 is attempting to designate a point of focus. For example, while the button on the main body of the imaging device 5a is pressed, the point of focus designation determination unit 198 determines that the imager 8 has designated a point of focus. When the button is not pressed, it determines that no point of focus has been designated.

[0150] The zenith correction unit 206 performs a correction process based on the attitude information acquired by the attitude information acquisition unit 15a to ensure that the zenith direction of the generated 360-degree image matches a predetermined reference direction. Specifically, it corrects the conversion table, which will be described later. Here, the predetermined reference direction is typically the vertical direction, which is the direction in which gravitational acceleration acts. By correcting the zenith direction of the 360-degree image to match the vertical direction (heavenly direction), it becomes possible to prevent users from experiencing discomfort such as motion sickness, especially when changing the field of view during viewing of moving images. The conversion table will be explained in Figure 30.

[0151] The point of interest conversion unit 196 converts the coordinates of the point of interest (x,y) in planar coordinates to the coordinates of the point of interest (θ,φ) in spherical coordinates, based on the point of interest definition unit 194, the point of interest specification determination unit 198, and the corrected conversion table. In other words, it determines the coordinates of the point of interest after zenith correction based on attitude information. The point of interest conversion unit 196 sets the coordinates of the point of interest (θ,φ) in spherical coordinates in the point of interest information.

[0152] [Table 1] Table 1 shows an example of point of interest information. Point of interest information includes the coordinates (θ,φ) of the point of interest in a spherical coordinate system and whether or not the point of interest is specified. The point of interest specification determination unit 198 sets "Yes" to "Point of Interest Specification" if the point of interest is specified, and "No" to "Point of Interest Specification" if it is not specified. Note that the point of interest itself is always included in the point of interest information.

[0153] The 360-degree image generation unit 208 performs the process of generating a 360-degree image from the two captured partial images, with the processing results of the point of interest transformation unit 196 reflected. In the embodiment described, a transformation table is also used to generate a 360-degree image from the two partial images. The 360-degree image generation unit 208 generates a 360-degree image from the two partial images using the corrected transformation table. By processing in this way, the processing load required to obtain the final 360-degree image can be reduced.

[0154] However, the system is not limited to the embodiments described above. It is also possible to configure the system to generate a full-sphere image by stitching together two partial images, and then apply zenith correction processing to the generated full-sphere image to produce a zenith-corrected full-sphere image.

[0155] The image compression unit 210 includes a still image compression block and, when capturing still images, compresses the captured images into image data in a predetermined still image format such as JPEG (Joint Photographic Experts Group). When capturing video, the image compression unit 210 compresses the captured consecutive image frames into image data in a predetermined video format. The video compression format is not particularly limited, but various video compression formats such as H.264 / MPEG-4 AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), Motion JPEG, and Motion JPEG2000 can be cited. The generated image data is transmitted to other locations B to D by the transmission unit 211. The transmission unit 211 corresponds to the communication unit 18a of the imaging device 5a and the transmitting / receiving unit 91 of the communication terminal 1.

[0156] <Functions of the display control unit of each communication terminal> When a 360-degree image is received from another location, communication terminals 1-4 display the 360-degree image.

[0157] As shown in Figure 29, the display control unit 74 of the communication terminal 2 includes an image unfolding unit 212, a point of interest determination unit 214, an image rotation unit 216, a cropping processing unit 220, an enlargement / letterbox processing unit 222, and an output unit 224. Although Figure 29 describes the functions of the display control unit 74, the display control unit 34c of the communication terminal 3, the display control unit 34d of the communication terminal 4, and the display control unit 94 of the communication terminal 1 are similar.

[0158] The image processing unit 212 reads the 360-degree image transmitted from the imaging device 5a and acquires the 360-degree image. The acquired 360-degree image is then processed into memory.

[0159] The image rotation unit 216 rotates the 360-degree image according to the point of interest determined by the point of interest transformation unit 196. As a result, the coordinates of the point of interest move to the center of the equirectangular image. The rotation process is explained in Figure 32.

[0160] The cropping processing unit 220 crops a portion (center) of the rotated 360-degree image to generate a cropped image. Cropping means extracting a certain part. It can also be called trimming. In a preferred embodiment, the cropping processing unit 220 is a process that crops the central portion of the converted 360-degree image, thereby extracting an image corresponding to a certain size portion centered on the point of interest in the 360-degree image.

[0161] In the embodiments described, the cropping processing unit 220 is described as having the function of cropping a portion of an image to generate a cropped image. However, in other embodiments, the functions of the cropping processing unit 220 may include not only the function of cropping a portion of an image to generate a cropped image, but also the function of reducing the resolution.

[0162] The enlargement / letterboxing processing unit 222 performs an enlargement process on the image cropped by the cropping processing unit 220 according to the resolution and aspect ratio of the video output device, such as a display or projector, and adds black bars to the top and bottom of the cropped image portion to generate the display image. The output unit 224 outputs (displays) the display image processed and generated by the enlargement / letterboxing processing unit 222 via the video output interface 129. Note that the processing by the enlargement / letterboxing processing unit 222 can be omitted if the cropped image corresponds to the resolution and aspect ratio of the video output device.

[0163] In the case of still images, the video output processing by the above-mentioned functional units (image rotation unit 216, cropping processing unit 220, enlargement / letterboxing processing unit 222, and output unit 224) is repeatedly performed on the same 360-degree image at least each time the point of interest changes, typically at predetermined intervals, and the displayed image is updated according to the point of interest at that time. In the case of video, the video output processing by the above-mentioned functional units is repeatedly performed typically for each frame, and the displayed image is updated.

[0164] The imager 8 can change the point of focus by tilting the imaging device 5a forward, backward, left, or right, or by rotating it, using the initial position of the imaging device 5a facing directly upward as a baseline. The user can then view the displayed 360-degree spherical image corresponding to the changed point of focus.

[0165] <About the conversion table> Figure 30 illustrates a transformation table and the transformation from a planar coordinate system to a spherical coordinate system. Figure 30(a) illustrates the transformation table used by the imaging device 5a according to this embodiment. The transformation table defines the projection from a partial image represented in the planar coordinate system (x, y) of the image sensor to an equirectangular image in the spherical coordinate system (θ, φ) (hereinafter referred to as the corrected image). For each fisheye lens, the transformation table holds information relating the coordinate values ​​(θ, φ) of the corrected image to the coordinate values ​​(x, y) of the uncorrected partial image mapped to those coordinate values ​​(θ, φ), for all coordinate values ​​(θ, φ). In the example in Figure 30, the angle represented by one pixel is 1 / 10 degree in both the φ direction and the θ direction, and the transformation table has information showing a 3600 × 1800 correspondence for each fisheye lens. The original transformation table can be calculated and formatted in advance by the manufacturer or the like after correcting for distortion from an ideal lens model.

[0166] As shown in Figure 30(b), the transformation table projects the coordinates of the point of interest 199A, expressed in the plane coordinate system (x, y), onto the spherical coordinate system (θ, φ). The coordinates of the point of interest 199A, expressed in the plane coordinate system (x, y), are always in a constant direction from the perspective of the imaging device 5a. Therefore, if the transformation table is fixed, the coordinates of the point of interest 199B in the spherical coordinate system (θ, φ) will also be constant.

[0167] However, since the orientation of the imaging device 5a changes depending on how the user holds it, the vertical orientation of the corrected image will not match the vertical orientation of the real space unless the conversion table is corrected according to the orientation information. For this reason, in zenith correction, this conversion table is corrected according to the orientation information.

[0168] <Correction of conversion table> Figure 31 illustrates the correction of the transformation table based on attitude information. Here, the three-dimensional Cartesian coordinates before the coordinate transformation are denoted as (x1, y1, z1), and the spherical coordinates of (x1, y1, z1) are denoted as (θ1, φ1). The three-dimensional Cartesian coordinates after the coordinate transformation are denoted as (x2, y2, z2), and the spherical coordinates of (x2, y2, z2) are denoted as (θ2, φ2).

[0169] In the correction process for the conversion table, equations (1) to (6) are used to convert from spherical coordinates (θ1, φ1) to spherical coordinates (θ2, φ2).

[0170]

number

[0171] Next, using the attitude information α, β, and γ of the imaging device 5a that points to the target, the three-dimensional Cartesian coordinate system (x1, y1, z1) is transformed into a three-dimensional Cartesian coordinate system (x2, y2, z2) according to equation (4). Equation (4) means that the transformed coordinate system is obtained by rotating the original coordinate system by α around the x-axis, β around the y-axis, and γ around the z-axis.

[0172] Finally, using equations (5) and (6), a conversion is performed to return the converted three-dimensional orthogonal coordinates (x2, y2, z2) to spherical coordinates (θ2, φ2). If (θ1, φ1) are the coordinates of the spherical coordinate system before correction of the conversion table, the spherical coordinates of the conversion table can be corrected to (θ2, φ2) according to the attitude information of the imaging device 5a.

[0173] If the point-of-interest conversion unit 196 converts the point-of-interest coordinates in the plane coordinate system to the point-of-interest coordinates in the spherical coordinate system using the corrected conversion table, it becomes the point-of-interest coordinates in the spherical coordinate system after zenith correction.

[0174] <Image rotation process by the image rotation unit> Subsequently, using FIG. 32, the rotation process by the image rotation unit 216 will be described. FIG. 32 is a diagram schematically showing the image rotation process of the image rotation unit 216. Image rotation refers to a process of moving the point-of-interest coordinates to the center of the image on the spherical coordinates. · The new coordinates after rotation are (θ N , φ N ) · The point-of-interest coordinates are (θ0, φ0) · Any coordinates to be rotated are (θ, φ) Let them be.

[0175] The point-of-interest coordinates can be moved to the center of the image by equations (7) and (8). θ N = θ - θ0 + 180° (θ - θ0 ≤ 180°) θ - θ0 - 180° (θ - θ0 > 180°)... (7) φ N = φ - φ0 + 90° (φ - φ0 ≤ 90°) φ - φ0 - 90° (φ - φ0 > 90°)... (8) Equations (7) and (8) rotate (move) an arbitrary pixel by the differences in the θ direction and the φ direction between the center point and the point-of-interest coordinates.

[0176] <Transmission procedure of the all-sky image> Next, using Figure 33, we will explain the process by which the 360-degree image and sound data obtained at base A are transmitted to the other communication terminals 2-4 via the communication management system 50. Figure 33 is a sequence diagram showing the communication process of 360-degree images and sound data in a video call.

[0177] First, the communication unit 18a of the imaging device 5a transmits a 360-degree image obtained by imaging a subject or landscape, and sound data obtained by collecting sound, to the communication unit 98 of the communication terminal 1 (step S101). The imaging device 5a attaches the point of interest information regardless of the judgment result of the point of interest determination unit 198. As a result, the communication unit 98 of the communication terminal 1 receives the 360-degree image and sound data.

[0178] Next, the transmitting / receiving unit 91 of the communication terminal 1 transmits the 360-degree image, sound data, and point of interest information sent from the imaging device 5a to the communication management system 50 (step S102). This transmission includes an image data ID to identify the image data being transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 50 receives the 360-degree image (image data ID), sound data, and point of interest information.

[0179] Next, the transmitting / receiving unit 51 of the communication management system 50 transmits a 360-degree image (image data ID), sound data, and point of interest information to communication terminals 2 to 4, which are participating in the same video call as communication terminal 1 (steps S103, S104, S105). Each of these transmissions includes an image data ID to identify the 360-degree image being transmitted. As a result, the transmitting / receiving unit 71 of communication terminal 2, the transmitting / receiving unit 31c of communication terminal 3, and the transmitting / receiving unit 31d of communication terminal 4 receive the 360-degree image (image data ID), sound data, and point of interest information, respectively. Generally, communication terminal 1 also receives and displays a 360-degree image of its own location from the communication management system 50, but this is omitted in Figure 33.

[0180] Next, using Figure 34, we will explain in detail the process by which the imaging device 5a generates a 360-degree image and the process by which the communication terminals 2-4 display the 360-degree image. Figure 34 is an example of a flowchart showing the process by which the imaging device 5a generates a 360-degree image and the process by which the communication terminals 2-4 display the 360-degree image.

[0181] Figure 34(a) shows the processing of the imaging device 5a. Note that the communication terminal 1 has been omitted.

[0182] The coordinates of the point of interest in the planar coordinate system are defined in the point of interest definition section 194 (S200). The definition is set statically in advance, as explained in Figure 29.

[0183] During imaging, the focus point designation determination unit 198 determines whether the imager 8 has designated a focus point based on sensor information (S201). Sensor information refers to information detecting whether or not the button on the imaging device 5a is pressed. If the button is pressed, it is determined that a focus point has been designated. If a focus point has been designated, the focus point designation in the focus point information of Table 1 is set to "Yes".

[0184] The image acquisition unit 202 acquires images from the image sensors 130A and 130B (S202).

[0185] Next, the stitching processing unit 204 detects the stitching position in the overlapping region between the two acquired partial images and reflects the result of the stitching position detection in the conversion table (S203). By reflecting the result of the stitching position detection, the conversion table shown in Figure 30(a) is modified so that the coordinate values ​​(x, y) of the partial image, which reflect the correction of the stitching position, are associated with the coordinate values ​​(θ, φ) of the corrected image.

[0186] The zenith correction unit 206 corrects the conversion table based on the attitude information (S204). In other words, it performs zenith correction.

[0187] Next, the point of interest transformation unit 196 uses a transformation table to transform the point of interest coordinates defined by the point of interest definition unit 194 into point of interest coordinates in a spherical coordinate system (S205). The spherical coordinate system coordinates obtained from the transformation are then set in the point of interest information in Table 1. Note that even if it is determined in step S201 that no point of interest has been specified, the point of interest coordinates are still set; however, if it is determined that no point of interest has been specified, the point of interest coordinates do not need to be set.

[0188] The 360-degree image and point of interest information generated by the above process are transmitted to communication terminals 2-4, as explained in Figure 33.

[0189] Figure 34(b) is a flowchart showing the processing of the display control unit 34 for communication terminals 2 to 4.

[0190] First, the image rotation unit 216 determines whether or not there is a designated point of interest in the point of interest information attached to the 360-degree image (S210).

[0191] If a point of interest is specified, the image rotation unit 216 rotates the image so that the point of interest is in the center of the equirectangular image (S211). The point of interest is then forcibly displayed, regardless of the previously displayed predetermined region image.

[0192] Next, the cropping processing unit 220 crops the central portion of the full-sphere image to generate a cropped image (S212). The area to be cropped is predetermined. Using Figure 8 as an example, the center point CP is the coordinate of the point of interest, and the field of view α and distance f are predetermined.

[0193] If no point of interest is specified, the user can rotate the 360-degree image at will. In the case of a still image, the predetermined region T that the user last rotated is displayed, and in the case of a video, the display of the predetermined region T that the user last rotated is maintained. The cropping processing unit 220 crops the predetermined region T determined by the user's operation (S213).

[0194] The enlargement / letterboxing processing unit 222 enlarges the cropped image and adds black bars according to the resolution and aspect ratio of the output destination to generate the display image (S214).

[0195] The output unit 224 outputs the generated display image to the display I / F 317 and the display 917 (S215).

[0196] <Example of display on a communication terminal> Figure 35 shows an example of the video display screen 250 displayed on the display 917 of the communication terminal 2 at site B. The left display area (layout number "1") of the video display screen 250 displays a panoramic image of site A, and the upper right display area (layout number "2") displays a panoramic image of site C. Furthermore, the middle right display area (layout number "3") of the video display screen 250 displays an image of site D, and the lower right display area (layout number "4") displays an image of site B (the local site). The display area with layout number "1" is the main display area, and the display areas with layout numbers "2", "3", and "4" are secondary display areas. The images in the main display area and the secondary display areas can be changed at each communication terminal. Normally, at each site, the main display area displays an image of the site where the central person in the video call is located.

[0197] Furthermore, a 360-degree icon 192 is displayed in the display areas of layout number 1 and layout number 2. This indicates that the image displayed in the display area is a 360-degree image, and the user can change the predetermined area T. In addition, a point of interest icon 193 is displayed in the display area of ​​layout number 1. The point of interest icon 193 is displayed by the display control unit 74 when a point of interest is specified in the point of interest information. The point of interest icon 193 indicates that a point of interest has been specified. This allows the user to understand that a point of interest is currently being displayed and that the predetermined area T cannot be changed.

[0198] The inability to change the predetermined area T means that the reception unit 72 does not accept the change of the display area, or that it accepts the change of the display area, but when the user 9b stops the operation to change the display area, the point of interest conversion unit 196 converts the point of interest, and the extraction processing unit 220 extracts it from the image and displays it again. For example, the user 9b can display any display area only while dragging (clicking the mouse) or while swiping and keeping their finger touching it.

[0199] Furthermore, user 9b may be able to stop the display of the point of interest. In this case, for example, a designated button may be provided. Pressing this button once will stop the display of the point of interest, and user 9b can display any designated area T. Pressing this button again will automatically display the point of interest.

[0200] As shown in Figure 35, since video communication is possible at multiple locations, even if users 9b and 9d know each other but cannot go to the real estate office together for a viewing, a realistic viewing experience becomes possible. For example, if users 9b and 9d are a married couple, and one can go to the office but the other cannot, they can view the property from their respective locations and share their impressions on the spot.

[0201] <Summary> As described above, in this embodiment, the image communication system 10 has a predetermined coordinate of the point of interest defined in the imaging device 5a. By pointing to the object that the imager wants to show using this coordinate of the point of interest, the imager 8 can make users at other locations focus on the point of interest in real time using only the imaging device.

[0202] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0203] For example, in this embodiment, a property viewing was used as an example, but the application of the image communication system 10 is not limited to this. For example, it can be applied when referring to a local object such as an exhibition, trade show, factory tour, sightseeing, or inspection.

[0204] Furthermore, in this embodiment, a human pointed to the object with the imaging device 5a, but a machine, robot, or animal may also point to the object. For example, if the imaging device 5a is fixed in the direction of movement of a mobile machine or the like, the object in the direction of movement can always be displayed, and if necessary, the image can be rotated to check the surrounding situation. In this case, it would be desirable for the receiving side of the 360-degree image to be able to switch whether or not to display the coordinates of the point of interest.

[0205] Furthermore, in this embodiment, a 360-degree spherical image was used as an example to display some points of interest, but a 360-degree spherical image does not necessarily have to capture the entire 360 ​​degrees. For example, it may capture only a hemisphere, or it may capture 360 ​​degrees only in the horizontal direction. Alternatively, a planar image with a large number of pixels that cannot fit on the display may also be used.

[0206] Furthermore, the configuration examples in Figures 15, 16, and 29 are divided according to their main functions to facilitate understanding of the processing performed by the image communication system 10. The present invention is not limited by the way the processing units are divided or their names. The processing of the image communication system 10 can be further divided into many more processing units depending on the processing content. Also, one processing unit can be divided to include even more processing.

[0207] Multiple communication management systems 50 may exist, and the functions of the communication management system 50 may be distributed across multiple servers. There may also be relay devices that relay image data and sound data.

[0208] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above. [Explanation of Symbols]

[0209] 1-4 Communication terminals 5. Imaging device 8. Imager 10 Image communication system [Prior art documents] [Patent Documents]

[0210] [Patent Document 1] Japanese Patent Publication No. 2016-167739

Claims

1. An imaging device for capturing images, A point of interest definition unit that defines points of interest in the aforementioned image, A posture information acquisition unit that acquires posture information of the imaging device, A point of focus conversion unit converts the points of focus defined by the point of focus definition unit according to the posture information acquired by the posture information acquisition unit, An extraction processing unit extracts a predetermined region containing the point of interest converted by the point of interest conversion unit from the image, An imaging device characterized by having the following features.

2. The system includes a zenith correction unit that performs processing related to zenith correction of the image using the posture information acquired by the posture information acquisition unit, The imaging apparatus according to claim 1, characterized in that the point of interest conversion unit converts the point of interest defined by the point of interest definition unit using the result of the zenith correction by the zenith correction unit.

3. The zenith correction unit applies zenith correction using the attitude information to a conversion table that converts the coordinate values ​​of the image sensor to coordinate values ​​in a spherical coordinate system. The imaging apparatus according to claim 2, characterized in that the point of interest conversion unit uses the point of interest defined by the point of interest definition unit as the coordinate value of the image sensor, and converts the point of interest defined by the point of interest definition unit into a spherical coordinate system coordinate value using a conversion table on which zenith correction has been applied using the attitude information.

4. The aforementioned zenith correction corrects the image so that the upper part of real space is captured on the upper side, regardless of the orientation of the imaging device. The imaging apparatus according to claim 2 or 3, characterized in that the coordinate values ​​of the image sensor in which the object pointed to by the upper side in the longitudinal direction of the imaging apparatus is captured are defined as the point of interest by the point of interest definition unit.

5. The imaging device has a unit that determines whether or not the user has specified a point of interest when taking an image, If the unit that determines whether a point of interest has been designated determines that the imager has designated a point of interest, it attaches a note to the image indicating that a point of interest has been designated. The imaging apparatus according to any one of claims 1 to 4, characterized in that, if the image is accompanied by a statement indicating that the aforementioned points of interest are specified, the points of interest conversion unit converts the points of interest defined by the points of interest definition unit.

6. An image communication system in which an imaging device captures an image and a communication terminal displays the image on its screen, A point of interest definition unit that defines points of interest in the aforementioned image, A posture information acquisition unit that acquires posture information of the imaging device, A point of focus conversion unit converts the points of focus defined by the point of focus definition unit according to the posture information acquired by the posture information acquisition unit, An extraction processing unit extracts a predetermined region containing the point of interest converted by the point of interest conversion unit from the image, An output unit that displays a predetermined region including the point of interest extracted by the extraction processing unit on the screen, An image communication system characterized by having the following features.

7. The output unit has a receiving unit that accepts changes to the display area of ​​the image displayed, When the point of interest converted by the point of interest conversion unit is extracted from the image by the extraction processing unit and displayed by the output unit, The reception unit either does not accept the change of the display area, or The image communication system according to claim 6, which accepts changes to the display area, but when it stops accepting changes to the display area, the point of interest converted by the point of interest conversion unit is cut out from the image by the cutting processing unit and displayed by the output unit.

8. The image communication system according to claim 6 or 7, characterized in that the output unit displays a statement indicating that a point of interest has been designated when such a statement is attached to the image.

9. An image processing method performed by an image communication system having an imaging device for capturing images and a communication terminal for displaying the images on a screen, The posture information acquisition unit acquires posture information of the imaging device, The steps include: a step in which the focus point conversion unit converts the focus point defined by the focus point definition unit according to the posture information acquired by the posture information acquisition unit, The cutting processing unit performs the step of cutting out a predetermined region from the image that includes the point of interest converted by the point of interest conversion unit, The output unit performs the step of displaying a predetermined region including the point of interest extracted by the extraction processing unit on the screen, An image processing method characterized by having the following features.

10. An imaging device that captures images, A point of interest definition unit that defines points of interest in the aforementioned image, A posture information acquisition unit that acquires posture information of the imaging device, A point of focus conversion unit converts the points of focus defined by the point of focus definition unit according to the posture information acquired by the posture information acquisition unit, An extraction processing unit extracts a predetermined region from the image that includes the point of interest converted by the point of interest conversion unit. A program designed to function as such.

11. A communication terminal that displays the image acquired from an imaging device on a screen, The point of interest defined by the point of interest definition unit, which is converted according to the posture information of the imaging device acquired by the posture information acquisition unit, and the receiving unit that receives the image, A cutting processing unit that cuts out a predetermined region including the point of interest received by the receiving unit from the image, An output unit that displays a predetermined region including the point of interest extracted by the extraction processing unit on the screen. A program designed to function as such.

Images