Communication management system, communication system, communication management method, and program
The communication management system addresses the issue of distorted panoramic images by using image type information to adapt processing and display, ensuring clear visibility across different imaging devices.
Patent Information
- Application Number
- JP2026200051
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2017-06-20
- Filing Date
- 2026-09-03
- Publication Date
- 2026-11-19
AI Technical Summary
Existing communication systems struggle with displaying images from different types of imaging devices, such as general and special imaging devices, resulting in distorted or difficult-to-view panoramic images due to the mismatch in image types.
A communication management system that includes receiving means for image type information, generating image data identification, and transmitting appropriate image data based on the received type information to ensure proper processing and display on the receiving terminal.
Enables clear and easy-to-view image display on communication terminals by adapting image processing according to the type of imaging device used, whether general or special, thereby improving the visibility of panoramic images.
Smart Images

Figure 2026182990000001_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present invention relates to a communication management system, a communication system, a communication management method, and a program. BACKGROUND ART
[0002] Conference systems that conduct remote conferences with remote locations via communication networks such as the Internet have become widespread. In this conference system, in a conference room where one of the parties such as attendees holding the remote conference is present, the communication terminal of the remote conference system is used to capture and collect images of the conference room such as the conference parties and audio such as speeches, convert these into digital data, and transmit the digital data to the counterpart communication terminal. As a result, images can be displayed on the display of the counterpart conference room and audio can be output via a speaker to perform a video call, so a technology for holding conferences between remote locations in a state close to an actual conference is known (see Patent Document 1).
[0003] On the other hand, a known technology connects an imaging device capable of acquiring an omnidirectional panoramic image in real time to a communication terminal, distributes the omnidirectional panoramic image acquired from this imaging device to each counterpart communication terminal, and sequentially performs image conversion processing on the omnidirectional panoramic image into a rectangular planar image for display on a display or the like at each counterpart communication terminal (see Patent Document 2). SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION
[0004] However, when the types of imaging devices used at each site are different, such as general imaging devices that obtain rectangular planar images and special imaging devices that obtain two hemispherical images that serve as the source of omnidirectional panoramic images, the following problems arise on the communication terminal side that has received the image data.
[0005] In other words, if a communication terminal that receives image data acquired from a special imaging device displays it along with a planar image, it will display two hemispherical images that form the basis of a full-sphere panoramic image. As shown in Figure 23(a), user A1, etc., will be displayed facing downwards within the spherical image, resulting in an image that is very difficult to see. On the other hand, if a communication terminal that receives image data acquired from a general imaging device attempts to create a full-sphere panoramic image, the image will be corrupted and will be very difficult to see. [Means for solving the problem]
[0006] The invention according to claim 1 is a communication management system characterized by comprising: receiving means for receiving image type information indicating the type of image from a first communication terminal; generating means for generating image data identification information for identifying image data to be transmitted from the first communication terminal to a second communication terminal based on the reception of the image type information; and transmitting means for transmitting the generated image data identification information to the first communication terminal and transmitting the generated image data identification information and the received image type information to the second communication terminal. [Effects of the Invention]
[0007] According to the present invention, the second communication terminal that receives the image data can perform image processing according to the image type information, thereby achieving the effect of displaying an image that is easy to see. [Brief explanation of the drawing]
[0008] [Figure 1] Figure 1(a) is a left side view of the imaging device, Figure 1(b) is a front view of the imaging device, and Figure 1(c) is a top view of the imaging device. [Figure 2] Figure 2 is an illustrative diagram showing how the imaging device is used. [Figure 3] Figure 3(a) shows a hemispherical image (front) taken with the imaging device, Figure 3(b) shows a hemispherical image (back) taken with the imaging device, and Figure 3(c) shows an image represented using the Mercator projection. [Figure 4]Figure 4(a) is a conceptual diagram showing the state of covering a sphere with a Mercator image, and Figure 4(b) is a diagram showing a panoramic image of the entire sphere. [Figure 5] Figure 5 shows the positions of a virtual camera and a predetermined region when a full-spherical panoramic image is treated as a three-dimensional sphere. [Figure 6] Figure 6(a) is a stereoscopic perspective view of Figure 5, and Figure 6(b) shows the state in which an image of a predetermined area is displayed on the display of a communication terminal. [Figure 7] Figure 7 shows the relationship between information about a predetermined region and an image of that region. [Figure 8] Figure 8 is a schematic diagram of an image communication system according to an embodiment of the present invention. [Figure 9] Figure 9 is a hardware configuration diagram of the imaging device. [Figure 10] Figure 10 is a hardware configuration diagram of a video conferencing terminal. [Figure 11] Figure 11 is a hardware configuration diagram of the communication management system and the PC. [Figure 12] Figure 12 is a hardware configuration diagram of a smartphone. [Figure 13] Figure 13 is a functional block diagram of the image communication system. [Figure 14] Figure 14 is a conceptual diagram showing the image type management table. [Figure 15] Figure 15 is a conceptual diagram showing the imaging equipment management table. [Figure 16] Figure 16 is a conceptual diagram showing the session management table. [Figure 17] Figure 17 is a conceptual diagram showing the image type management table. [Figure 18] Figure 18 is a sequence diagram showing the process of joining a specific communication session. [Figure 19] Figure 19 shows the screen for selecting a communication session (virtual meeting room). [Figure 20] Figure 20 is a sequence diagram showing the image type information management process. [Figure 21]FIG. 21 is a sequence diagram illustrating communication processing of image data in a video call. [Figure 22] FIG. 22(a) is a diagram illustrating a case where the imaging device is not used, and FIG. 22(b) is a diagram illustrating a case where the imaging device is used. [Figure 23] FIG. 23 is a diagram illustrating a display example of a display at site B, where FIG. 23(a) is a diagram illustrating a case where image data transmitted from the imaging devices 1a and 1b is displayed as-is without generating an omnidirectional spherical panoramic image and a predetermined area image, and FIG. 23(b) is a diagram illustrating a case where an omnidirectional spherical panoramic image and a predetermined area image are generated from the image data transmitted from the imaging devices 1a and 1b. [Figure 24] FIG. 24 is a sequence diagram illustrating another example of the management processing of image type information. DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION
[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0010] <<Overview of Embodiment>> <Method for Generating Omnidirectional Spherical Panoramic Image> A method for generating an omnidirectional spherical panoramic image will be described with reference to FIG. 1 to FIG. 7.
[0011] First, the external appearance of the imaging device 1 will be described with reference to FIG. 1. The imaging device 1 is a digital camera for obtaining captured images that serve as the source of an omnidirectional spherical (360°) panoramic image. Note that FIG. 1(a) is a left side view of the imaging device, FIG. 1(b) is a front view of the imaging device, and FIG. 1(c) is a plan view of the imaging device.
[0012] As shown in Figure 1(a), the imaging device 1 is small enough to be held in one hand. Also, as shown in Figures 1(a), 1(b), and 1(c), the imaging device 1 has an image sensor 103a on the front side and an image sensor 103b on the rear side. These image sensors 103a and 103b are used in conjunction with an optical component (for example, a fisheye lens 102a or 102b, which will be described later) that can capture hemispherical images (angle of view of 180° or more). Also, as shown in Figure 1(b), an operating section 115, such as a shutter button, is provided on the side of the imaging device 1 opposite to the front side.
[0013] Next, the usage of the imaging device 1 will be explained using Figure 2. Figure 2 is an illustrative diagram of the imaging device in use. As shown in Figure 2, the imaging device 1 is used, for example, by a user holding it in their hand to photograph subjects around the user. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image sensors 103a and 103b shown in Figure 1.
[0014] Next, using Figures 3 and 4, we will outline the process from the image captured by the imaging device 1 to the creation of a full-sphere panoramic image. Figure 3(a) shows the hemispherical image (front) captured by the imaging device, Figure 3(b) shows the hemispherical image (back) captured by the imaging device, and Figure 3(c) shows the image represented by the Mercator projection (hereinafter referred to as the "Mercator image"). Figure 4(a) is a conceptual diagram showing the state in which the sphere is covered by the Mercator image, and Figure 4(b) shows the full-sphere panoramic image.
[0015] As shown in Figure 3(a), the image obtained by the image sensor 103a becomes a curved hemispherical image (front side) by the fisheye lens 102a described later. Similarly, as shown in Figure 3(b), the image obtained by the image sensor 103b becomes a curved hemispherical image (rear side) by the fisheye lens 102b described later. The hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) are then combined by the imaging device 1 to create a Mercator image, as shown in Figure 3(c).
[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is superimposed to cover the sphere, as shown in Figure 4(a), creating a full-sphere panoramic image as shown in Figure 4(b). In this way, the full-sphere panoramic image is represented as an image where the Mercator image is oriented towards the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. The full-sphere panoramic image can be a still image or a video.
[0017] As described above, a full-sphere panoramic image is an image pasted over a sphere, which can cause discomfort to the human eye. Therefore, by displaying a predetermined area of the full-sphere panoramic image (hereinafter referred to as the "predetermined area image") as a flat image with less curvature, it is possible to display it in a way that does not cause discomfort to the human eye. This will be explained using Figures 5 and 6.
[0018] Figure 5 shows the positions of the virtual camera and predetermined region when the 360-degree panoramic image is treated as a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of the user viewing the 360-degree panoramic image, which is displayed as a three-dimensional sphere. Figure 6(a) is a stereoscopic perspective view of Figure 5, and Figure 6(b) shows the predetermined region image as it appears on a display. In Figure 6(a), the 360-degree panoramic image shown in Figure 4 is represented as a three-dimensional sphere CS. If the generated 360-degree panoramic image is a sphere CS, then, as shown in Figure 5, the virtual camera IC is located outside the 360-degree panoramic image. The predetermined region T in the 360-degree panoramic image is the shooting area of the virtual camera IC and is identified by predetermined region information that shows the position coordinates (x(rH), y(rV), angle α(angle)) including the field of view of the virtual camera IC in the three-dimensional virtual space including the 360-degree panoramic image. Zooming in a predetermined region T can be represented by widening or narrowing the field of view α (arc). Alternatively, zooming in a predetermined region T can also be represented by moving the virtual camera IC closer to or further away from the 360-degree panoramic image.
[0019] Then, the predetermined region image, which is an image of the predetermined region T shown in Figure 6(a), is displayed on a predetermined display as an image of the shooting area of the virtual camera IC, as shown in Figure 6(b). The image shown in Figure 6(b) is the predetermined region image represented by the predetermined region information that has been initially set (default). Note that the predetermined region information may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined region T, rather than the position coordinates of the virtual camera IC. In the following explanation, we will use the position coordinates (x(rH), y(rV), and field of view α(angle)) of the virtual camera IC.
[0020] Next, we will explain the relationship between the predetermined region information and the image of the predetermined region T using Figure 7. Figure 7 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 7, the center point CP of the predetermined region information becomes the (x,y) parameter when the diagonal field of view of the predetermined region T, represented by the field of view α of the virtual camera IC, is 2L. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined region T and the center point CP (2L is the diagonal). And, in Figure 7, the trigonometric function shown in the following equation (1) generally holds true.
[0021] Lf = tan(α / 2) ... (Equation 1)
[0022] <Overview of the image communication system> Next, the general configuration of the image communication system of this embodiment will be explained using Figure 8. Figure 8 is a schematic diagram of the configuration of the image communication system of this embodiment.
[0023] As shown in Figure 8, the image communication system of this embodiment consists of a camera 1a, 1b, a video conferencing terminal 3, a communication management system 5, a PC (Personal Computer) 7, a camera 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection method of the communication network 100 may be either wireless or wired.
[0024] Of these, the shooting devices 1a and 1b are special digital cameras used to capture subjects, landscapes, etc., and obtain two hemispherical images that form the basis of a full-sphere panoramic image, as described above. On the other hand, the shooting device 8 is a general-purpose digital camera used to capture subjects, landscapes, etc., and obtain a general planar image.
[0025] The video conferencing terminal 3 is a terminal dedicated to video conferencing and displays the video call image on the display 4 via a wired cable such as a USB (Universal Serial Bus) cable. Normally, the video conferencing terminal 3 captures images with the camera 312 described later, but when connected via a wired cable to the cradle 2a to which the camera 1a is attached, the camera 1a takes priority, and two hemispherical images that form the basis of a full-spherical panoramic image can be obtained. When using a wired cable, the cradle 2a not only handles communication between the camera 1a and the video conferencing terminal 3, but also supplies power to the camera 1a and supports the camera 1a. Here, the camera 1a, cradle 2a, video conferencing terminal 3, and display 4 are all located at the same location, location A. Also, four users A1, A2, A3, and A4 are participating in the video call at location A.
[0026] The communication management system 5 manages communication between the video conferencing terminal 3, PC 7, and smartphone 9, and manages the types of image data transmitted and received (general images and special images). In this case, special images are 360-degree panoramic images. The communication management system 5 is installed in service companies that provide video communication services. The communication management system 5 may be constructed using a single computer, or it may be constructed using multiple computers, each part (function, means, or storage unit) divided and assigned as appropriate.
[0027] PC7 becomes capable of video calls when the camera 8 is attached. In this case, both PC7 and camera 8 are located at the same location, location C. Additionally, one user, C1, is participating in the video call at location C.
[0028] Smartphone 9 displays the video call image on a display 917 located on its own device, as described later. Normally, smartphone 9 captures images using a CMOS (Complementary Metal Oxide Semiconductor) sensor 905 located on its own device, as described later. However, it can also acquire two hemispherical image data, which form the basis of the full-sphere panoramic image obtained by the imaging device 1b, using wireless communication technologies such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark). When using wireless communication technology, cradle 2b only serves to supply power to imaging device 1b and support it. In this case, imaging device 1b, cradle 2b, and smartphone 9 are all located at the same location, location B. Two users, B1 and B2, are participating in the video call at location B.
[0029] Furthermore, the video conferencing terminal 3, PC 7, and smartphone 9 are examples of communication terminals. Each communication terminal has OpenGL ES installed and can create predetermined region information that indicates a portion of a full-spherical panoramic image, or create predetermined region images from full-spherical panoramic images sent from other communication terminals.
[0030] Note that the arrangement of terminals, devices, and users shown in Figure 8 is just one example, and other arrangements are possible. For example, at base C, a shooting device capable of capturing 360-degree panoramic images may be used instead of shooting device 8. Communication terminals also include digital televisions, smartwatches, car navigation devices, etc. Furthermore, from now on, any shooting device among shooting devices 1a and 1b will be referred to as "shooting device 1".
[0031] <<Hardware configuration of the embodiment>> Next, the hardware configuration of the camera 1, video conferencing terminal 3, communication management system 5, PC 7, and smartphone 9 of this embodiment will be described in detail with reference to Figures 9 to 12. Note that the camera 8 is a general-purpose camera, so a detailed explanation will be omitted.
[0032] <Hardware configuration of imaging device 1> First, Figure 9 will be used to explain the hardware configuration of the imaging device 1. Figure 9 is a hardware configuration diagram of the imaging device 1. In the following, the imaging device 1 will be assumed to be a 360-degree (omnidirectional) imaging device using two image sensors, but it may have more than two image sensors. Furthermore, it is not necessarily required to be a device dedicated to omnidirectional imaging; a standard digital camera or smartphone can be fitted with an aftermarket omnidirectional imaging unit to achieve essentially the same functionality as the imaging device 1.
[0033] As shown in Figure 9, the imaging device 1 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network interface 116, a communication unit 117, an electronic compass 118, and an antenna 117a.
[0034] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b, each having a field of view of 180° or more for forming hemispherical images, and two image sensors 103a and 103b, each corresponding to the wide-angle lens. The image sensors 103a and 103b include image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert the optical image from the fisheye lenses 102a and 102b into electrical signal image data and output it, timing generation circuits that generate horizontal or vertical synchronization signals and pixel clocks for these image sensors, and a group of registers in which various commands and parameters necessary for the operation of these image sensors are set.
[0035] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are connected separately from the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via bus 110. Furthermore, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are also connected to bus 110.
[0036] The image processing unit 104 receives image data output from image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create Mercator image data as shown in Figure 3(c).
[0037] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 acting as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.
[0038] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 1, there may also be functions to display a preview or video on a display (for example, the display of the video conferencing terminal 3). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).
[0039] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 1 is not provided with a display unit, but a display unit may be provided.
[0040] Microphone 108 converts sound into sound (signal) data. Sound processing unit 109 receives the sound data output from microphone 108 through the I / F bus and performs predetermined processing on the sound data.
[0041] The CPU 111 controls the overall operation of the imaging device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress and processed Mercator image data from the image processing unit 104.
[0042] The control unit 115 is a collective term for various operation buttons, a power switch, a shutter button, and a touch panel that combines display and operation functions. The user inputs various shooting modes and shooting conditions by operating the operation buttons.
[0043] The network interface 116 is a general term for interface circuits (such as USB interfaces) that connect to external media such as SD cards or personal computers. The network interface 116 can be wireless or wired. The Mercator image data stored in the DRAM 114 is recorded to external media via this network interface 116, or transmitted to external devices such as the video conferencing terminal 3 via the network interface 116 as needed.
[0044] The communication unit 117 communicates with external devices such as the video conferencing terminal 3 via the antenna 117a provided on the imaging device 1, using short-range wireless technologies such as Wi-Fi or NFC (Near Field Communication). This communication unit 117 can also transmit Mercator image data to the external devices of the video conferencing terminal 3.
[0045] The electronic compass 118 calculates the orientation and tilt (roll angle) of the imaging device 1 from the Earth's magnetic field and outputs orientation and tilt information. This orientation and tilt information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was taken and the data size of the image data.
[0046] <Hardware configuration of video conferencing terminals> Next, the hardware configuration of the video conferencing terminal 3 will be explained using Figure 10. Figure 10 is a hardware configuration diagram of the video conferencing terminal. As shown in Figure 10, the video conferencing terminal 3 is equipped with a CPU 301, ROM 302, RAM 303, flash memory 304, SSD 305, media I / F 307, operation buttons 308, power switch 309, bus line 310, network I / F 311, camera 312, image sensor I / F 313, microphone 314, speaker 315, sound input / output I / F 316, display I / F 317, external device connection I / F 318, short-range communication circuit 319, and antenna 319a of the short-range communication circuit 319.
[0047] Of these components, the CPU 301 controls the overall operation of the video conferencing terminal 3. The ROM 302 stores programs used to drive the CPU 301, such as the IPL (Initial Program Loader). The RAM 303 is used as the work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and sound data. The SSD (Solid State Drive) 305 controls the reading or writing of various data to the flash memory 304 according to the control of the CPU 301. Note that an HDD may be used instead of the SSD. The media I / F 307 controls the reading or writing (storage) of data to the recording media 306, such as the flash memory. The operation button 308 is used to select a destination on the video conferencing terminal 3, etc. The power switch 309 is a switch for turning the power of the video conferencing terminal 3 ON / OFF.
[0048] Furthermore, the network I / F 311 is an interface for data communication using a communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 301. The image sensor I / F 313 is a circuit that controls the driving of the camera 312. The microphone 314 is a type of built-in sound collection means that inputs sound. The sound input / output I / F 316 is a circuit that processes the input and output of sound signals between the microphone 314 and the speaker 315 according to the control of the CPU 301. The display I / F 317 is a circuit that transmits image data to an external display 4 according to the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) or Bluetooth (registered trademark).
[0049] Furthermore, the bus line 310 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 301 shown in Figure 10.
[0050] Display 4 is a type of display device composed of liquid crystal or organic EL that displays images of the subject and operation icons. Display 4 is connected to display I / F 317 by cable 4c. This cable 4c may be an analog RGB (VGA) signal cable, a component video cable, or an HDMI (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signal cable. HDMI is a registered trademark.
[0051] The camera 312 includes a lens and a solid-state image sensor that converts light into electric charge to create an image (video) of the subject. CMOS sensors and CCD sensors are used as the solid-state image sensor. External devices such as external cameras, external microphones, and external speakers can be connected to the external device connection I / F 318 via USB (Universal Serial Bus) cables, etc. When an external camera is connected, it is driven in priority over the built-in camera 312, according to the control of the CPU 301. Similarly, when an external microphone or external speaker is connected, it is driven in priority over the built-in microphone 314 and built-in speaker 315, respectively, according to the control of the CPU 301.
[0052] Furthermore, the recording medium 306 is configured to be detachable from the video conferencing terminal 3. In addition, any non-volatile memory that reads or writes data according to the control of the CPU 301 may be used, not limited to flash memory 304, but also including EEPROM (Electrically Erasable and Programmable ROM), etc.
[0053] <Communication management system, PC hardware configuration> Next, we will explain the hardware configuration of the communication management system 5 and PC 7 using Figure 11. Figure 11 is a hardware configuration diagram of the communication management system and the PC. Since both the communication management system 5 and PC 7 are computers with the same configuration, we will explain the configuration of the communication management system 5 below, and omit the explanation of the configuration of PC 7.
[0054] The communication management system 5 includes a CPU 501 that controls the overall operation of the communication management system 5, a ROM 502 that stores programs used to drive the CPU 501 such as an IPL, a RAM 503 used as the work area of the CPU 501, an HD 504 that stores various data such as programs for the communication management system 5, an HDD (Hard Disk Drive) 505 that controls the reading or writing of various data to the HD 504 according to the control of the CPU 501, a media drive 507 that controls the reading or writing (storage) of data to a recording medium 506 such as flash memory, a display 508 that displays various information such as cursors, menus, windows, characters, or images, a network I / F 509 for data communication using the communication network 100, a keyboard 511 with multiple keys for inputting characters, numbers, and various instructions, a mouse 512 for selecting and executing various instructions, selecting processing targets, moving the cursor, etc., and a CD-RW (Compact) as an example of a removable recording medium. The system includes a CD-RW drive 514 that controls the reading or writing of various data to the Disc-ReWritable) 513, and bus lines 510 such as an address bus and a data bus for electrically connecting the above components as shown in Figure 11.
[0055] <Smartphone Hardware Configuration> Next, the hardware of the smartphone will be described using Figure 12. Figure 12 is a hardware configuration diagram of the smartphone. As shown in Figure 12, the smartphone 9 is equipped with a CPU 901, ROM 902, RAM 903, EEPROM 904, CMOS sensor 905, accelerometer / compass sensor 906, media I / F 908, and GPS receiver 909.
[0056] Of these, the CPU 901 controls the overall operation of the smartphone 9. The ROM 902 stores programs used to drive the CPU 901, such as the IPL. The RAM 903 is used as the work area for the CPU 901. The EEPROM 904 reads or writes various data, such as smartphone programs, according to the control of the CPU 901. The CMOS (Complementary Metal Oxide Semiconductor) sensor 905 captures images of a subject (mainly a self-portrait) and obtains image data according to the control of the CPU 901. The acceleration / direction sensor 906 is a variety of sensors, such as an electronic magnetic compass, gyrocompass, and acceleration sensor that detect the Earth's magnetic field. The media I / F 908 controls the reading or writing (storage) of data to or from the recording medium 907, such as flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.
[0057] The smartphone 9 also includes a long-range communication circuit 911, a camera 912, an image sensor interface 913, a microphone 914, a speaker 915, an audio input / output interface 916, a display 917, an external device connection interface 918, a short-range communication circuit 919, an antenna 919a for the short-range communication circuit 919, and a touch panel 921.
[0058] Of these, the long-range communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 901. The image sensor interface 913 is a circuit that controls the driving of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs sound. The sound input / output interface 916 is a circuit that processes the input and output of sound signals between the microphone 914 and the speaker 915 according to the control of the CPU 901. The display 917 is a type of display means such as a liquid crystal or organic EL that displays images of the subject and various icons. The external device connection interface 918 is an interface for connecting various external devices. The short-range communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means that allows the user to operate the smartphone 9 by pressing the display 917.
[0059] Furthermore, the smartphone 9 is equipped with a bus line 910. The bus line 910 is an address bus, data bus, etc., for electrically connecting various components such as the CPU 901.
[0060] Furthermore, recording media such as CD-ROMs on which the above programs are stored, as well as hard drives on which these programs are stored, may be provided domestically or internationally as program products.
[0061] <<Embodiment Functional Configuration>> Next, the functional configuration of this embodiment will be described using Figures 13 to 17. Figure 13 is a functional block diagram of the imaging devices 1a and 1b, the video conferencing terminal 3, the communication management system 5, the PC 7, and the smartphone 9, which constitute a part of the image communication system of this embodiment.
[0062] <Functional configuration of imaging device 1a> As shown in Figure 13, the imaging device 1a includes a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a storage / reading unit 19a. Each of these units is a function or means realized by any of the components shown in Figure 9 operating according to instructions from the CPU 111 that follow a program for the imaging device deployed on the SRAM 113 to the DRAM 114.
[0063] Furthermore, the imaging device 1a has a storage unit 1000a constructed from ROM 112, SRAM 113, and DRAM 114 as shown in Figure 9. The storage unit 1000a stores the GUID (Globally Unique Identifier) of the device itself.
[0064] (Functional configuration of each part of the imaging device 1a) Next, the functional configurations of the imaging device 1a will be explained in more detail using Figures 9 and 13.
[0065] The reception unit 12a of the imaging device 1a is mainly implemented by the operation unit 115 and the CPU 111 shown in Figure 9, and receives operation input from the user.
[0066] The imaging unit 13a is mainly realized by the imaging unit 101, image processing unit 104, and imaging control unit 105 shown in Figure 9, as well as the processing of the CPU 111, to capture images of landscapes, etc., and obtain captured image data.
[0067] The sound collection unit 14a is realized by the microphone 108 and sound processing unit 109 shown in Figure 9, as well as the processing of the CPU 111, and collects sounds from the surrounding area of the imaging device 1a.
[0068] The communication unit 18a is primarily implemented by the processing of the CPU 111 and can communicate with the communication unit 38 of the video conferencing terminal 3 using short-range wireless communication technologies such as NFC, Bluetooth, and WiFi.
[0069] The storage and reading unit 19a is mainly implemented by the processing of the CPU 111 shown in Figure 9, and stores various data (or information) in the storage unit 1000a and reads various data (or information) from the storage unit 1000a.
[0070] <Functional Configuration of Imaging Device 1b> The imaging device 1b includes a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, and a storage / reading unit 19b. These have the same functions as the reception unit 12a, imaging unit 13a, sound collection unit 14a, communication unit 18a, and storage / reading unit 19a in the imaging device 1a, so their explanation is omitted. The imaging device 1b also has a storage unit 1000b constructed from ROM 112, SRAM 113, and DRAM 114 as shown in Figure 9. The GUID of the device is stored in the storage unit 1000b.
[0071] <Functional Configuration of Video Conferencing Terminal 3> As shown in Figure 13, the video conferencing terminal 3 includes a transmitting / receiving unit 31, a receiving unit 32, an image / sound processing unit 33, a display control unit 34, a judgment unit 35, a creation unit 36, a communication unit 38, and a storage / reading unit 39. Each of these units is a function or means realized by any of the components shown in Figure 10 operating according to instructions from the CPU 301 that follow the program for the video conferencing terminal 3 deployed from the flash memory 304 onto the RAM 303.
[0072] Furthermore, the video conferencing terminal 3 has a storage unit 3000 constructed from ROM 302, RAM 303, and flash memory 304 as shown in Figure 10. This storage unit 3000 contains an image type management DB 3001 and an imaging device management DB 3002. Of these, the image type management DB 3001 is composed of the image type management table shown in Figure 14. The imaging device management DB 3002 is composed of the imaging device management table shown in Figure 15.
[0073] (Image type management table) Figure 14 is a conceptual diagram showing the image type management table. In this image type management table, the image data ID, the IP address (an example of the destination of the sending terminal), and the source name are stored and managed in association with each other. Of these, the image data ID is an example of image data identification information used to identify image data when performing video communication. Image data sent from the same sending terminal is assigned the same image data ID. This allows the receiving terminal (the receiving communication terminal) to identify the sending terminal of the received image data. The IP address of the sending terminal indicates the IP address of the communication terminal that sends the image data indicated by the associated image data ID. The source name is a name used to identify the imaging device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal, such as video conferencing terminal 3, according to a predetermined naming convention.
[0074] For example, the three communication terminals with IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1.3" are shown to be transmitting image data indicated by image data IDs "RS001", "RS002", and "RS003", respectively. Furthermore, the image types indicated by the source names of each communication terminal are "Video_Theta", "Video", and "Video_Theta", which indicate that the image types are "Special Image", "General Image", and "Special Image", respectively. In this case, the special image is a 360-degree panoramic image.
[0075] Furthermore, data other than image data may also be managed in association with image data IDs. Examples of data other than image data include audio data and document data used during screen sharing.
[0076] (Imaging equipment management table) Figure 15 is a conceptual diagram showing the imaging device management table. This imaging device management table stores and manages the vendor ID and product ID of the GUID of the imaging device that can obtain the two hemispherical images that form the basis of a full-spherical panoramic image. As GUIDs, for example, the vendor ID (VID) and product ID (PID) used in USB devices can be used. These vendor ID and product ID are stored from the time of factory shipment of communication terminals such as video conferencing terminals, but they may also be stored after factory shipment.
[0077] (Functional configuration of video conferencing terminal 3) Next, we will explain the functional configuration of the video conferencing terminal 3 in more detail using Figures 10 and 13.
[0078] The transmitting / receiving unit 31 of the video conferencing terminal 3 is mainly realized by the processing of the network interface 311 and CPU 301 shown in Figure 10, and transmits and receives various data (or information) with the communication management system 5 via the communication network 100.
[0079] The reception unit 32 is primarily implemented by the operation buttons 308 and the CPU 301, and accepts various selections or inputs from the user. In addition to the operation buttons 308, other input means such as a touch panel may also be used.
[0080] The image and sound processing unit 33 is implemented by instructions from the CPU 301 shown in Figure 10, and performs image processing on the image data obtained when the camera 312 captures a subject. In addition, the image and sound processing unit 33 performs audio processing on the sound data related to the audio signal after the user's voice is converted into an audio signal by the microphone 314.
[0081] Furthermore, the image / sound processing unit 33 performs image processing on image data received from other communication terminals based on image type information such as the source name, in order for the display control unit 34 to display the image on the display 4. Specifically, if the image type information indicates that it is a special image, the image / sound processing unit 33 creates a full-sphere panoramic image data by converting the image data (for example, the data of each hemispheric image as shown in Figures 3(a) and (b)) into a full-sphere panoramic image data as shown in Figure 4(b), and further creates a predetermined region image as shown in Figure 6(b). In addition, the image / sound processing unit 33 outputs an audio signal related to the audio data received from other communication terminals via the communication management system 5 to the speaker 315, and outputs audio from the speaker 315.
[0082] The display control unit 34 is mainly implemented by the processing of the display I / F 317 and the CPU 301, and controls the display of various images, characters, etc. on the display 4.
[0083] The determination unit 35 is mainly implemented by the processing of the CPU 301 and, for example, determines the type of image related to the image data received from the imaging device 1a.
[0084] The creation unit 36 is mainly implemented by the CPU 301 and, based on the determination by the judgment unit 35 that it is a general image or a special image (in this case, a full-sphere panoramic image), creates a source name, which is an example of image type information, according to the naming rules described above. For example, if the judgment unit 35 determines that it is a general image, the creation unit 36 creates a source name "Video" that indicates it is a general image. On the other hand, if the judgment unit 35 determines that it is a special image, the creation unit 36 creates a source name "Video_Theta" that indicates it is a special image.
[0085] The communication unit 38 is mainly realized by the processing of the short-range communication circuit 319, antenna 318a, and CPU 301, and can communicate with the communication unit 18a of the imaging device 1a using short-range wireless technologies such as NFC, Bluetooth, and WiFi. Although the communication unit 38 and the transmitting / receiving unit 31 have been described as having separate communication units, they may also be in a shared configuration.
[0086] The storage and reading unit 39 is mainly implemented by the processing of the CPU 301 shown in Figure 10, and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000.
[0087] <Functional Configuration of Communication Management System> Next, the functional configurations of the communication management system 5 will be described in detail using Figures 11 and 13. The communication management system 5 includes a transmitting / receiving unit 51, a determination unit 55, a generation unit 56, and a storage / reading unit 59. Each of these units is a function or means realized by any of the components shown in Figure 11 operating according to instructions from the CPU 501 that follow the program for the communication management system 5 deployed from the HD 504 onto the RAM 503.
[0088] Furthermore, the communication management system 5 has a storage unit 5000 constructed from the RAM 503 and HD 504 shown in Figure 11. This storage unit 5000 contains a session management DB 5001 and an image type management DB 5002. Of these, the session management DB 5001 is composed of the session management table shown in Figure 16. The image type management DB 5002 is composed of the image type management table shown in Figure 17.
[0089] (Session management table) Figure 16 is a conceptual diagram showing the session management table. In this session management table, the session ID and the IP address of the participating communication terminal are stored and managed in association. The session ID is an example of session identification information used to identify a communication session that enables video calls, and is generated for each virtual conference room. The session ID is also managed by each communication terminal, such as video conferencing terminal 3, and is used when selecting a communication session at each communication terminal. The IP address of the participating communication terminal indicates the IP address of the communication terminal that joined the virtual conference room indicated by the associated session ID.
[0090] (Image type management table) Figure 17 is a conceptual diagram showing the image type management table. The image type management table shown in Figure 17 manages the same session IDs as those managed in the image type management table shown in Figure 14, in addition to the information managed in the session management table. Here, it is shown that three communication terminals with IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1.3" are participating in the virtual conference room indicated by the same session ID "se101". Note that the reason why the communication management system 5 manages the same image data ID, source terminal IP address, and image type information for communication terminals such as the video conferencing terminal 3 is to send image type information etc. to the communication terminal already in a video call and the newly joined communication terminal when a new communication terminal enters the virtual conference room. This eliminates the need to send and receive image type information etc. between the communication terminal already in a video call and the newly joined communication terminal.
[0091] (Functional configuration of each function in the communication management system) Next, we will explain in detail the functional configurations of the communication management system 5 using Figures 11 and 13.
[0092] The transmission / reception unit 51 of the communication management system 5 is mainly implemented by the processing of the network I / F 509 and the CPU 501 shown in FIG. 11, and transmits and receives various data (or information) to and from the video conference terminal 3 or the PC 7 via the communication network 100.
[0093] The determination unit 55 is mainly implemented by the processing of the CPU 501, and performs various determinations.
[0094] The generation unit 56 is mainly implemented by the processing of the CPU 501, and generates an image data ID.
[0095] The storage / reading unit 59 is mainly implemented by the processing of the HDD 505 and the CPU 501 shown in FIG. 11, and stores various data (or information) in the storage unit 5000, and reads various data (or information) from the storage unit 5000.
[0096] <Functional Configuration of PC> Next, the functional configuration of the PC 7 will be described in detail with reference to FIG. 11 and FIG. 13. The PC 7 basically has the same functions as the video conference terminal 3. That is, as shown in FIG. 13, the PC 7 includes a transmission / reception unit 71, a reception unit 72, an image / audio processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a communication unit 78, and a storage / reading unit 79. Each of these units is a function or means implemented by any of the components shown in FIG. 11 operating according to an instruction from the CPU 501 in accordance with a program for the PC 7 expanded from the HD 504 onto the RAM 503.
[0097] In addition, the PC 7 has a storage unit 7000 constructed by the ROM 502, the RAM 503, and the HD 504 shown in FIG. 11. In this storage unit 7000, an image type management DB 7001 and an imaging device management DB 7002 are constructed. Note that the image type management DB 7001 and the imaging device management DB 7002 have the same configurations as the image type management DB 3001 and the imaging device management DB 3002, respectively, and thus descriptions thereof are omitted.
[0098] (The functional configuration of a PC) The transmit / receive unit 71 of PC7 is mainly implemented by the processing of the network interface 509 and CPU 501 shown in Figure 11, and performs the same functions as the transmit / receive unit 31.
[0099] The reception unit 72 is mainly implemented by the keyboard 511, mouse 512, and CPU 501, and performs the same functions as the reception unit 32. The image / sound processing unit 73 is mainly implemented by instructions from the CPU 501, and performs the same functions as the image / sound processing unit 33. The display control unit 74 is mainly implemented by the CPU 501, and performs the same functions as the display control unit 34. The judgment unit 75 is mainly implemented by the CPU 501, and performs the same functions as the judgment unit 35. The creation unit 76 is mainly implemented by the CPU 501, and performs the same functions as the creation unit 36. The communication unit 78 is mainly implemented by the CPU 501, and performs the same functions as the communication unit 38. The storage / reading unit 79 is implemented by the CPU 501, and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000.
[0100] <Smartphone Functionality Configuration> Next, the functional configuration of the smartphone 9 will be explained in detail using Figures 12 and 13. The smartphone 9 basically has the same functions as the video conferencing terminal 3. That is, as shown in Figure 13, the smartphone 9 has a transmitting / receiving unit 91, a receiving unit 92, an image / sound processing unit 93, a display control unit 94, a judgment unit 95, a creation unit 96, a communication unit 98, and a storage / reading unit 99. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 901 that follow the smartphone 9 program deployed from the EEPROM 904 onto the RAM 903.
[0101] Furthermore, the smartphone 9 has a storage unit 9000 constructed from ROM 902, RAM 903, and EEPROM 904 as shown in Figure 12. This storage unit 9000 contains an image type management DB 9001 and a shooting device management DB 9002. Note that the image type management DB 9001 and the shooting device management DB 9002 have the same configuration as the image type management DB 3001 and the shooting device management DB 3002, respectively, so their explanation is omitted.
[0102] (Smartphone function configurations) The transmitting / receiving unit 91 of the smartphone 9 is mainly realized by the processing of the long-distance communication circuit 911 and CPU 901 shown in Figure 12, and performs the same functions as the transmitting / receiving unit 31.
[0103] The reception unit 92 is mainly implemented by the processing of the touch panel 921 and the CPU 901, and performs the same functions as the reception unit 32.
[0104] The image and sound processing unit 93 is mainly implemented by instructions from the CPU 901 and performs the same functions as the image and sound processing unit 33.
[0105] The display control unit 94 is mainly implemented by the processing of the CPU 901 and performs the same functions as the display control unit 34.
[0106] The decision unit 95 is mainly implemented by the processing of the CPU 901 and performs the same functions as the decision unit 35.
[0107] The creation unit 96 is mainly implemented by the processing of the CPU 901 and performs the same functions as the creation unit 36.
[0108] The communication unit 98 is primarily implemented by the processing of the CPU 901 and performs the same functions as the communication unit 38.
[0109] The memory / read unit 99 is implemented by the CPU 901 and stores various data (or information) in the memory unit 9000 and reads various data (or information) from the memory unit 9000.
[0110] <<Processing or operation of the embodiment>> <Processing of participation> Next, the processing or operation of this embodiment will be explained using Figures 18 to 22. First, the process of joining a specific communication session will be explained using Figures 18 and 19. Figure 18 is a sequence diagram showing the process of joining a specific communication session. Figure 19 is a diagram showing the selection screen for a communication session (virtual conference room).
[0111] First, when a user at site A (for example, user A1) performs an operation on the video conferencing terminal 3 to display the selection screen for a communication session (virtual meeting room), the reception unit 32 receives the operation to display the selection screen, and the display control unit 34 displays the selection screen shown in Figure 19 on the display 4 (step S21). This selection screen displays selection buttons b1, b2, b3, etc., which indicate the virtual meeting rooms R1, R2, R3, etc. that are the target of selection. In addition, each selection button b1, etc., is associated with each session ID.
[0112] Here, when user A1 selects the desired selection button for the virtual meeting room (in this case, selection button b1), the reception unit 32 accepts the selection of the communication session (step S22). Then, the transmitting / receiving unit 31 sends a request to join the virtual meeting room to the communication management system 5 (step S23). This request includes the session ID indicating the communication session accepted in step S22, and the IP address of the requesting terminal, the video conferencing terminal 3. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request to join.
[0113] Next, the storage / reading unit 99 performs the process of joining the communication session by adding the IP address received in step S23 to the field of the participating terminal IP address in the record with the same session ID as the session ID received in step S23 in the session management DB 5001 (see Figure 16) (step S24). Then, the transmitting / receiving unit 51 sends a join request response to the video conferencing terminal 3 (step S25). This join request response includes the session ID received in step S23 and the result of the join process. As a result, the transmitting / receiving unit 31 of the video conferencing terminal 3 receives the join request response. The following describes the case when the join process is successful.
[0114] <Image type information management process> Next, we will explain the image type information management process using Figure 20. Figure 20 is a sequence diagram showing the image type information management process.
[0115] First, when a user at site A (for example, user A1) connects the USB cable of the cradle 2a, with the camera 1a attached, to the video conferencing terminal 3, the memory / read unit 19a of the camera 1a reads the GUID of its own device (camera 1a) stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of its own device to the communication unit 38 of the video conferencing terminal 3 (step S51). As a result, the communication unit 38 of the video conferencing terminal 3 receives the GUID of the camera 1a.
[0116] Next, the determination unit 35 of the video conferencing terminal 3 determines the image type (step S52) by checking whether the same vendor ID and product ID as those in the GUID received in step S51 are managed in the shooting device management DB 3002 (see Figure 15). Specifically, if the same vendor ID and product ID are managed in the shooting device management DB 3002, the determination unit 35 determines that the shooting device 1a is a shooting device that captures special images (in this case, a 360-degree panoramic image). Conversely, if the same vendor ID and product ID are not managed in the shooting device management DB 3002, the determination unit 35 determines that the shooting device 1a is a shooting device that captures general images.
[0117] Next, the storage / reading unit 39 stores the IP address of the source terminal (video conferencing terminal 3) and the image type information, which is the result of the determination made in step S52, in association with the image type management DB 3001 (see Figure 14) (step S53). In this state, the image data ID is not associated. The image type information is, for example, the source name determined according to a predetermined naming convention, or the image type (general image, special image).
[0118] Next, the transmitting / receiving unit 31 sends a request to the communication management system 5 for additional image type information (step S54). This request for additional image type information includes the IP address of the transmitting terminal, which is the local terminal, and the image type information, which were stored in step S53. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request for additional image type information.
[0119] Next, the storage and reading unit 59 of the communication management system 5 uses the IP address of the source terminal received in step S54 as a search key to search the session management DB 5001 (see Figure 16) and read the corresponding session ID (step S55).
[0120] Next, the generation unit 56 generates a unique image data ID (step S56). Then, the storage / reading unit 59 stores the session ID read in step S55, the image data ID generated in step S56, and the IP address and image type information of the source terminal received in step S54 as a new record in the image type management DB 5002 (see Figure 17) (step S57). Then, the transmitting / receiving unit 51 transmits the image data ID generated in step S56 to the video conferencing terminal 3. As a result, the transmitting / receiving unit 31 of the video conferencing terminal 3 receives the image data ID (step S58).
[0121] Next, the memory and retrieval unit 39 of the video conferencing terminal 3 stores the image data ID received in step S58 in the image type management DB 3001 (see Figure 14), associating it with the IP address of the source terminal, the local terminal (video conferencing terminal 3), and the image type information stored in step S53 (step S59).
[0122] Meanwhile, the transmitting / receiving unit 51 of the communication management system 5 sends an additional notification of image type information to another communication terminal, the smartphone 9 (step S60). This additional notification of image type information includes the image data ID generated in step S56, as well as the IP address of the source terminal (video conferencing terminal 3) and the image type information stored in step S53. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the additional notification of image type information. The destination of the transmitting / receiving unit 51 is another IP address associated with the same session ID as the IP address of the video conferencing terminal 3 in the session management DB 5001 (see Figure 16). In other words, the destination is another communication terminal that is in the same virtual conference room as the video conferencing terminal 3.
[0123] Next, the memory and retrieval unit 99 of the smartphone 9 stores the image data ID, the IP address of the sending terminal, and the image type information received in step S60 as a new record in the image type management DB 9001 (see Figure 14) (step S61). Similarly, additional communication regarding image type information is sent to another communication terminal, PC 7, and is also stored in the image type management DB 7001 (see Figure 14) on PC 7. Thus, each communication terminal can share the same information in its respective image type management DBs 3001, 7001, and 9001.
[0124] <Image data communication processing> Figures 21 to 23 will be used to explain the communication processing of image data in video calls. Figure 21 is a sequence diagram showing the communication processing of image data in video calls.
[0125] First, the communication unit 18a of the shooting device 1a transmits image data obtained by shooting a subject or landscape to the communication unit 38 of the video conferencing terminal 3 (step S101). In this case, since the shooting device 1a is a device that can obtain two hemispherical images that will become the basis of a full-sphere panoramic image, the image data consists of data from the two hemispherical images, as shown in Figures 3(a) and (b). As a result, the communication unit 38 of the video conferencing terminal 3 receives the image data.
[0126] Next, the transmitting / receiving unit 31 of the video conferencing terminal 3 transmits the image data sent from the imaging device 1a to the communication management system 5 (step S102). This transmission includes an image data ID to identify the image data to be transmitted. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the image data and the image data ID.
[0127] Next, the transmitting / receiving unit 51 of the communication management system 5 transmits the image data sent from the video conferencing terminal 3 to the smartphone 9 (step S103). This transmission includes an image data ID to identify the image data to be transmitted. As a result, the transmitting / receiving unit 51 of the smartphone 9 receives the image data and the image data ID.
[0128] Next, in the smartphone 9, the storage and reading unit 99 searches the image type management DB 9001 (see Figure 14) using the image data ID received in step S103 as a search key, and reads the corresponding image type information (source name) (step S104). If the image type information indicates that it is a special image (in this case, a 360-degree panoramic image) (Video_Theta), the image and sound processing unit 93 creates a 360-degree panoramic image from the image data received in step S103, and further creates a predetermined region image (step S105). In this case, based on the image type information indicating that it is a special image (Video_Theta), the image and sound processing unit 93 composites an icon 191, which indicates that it is a 360-degree panoramic image (described later), onto the predetermined region image.
[0129] Next, the display control unit 94 displays a predetermined area image including the icon 191 on the display 917 of the smartphone 9 (step S106). If the image type information indicates that it is a general image (Video), the image / sound processing unit 93 does not create a full-sphere panoramic image from the image data received in step S103, and the display control unit 94 displays a general image that does not include the icon 191.
[0130] Next, we will explain the state of the video call using Figure 22. Figure 22 is an illustrative diagram showing the state of the video call. Of these, Figure 22(a) shows the case when the camera 1a is not used, and Figure 22(b) shows the case when the camera 1a is used.
[0131] First, as shown in Figure 22(a), if the camera 312 (see Figure 10) pre-installed on the video conferencing terminal 3 is used instead of the shooting device 1a, the field of view is 125 degrees horizontally and 70 degrees vertically. Therefore, the video conferencing terminal 3 must be placed at the edge of the desk where each user A1, etc., is visible. As a result, each user A1, etc., must face the video conferencing terminal 3 when speaking. Also, because each user A1, etc., is facing the video conferencing terminal 3, the display 4 must also be placed on the side of the video conferencing terminal 3. Consequently, users A2 and A4, who are far from the video conferencing terminal 3, are far from the microphone 314 (see Figure 10), so they have to speak relatively loudly, and the content displayed on the display 4 is difficult to see.
[0132] In contrast, as shown in Figure 22(b), when using the imaging device 1a, two hemispherical images that form the basis of the full-spherical panoramic image can be obtained, allowing the video conferencing terminal 3 and display 4 to be placed relatively close to the center of the desk. As a result, each user A1, etc., is closer to the microphone 314, allowing them to speak in a relatively low voice, and the content displayed on the display 4 is also easier to see.
[0133] Next, we will explain an example of the display on the display 917 at site B using Figure 23. Figure 23 shows an example of the display on the display at site B. Of these, Figure 23(a) shows the case where a 360-degree panoramic image and a predetermined area image are displayed directly from the image data sent from video conferencing terminal 3 (shooting device 1a) and shooting device 1b without creating a full-sphere panoramic image and a predetermined area image, and the case where the image data sent from PC 7 (shooting device 8) is displayed directly. Figure 23(b) shows the case where a 360-degree panoramic image and a predetermined area image are created from the image data sent from video conferencing terminal 3 (shooting device 1a) and shooting device 1b, and the case where the image data sent from PC 7 (shooting device 8) is displayed directly. Note that the image of site A is displayed in the upper left display area of the display 917, the image of site B (own site) is displayed in the upper right display area, and the image of site C is displayed in the lower left display area. Also, in this example, simultaneous video calls are being made between three sites, so nothing is displayed in the lower right display area.
[0134] When the image data sent from the imaging devices 1a and 1b, which are capable of capturing full-spherical panoramic images, is displayed as is, it appears as shown in Figure 23(a). In this case, the upper left point A and the upper right point B (the local point) are displayed as the front hemisphere image and the rear hemisphere image, respectively, as shown in Figures 3(a) and (b).
[0135] In response, the image and sound processing unit 93 creates a full-sphere panoramic image from the image data sent from the imaging devices 1a and 1b, which can obtain two hemispherical images that will form the basis of the full-sphere panoramic image. Furthermore, it creates a predetermined region image, and as shown in Figure 23(b), the predetermined region image, which is a planar image, is displayed. Note that since base C is equipped with an imaging device 8 that obtains a general image, a general image is displayed in both Figures 23(a) and (b).
[0136] Furthermore, an icon 191 indicating that it is a 360-degree panoramic image is displayed in the upper left corner of each image of base A and base B (the user's own base). Note that the display position of icon 191 is not limited to the upper left, but can be anywhere, such as the upper right, lower left, or lower right. Also, the type of icon 191 is not limited to that shown in Figure 23(b). In addition, instead of icon 191, it may be text such as "360-degree image," or a combination of icon and text may be used.
[0137] Furthermore, users can change a predetermined region of the image within the same 360-degree panoramic image. Specifically, users B1 and B2 can move their fingers on the touch panel 921 of the smartphone 9, allowing the reception unit 92 to receive the finger movement, and the display control unit 94 to shift, rotate, shrink, or enlarge the predetermined region image. As a result, as shown in Figure 23(b), even if only users A1 and A2 of site A are displayed in the initial (default) settings, users A4 and A3 can be displayed by shifting the predetermined region image.
[0138] <<Main Effects of This Embodiment>> As described above, according to this embodiment, a communication terminal such as the video conferencing terminal 3 can create a full-spherical panoramic image based on the image data ID sent along with the image data, and further create a predetermined region image based on the corresponding image type information. This has the effect of preventing the display of the front hemisphere image and the rear hemisphere image, as shown in Figure 23(a).
[0139] [Other examples] Figure 24 illustrates another example of the image type information management process. Figure 24 is a sequence diagram showing another example of the image type information management process.
[0140] First, when a user at site A (for example, user A1) connects the USB cable of the cradle 2a, with the camera 1a attached, to the video conferencing terminal 3, the memory / read unit 19a of the camera 1a reads the GUID of its own device (camera 1a) stored in the memory unit 1000a, and the communication unit 18a transmits the GUID of its own device to the communication unit 38 of the video conferencing terminal 3 (step S151). As a result, the communication unit 38 of the video conferencing terminal 3 receives the GUID of the camera 1a.
[0141] Next, the transmitting / receiving unit 31 of the video conferencing terminal 3 sends a request for image data ID to the communication management system 5 (step S152). This request includes the IP address of the source terminal (itself), the video conferencing terminal 3. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the request for image data ID.
[0142] Next, the generation unit 56 generates a unique image data ID (step S153). Then, the communication management system 5 sends a notification of the image data ID to the smartphone 9 (step S154). This notification includes the image data ID created in step S153 and the IP address of the video conferencing terminal 3, which is the source terminal, received in step S152. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the notification of the image data ID. Then, the storage / reading unit 99 of the smartphone 9 stores the image data ID received in step S154 and the IP address of the source terminal as a new record in the image type management DB 9001 (see Figure 14) (step S155). Note that at this point, the image type information is not stored in association.
[0143] Meanwhile, in response to the request for the image data ID in step S152, the communication management system 5 transmits the image data ID to the video conferencing terminal 3 via the transmitting / receiving unit 51 (step S156). As a result, the transmitting / receiving unit 31 of the video conferencing terminal 3 receives the image data ID. The storage / reading unit 39 of the video conferencing terminal 3 then stores the image data ID received in step S156 and the IP address of the source terminal (its own terminal) managed by the video conferencing terminal 3 as a new record in the image type management DB 3001 (see Figure 14) (step S157). Note that at this point, the image type information is not stored in association.
[0144] Next, the determination unit 35 of the video conferencing terminal 3 determines the image type (step S158) by checking whether the vendor ID and product ID of the GUID received in step S151 are managed in the imaging device management DB 3002 (see Figure 15). This determination is the same as the process in step S52 above.
[0145] Next, the storage / reading unit 39 stores the IP address of the source terminal (video conferencing terminal 3) and the image type information, which is the determination result made in step S158, in association with the image data ID already stored in step S157 in the image type management DB 3001 (see Figure 14) (step S159). Then, the transmitting / receiving unit 31 sends a notification of image type information to the communication management system 5 (step S160). This notification of image type information includes the image data ID stored in step S157, as well as the IP address of the source terminal (the local terminal) and the image type information stored in step S159. As a result, the transmitting / receiving unit 51 of the communication management system 5 receives the notification of image type information.
[0146] Next, the storage and reading unit 59 of the communication management system 5 uses the IP address of the source terminal received in step S160 as a search key to search the session management DB 5001 (see Figure 16) and read the corresponding session ID (step S161).
[0147] Next, the storage / reading unit 59 stores the session ID read in step S161, along with the IP address of the source terminal, the image data ID, and the image type information received in step S160, as a new record in the image type management DB 5002 (see Figure 17) (step S162). Then, the communication management system 5 sends a notification of image type information to the smartphone 9, which is another communication terminal (step S163). This notification of image type information includes the image data ID and image type information stored in step S162. As a result, the transmitting / receiving unit 91 of the smartphone 9 receives the notification of image type information.
[0148] Next, the storage and reading unit 99 of the smartphone 9 associates the image type information received in step S163 with the same image data ID received in step S163 and stores it in the image type management DB 9001 (see Figure 14) (step S164). This completes the image type information management process. [Explanation of Symbols]
[0149] 1a Imaging device 1b Imaging device 3. Video conferencing terminal (an example of a communication terminal, an example of the first communication terminal) 5. Communication Management System 7. PC (an example of a communication terminal) 8. Imaging device 9. Smartphone (an example of a communication terminal, an example of a second communication terminal) 31 Transmitter / Receiver 51 Transmitting and Receiving Unit (Example of transmitting means, example of receiving means) 56 Generation Unit (Example of Generation Means) [Prior art documents] [Patent Documents]
[0150] [Patent Document 1] Japanese Patent Publication No. 2012-178135 [Patent Document 2] Japanese Patent Publication No. 2011-223076
Claims
1. A receiving means for receiving image type information indicating the type of image from a first communication terminal, Based on the reception of the aforementioned image type information, a generation means generates image data identification information for identifying image data to be transmitted from the first communication terminal to the second communication terminal, A transmission means that transmits the generated image data identification information to the first communication terminal and transmits the generated image data identification information and the received image type information to the second communication terminal, A communication management system characterized by having the following features.
2. A communication management system according to claim 1, It has predetermined session identification information for identifying a predetermined video call communication session, and session management means for associating and managing the destinations of a predetermined number of communication terminals. The transmission means is characterized in that the session management means transmits the image data identification information and the image type information to the destination of the second communication terminal which is associated with a specific session identification information together with the destination of the first communication terminal.
3. The communication management system according to claim 1 or 2, characterized in that the image types are planar images and 360-degree panoramic images.
4. A communication management system according to any one of claims 1 to 3, The first communication terminal and, A communication system characterized by having the following features.
5. The communication system according to claim 4, characterized in that the first communication terminal determines the image type based on the GUID obtained from the imaging device that obtains the full-sphere panoramic image data, and transmits the image type information indicating the result of the determination to the communication management system.
6. A communication management system according to any one of claims 1 to 3, The aforementioned second communication terminal, A communication system characterized by having the following features.
7. The communication system according to claim 6, characterized in that, when the second communication terminal indicates that the image type information is a 360-degree panoramic image, it creates a 360-degree panoramic image data based on the image data transmitted by the first communication terminal.
8. The communication system according to any one of claims 1 to 3, characterized in that the first communication terminal and the second communication terminal are a video conferencing terminal, a personal computer, a smartphone, a digital television, a smartwatch, or a car navigation device.
9. A communication management method performed by a communication management system that manages communication between first and second communication terminals, The aforementioned communication management system is A receiving step of receiving image type information indicating the type of image from a first communication terminal, Based on the reception of the aforementioned image type information, a generation step is made to generate image data identification information for identifying image data to be transmitted from the first communication terminal to the second communication terminal, A transmission step of transmitting the generated image data identification information to the first communication terminal and transmitting the generated image data identification information and the received image type information to the second communication terminal, A communication management method characterized by performing the following.
10. A program for causing a computer to perform the method described in claim 9.
Citation Information
Patent Citations
Omnidirectional fisheye panoramic live moving picture distribution system
JP2011223076A
Screen sharing device, screen sharing method, program, program providing system, and maintenance system
JP2012178135A
