Communication terminal, display method, and program

Abstract

To provide a communication terminal, a display method, and a program, capable of easily gripping a non-language communication of a peron on a transmission side included in an image at the outside of a predetermined region in a wide visual field image against a user on a reception side of the wide visual field image even in the case where a predetermined region image as a predetermined region of the wide visual field image such as an entire-celestial sphere image is displayed onto a display part.SOLUTION: In a conference system that performs a remote conference from a remote site via a communication network, a communication terminal that causes a display part to display a predetermined region image as a predetermined region of a wide visual field image, includes: a reception part that receives a wide visual field image to be transmitted by another communication terminal; and a display control part that displays predetermined icons c1 and c2 corresponding to a predetermined non-language communication of users A1 and A2 included in an image at the outside of a predetermined region in the wide visual field image to be received by the reception part to a predetermined position in the predetermined region image.SELECTED DRAWING: Figure 28

Description

Technical Field

[0001] The present invention relates to a communication terminal, a display method, and a program.

Background Art

[0002] Conference systems for holding remote conferences with remote locations via a communication network such as the Internet have become widespread. In this conference system, in a conference room where one of the parties such as the attendees of the remote conference is located, the communication terminal of the remote conference system is used to capture and collect the image of the conference room and the voice such as the speech of the conference participants, and these are converted into digital data and transmitted to the communication terminal of the other party (see Patent Document 1). As a result, since image display can be performed on the display of the other party's conference room and voice output can be performed by a speaker to conduct a video call, a conference between remote locations can be conducted in a state close to an actual conference.

[0003] On the other hand, a technique is known in which a photographing device capable of acquiring a panoramic image in real time is connected to a communication terminal, the panoramic image acquired from this photographing device is distributed to each communication terminal of the other party, and in each communication terminal of the other party, a predetermined region image of a rectangle, which is a predetermined region of the panoramic image, is displayed on a display unit such as a display (see Patent Document 2).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when displaying a predetermined region image on each communication terminal of the other party, there has been a problem that non-verbal communication such as the facial expression and hand gestures of a person included in the image outside the predetermined region in the panoramic image is missed.

[0005] In order to solve the above problems, even when a predetermined region image, which is a predetermined region of a wide-angle image such as an omnidirectional image, is displayed on a display unit, it is an object of the present disclosure to make it easier for a user on the receiving side of the wide-angle image to grasp non-verbal communication of a person on the transmitting side included in an image outside the predetermined region in the wide-angle image.

Means for Solving the Problems

[0006] The present disclosure is a communication terminal that displays a predetermined region image, which is a predetermined region of a wide-angle image, on a display unit, and includes a receiving unit that receives the wide-angle image transmitted by another communication terminal, and a display control unit that displays a predetermined icon corresponding to predetermined non-verbal communication of a person included in an image outside the predetermined region in the wide-angle image received by the receiving unit at a predetermined position in the predetermined region image.

Effects of the Invention

[0007] As described above, according to the present disclosure, there is an effect that it is possible to make it easier for a user on the receiving side of a wide-angle image to grasp non-verbal communication of a person on the transmitting side included in an image outside a predetermined region in the wide-angle image.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0010] <<Overview of the Embodiment>> <Method for Generating an Entire Sky Image> Using FIGS. 1 to 7, a method for generating an entire sky image will be described. Note that the entire sky image is also referred to as an entire sky panorama image or a 360° panorama image, and is an example of a wide-angle video having a wide field of view. The wide-angle image includes a simple panorama image of about 180°.

[0011] First, the appearance of the imaging device 1 will be described with reference to FIG. 1. The imaging device 1 is a digital camera for obtaining a captured image that serves as the basis for the entire sky (360°) image. Note that FIG. 1(a) is a left side view of the imaging device, FIG. 1(b) is a front view of the imaging device, and FIG. 1(c) is a plan view of the imaging device.

[0012] As shown in FIG. 1(a), the imaging device 1 is sized such that it can be held in one hand by a human. Also, as shown in FIGS. 1(a), 1(b), and 1(c), on the upper part of the imaging device 1, an imaging element 103a is provided on the front side (front) and an imaging element 103b is provided on the back side (rear). These imaging elements (image sensors) 103a and 103b are used in combination with an optical member (for example, the fish-eye lenses 102a and 102b described later) capable of capturing a hemispherical image (viewing angle of 180° or more). Further, as shown in FIG. 1(b), an operation unit 115 such as a shutter button is provided on the surface of the imaging device 1 opposite to the front side.

[0013] Next, the usage situation of the imaging device 1 will be described with reference to FIG. 2. Note that FIG. 2 is an image diagram of the usage of the imaging device. As shown in FIG. 2, the imaging device 1 is used, for example, by a user holding it in their hand to capture a subject around the user. In this case, the imaging elements 103a and 103b shown in FIG. 1 are used to capture the subject around the user, respectively, to obtain two hemispherical images.

[0014] Next, with reference to FIGS. 3 and 4, an overview of the process from the images captured by the imaging device 1 to the creation of a full-spherical image will be described. Note that FIG. 3(a) shows a hemispherical image (front side) captured by the imaging device, FIG. 3(b) shows a hemispherical image (rear side) captured by the imaging device, and FIG. 3(c) is a diagram showing an image represented by the Mercator projection method (hereinafter referred to as a "transverse cylindrical projection image"). FIG. 4(a) is a conceptual diagram showing a state where a sphere is covered with a transverse cylindrical projection image, and FIG. 4(b) is a diagram showing a full-spherical image.

[0015] As shown in FIG. 3(a), the image obtained by the imaging device 103a becomes a hemispherical image (front side) curved by the fish-eye lens 102a described later. Also, as shown in FIG. 3(b), the image obtained by the imaging device 103b becomes a hemispherical image (rear side) curved by the fish-eye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image (rear side) inverted by 180 degrees are synthesized by the imaging device 1, and an orthographic cylindrical projection image is created as shown in FIG. 3(c).

[0016] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 4(a), the orthographic cylindrical projection image is pasted so as to cover the spherical surface, and an omnidirectional image as shown in FIG. 4(b) is created. Thus, the omnidirectional image is represented as an image in which the orthographic cylindrical projection image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image may be a still image or a moving image.

[0017] As described above, since the omnidirectional image is an image pasted so as to cover the spherical surface, it gives an uncomfortable feeling to humans when viewed. Therefore, by displaying a predetermined region (hereinafter referred to as "predetermined region image") of a part of the omnidirectional image as a planar image with less curvature, it is possible to perform a display that does not give an uncomfortable feeling to humans. This will be described with reference to FIGS. 5 and 6.

[0018] Note that FIG. 5 is a diagram showing the position of a virtual camera and a predetermined region when the omnidirectional image is a three-dimensional spherical object. The virtual camera IC corresponds to the position of the viewpoint of the user who views the image with respect to the omnidirectional image displayed as a three-dimensional spherical object. Further, FIG. 6(a) is a perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a predetermined region image when displayed on a display. Also, in FIG. 6(a), the omnidirectional image shown in FIG. 4 is represented by a three-dimensional spherical object CS. Assuming that the omnidirectional image generated in this way is the spherical object CS, as shown in FIG. 5, the virtual camera IC is located outside the omnidirectional image. The predetermined region T in the omnidirectional image is the shooting region of the virtual camera IC, and is specified by predetermined region information indicating the position coordinates (x(rH), y(rV), field angle α(angle)) including the field angle of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image. The zoom of the predetermined region T can be expressed by expanding or contracting the range (arc) of the field angle α. Also, the zoom of the predetermined region T can be expressed by moving the virtual camera IC closer to or farther from the omnidirectional image.

[0019] Then, the predetermined region image, which is an image of the predetermined region T shown in FIG. 6(a), is displayed on a predetermined display as an image of the shooting region of the virtual camera IC, as shown in FIG. 6(b). The image shown in FIG. 6(b) is a predetermined region image represented by predetermined region information set by default. Note that the predetermined region information may be indicated by the shooting region (X, Y, Z) of the virtual camera IC, which is the predetermined region T, instead of the position coordinates of the virtual camera IC. Hereinafter, the explanation will be given using the position coordinates (x(rH), y(rV), and field angle α(angle)) of the virtual camera IC.

[0020] Next, with reference to FIG. 7, the relationship between the predetermined region information and the image of the predetermined region T will be described. Note that FIG. 7 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in FIG. 7, when the diagonal angle of view of the predetermined region T represented by the angle of view α of the virtual camera IC is 2L, the center point CP becomes the (x, y) parameters of the predetermined region information. f is the distance from the virtual camera IC to the center point CP. L is the distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is the diagonal). And in FIG. 7, generally, the trigonometric function represented by the following formula (1) holds.

[0021] (L / f) = tan(α / 2) ··· (Formula 1) By changing the virtual viewpoint by the virtual camera IC, the displayed predetermined region image is also changed.

[0022] <Outline of the Image Communication System> Next, with reference to FIG. 8, the outline of the configuration of the image communication system of the present embodiment will be described. FIG. 8 is a schematic diagram of the configuration of the image communication system of the present embodiment.

[0023] As shown in FIG. 8, the image communication system of the present embodiment is composed of imaging devices 1a, 1b, video conference terminals 3, a communication management system 5, a PC (Personal Computer) 7, an imaging device 8, and a smartphone 9, and can communicate via a communication network 100 such as the Internet. The connection form of the communication network 100 may be either wireless or wired.

[0024] Among these, the imaging devices 1a, 1b are special digital cameras for photographing a subject, a landscape, etc. to obtain two hemispherical images that are the basis of the omnidirectional image. On the other hand, the imaging device 8 is a general digital camera for photographing a subject, a landscape, etc. to obtain a general planar image.

[0025] The video conferencing terminal 3 is a terminal dedicated to video conferencing, and displays the image of a video call (video conference) on the display 4 via a wired cable such as a USB (Universal Serial Bus) cable. The video conferencing terminal 3 usually takes pictures with the camera 312 described later. However, when it is connected by a wired cable to the cradle 2a to which the imaging device 1a is attached, the imaging device 1a is prioritized, and two hemispherical images that are the basis of the omnidirectional image can be obtained. When using a wired cable, the cradle 2a not only communicates between the imaging device 1a and the video conferencing terminal 3, but also supplies power to the imaging device 1a and supports the imaging device 1a. Here, the imaging device 1a, the cradle 2a, the video conferencing terminal 3, and the display 4 are placed at Site A, which is the same site. Also, at Site A, four users A1, A2, A3, and A4 are participating in the video call.

[0026] The communication management system 5 manages the communication of the video conferencing terminal 3, the PC 7, and the smartphone 9, and manages the types of image data (general image and special image types) transmitted and received. Here, the special image is an omnidirectional image. The communication management system 5 is installed in a service company or the like that provides video communication services. Also, the communication management system 5 may be constructed by a single computer, or may be constructed by a plurality of computers arbitrarily assigned by dividing each part (function, means, or storage part).

[0027] The PC 7 can make a video call by attaching the imaging device 8. Here, the PC 7 and the imaging device 8 are placed at Site C, which is the same site. Also, at Site C, one user C is participating in the video call.

[0028] The smartphone 9 displays the image of the video call on the display 917 provided in the device, which will be described later. The smartphone 9 usually takes pictures with the CMOS (Complementary Metal Oxide Semiconductor) sensor 905 and the like provided in the device itself, but it can also use wireless communication technologies such as Wi-Fi (Wireless Fidelity) and Bluetooth (registered trademark) to obtain two hemispherical image data that are the basis of the omnidirectional image obtained by the imaging device 1b. When using wireless communication technology, the cradle 2b only supplies power to the imaging device 1b and supports the imaging device 1b. Here, the imaging device 1b, the cradle 2b, and the smartphone 9 are placed at the same base point B. Also, at base point B, two users B1 and B2 are participating in the video call.

[0029] Also, the video conferencing terminal 3, the PC 7, and the smartphone 9 are examples of communication terminals. OpenGL ES is installed in each communication terminal, and it can create predetermined area information indicating a partial area of the omnidirectional image or create a predetermined area image from the omnidirectional image sent from another communication terminal.

[0030] Note that the arrangement of each terminal, device, and user shown in FIG. 8 is an example, and other examples are also possible. For example, at base point C, instead of the imaging device 8, an imaging device capable of taking pictures related to the omnidirectional image may be used by the user. Also, communication terminals include digital TVs, smartwatches, car navigation devices, and the like. Also, hereinafter, when representing any one of the imaging devices 1a and 1b, it is represented as "imaging device 1".

[0031] <<Hardware Configuration of the Embodiment>> Next, with reference to FIGS. 9 to 12, the hardware configurations of the imaging device 1, the video conferencing terminal 3, the communication management system 5, the PC 7, and the smartphone 9 of this embodiment will be described in detail. Since the imaging device 8 is a general camera, a detailed description thereof is omitted.

[0032] <Hardware Configuration of Imaging Device 1> First, the hardware configuration of imaging device 1 will be described with reference to FIG. 9. FIG. 9 is a hardware configuration diagram of imaging device 1. Hereinafter, imaging device 1 is an omnidirectional imaging device that uses two imaging elements, but the number of imaging elements may be two or more. Also, it does not necessarily have to be a device dedicated to omnidirectional imaging. By attaching an aftermarket omnidirectional imaging unit to a normal digital camera, smartphone, etc., it may be made to have substantially the same function as imaging device 1.

[0033] As shown in FIG. 9, imaging device 1 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I / F 116, a communication unit 117, and an antenna 117a.

[0034] Among these, imaging unit 101 includes wide-angle lenses 102a, 102b each having an angle of view of 180° or more for forming a hemispherical image (so-called fisheye lenses), and two imaging elements 103a, 103b provided corresponding to each wide-angle lens. Imaging elements 103a, 103b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors that convert the optical image by fisheye lenses 102a, 102b into image data of an electrical signal and output it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks of this image sensor, and a register group in which various commands and parameters necessary for the operation of this imaging element are set, etc.

[0035] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus) separately. The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via the bus 110. Further, the ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118, etc. are also connected to the bus 110.

[0036] The image processing unit 104 captures the image data output from the imaging elements 103a and 103b through the parallel I / F bus, performs predetermined processing on each image data, and then synthesizes these image data to create data of an orthographic cylindrical projection image as shown in Fig. 3(c).

[0037] The imaging control unit 105 generally uses the I2C bus with the imaging control unit 105 as the master device and the imaging elements 103a and 103b as slave devices to set commands, etc. in the register groups of the imaging elements 103a and 103b. The necessary commands, etc. are received from the CPU 111. Also, the imaging control unit 105 similarly uses the I2C bus to capture status data, etc. of the register groups of the imaging elements 103a and 103b and sends them to the CPU 111.

[0038] In addition, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Depending on the imaging device 1, there may be functions corresponding to a preview display function or a moving image display by a display (for example, the display of the video conferencing terminal 3). In this case, the output of the image data from the imaging elements 103a and 103b is continuously performed at a predetermined frame rate (frames / minute).

[0039] In addition, as will be described later, the imaging control unit 105 also functions as synchronization control means for synchronizing the output timings of the image data of the imaging elements 103a and 103b in cooperation with the CPU 111. Note that in this embodiment, the imaging device 1 is not provided with a display unit (display), but a display unit may be provided.

[0040] The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0041] The CPU 111 controls the overall operation of the imaging device 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data during processing, and the like. In particular, the DRAM 114 stores image data during processing in the image processing unit 104 and data of the processed orthographic cylindrical projection image.

[0042] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. The user inputs various shooting modes, shooting conditions, and the like by operating the operation buttons.

[0043] The network I / F 116 is a general term for interface circuits (such as USB I / F) with external media such as SD cards and personal computers. Also, the network I / F 116 may be wireless or wired. The data of the orthographic cylindrical projection image stored in the DRAM 114 is recorded on external media via this network I / F 116, or transmitted to an external device such as the video conferencing terminal 3 via the network I / F 116 as necessary.

[0044] The communication unit 117 communicates with an external device such as the video conferencing terminal 3 via the antenna 117a provided in the imaging device 1 by means of short-range wireless technologies such as Wi-Fi and NFC (Near Field Communication). The communication unit 117 can also transmit the data of the orthographic cylindrical projection image to the external device of the video conferencing terminal 3.

[0045] The electronic compass 118 calculates the orientation and tilt (Roll rotation angle) of the imaging device 1 from the earth's magnetism and outputs the orientation and tilt information. This orientation and tilt information is an example of related information (metadata) conforming to Exif and is used for image processing such as image correction of the captured image. Note that the related information also includes each data such as the shooting date and time of the image and the data capacity of the image data.

[0046] <Hardware Configuration of Video Conferencing Terminal> Next, the hardware configuration of the video conferencing terminal 3 will be described with reference to FIG. 10. FIG. 10 is a hardware configuration diagram of the video conferencing terminal. As shown in FIG. 10, the video conferencing terminal 3 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, operation buttons 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an image sensor I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319.

[0047] Among these, the CPU 301 controls the overall operation of the video conferencing terminal 3. The ROM 302 stores programs used for driving the CPU 301 such as the IPL (Initial Program Loader). The RAM 303 is used as the work area of the CPU 301. The flash memory 304 stores various data such as communication programs, image data, and audio data. The SSD (Solid State Drive) 305 controls the reading or writing of various data to and from the flash memory 304 according to the control of the CPU 301. Note that an HDD may be used instead of the SSD. The media I / F 307 controls the reading or writing (storage) of data to and from the recording medium 306 such as the flash memory. The operation button 308 is a button that is operated when selecting the destination of the video conferencing terminal 3, etc. The power switch 309 is a switch for switching the ON / OFF of the power of the video conferencing terminal 3.

[0048] Also, the network I / F 311 is an interface for performing data communication using the communication network 100 such as the Internet. The camera 312 is a type of built-in imaging means that captures a subject according to the control of the CPU 301 to obtain image data. The imaging device I / F 313 is a circuit that controls the driving of the camera 312. The microphone 314 is a type of built-in sound collection means for inputting sound. The audio input / output I / F 316 is a circuit that processes the input and output of audio signals between the microphone 314 and the speaker 315 according to the control of the CPU 301. The display I / F 317 is a circuit that transmits image data to the external display 320 according to the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (registered trademark) and Bluetooth (registered trademark).

[0049] Also, the bus line 310 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 301 shown in FIG. 10.

[0050] The display 4 is a type of display means composed of a liquid crystal or an organic EL that displays an image of a subject, an operation icon, etc. The display 4 is connected to the display I / F 317 by a cable 4c. This cable 4c may be a cable for an analog RGB (VGA) signal, a cable for component video, or a cable for an HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signal.

[0051] Note that the camera 312 includes a lens and a solid-state imaging device that converts light into electric charges to digitize an image (video) of a subject. As the solid-state imaging device, a CMOS sensor, a CCD sensor, etc. are used. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 by a USB (Universal Serial Bus) cable or the like. When an external camera is connected, the external camera is driven preferentially over the built-in camera 312 under the control of the CPU 301. Similarly, when an external microphone or an external speaker is connected, the external microphone or the external speaker is driven preferentially over the built-in microphone 314 or the built-in speaker 315 under the control of the CPU 301.

[0052] Also, the recording medium 306 is configured to be detachable from the video conferencing terminal 3. Also, as long as it is a non-volatile memory that reads or writes data according to the control of the CPU 301, not limited to the flash memory 304, an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used.

[0053] <Communication Management System, Hardware Configuration of PC> Next, the hardware configurations of the communication management system 5 and the PC 7 will be described with reference to FIG. 11. FIG. 11 is a hardware configuration diagram of the communication management system and the PC. Since both the communication management system 5 and the PC 7 are computers with the same configuration, the configuration of the communication management system 5 will be described below, and the description of the configuration of the PC 7 will be omitted.

[0054] The communication management system 5 includes a CPU 501 that controls the overall operation of the communication management system 5, a ROM 502 that stores programs used for driving the CPU 501 such as IPL, a RAM 503 used as a work area for the CPU 501, an HD 504 that stores various data such as programs for the communication management system 5, an HDD (Hard Disk Drive) 505 that controls the reading or writing of various data to and from the HD 504 according to the control of the CPU 501, a media drive 507 that controls the reading or writing (storage) of data to and from a recording medium 506 such as a flash memory, a display 508 that displays various information such as a cursor, menu, window, characters, or images, a network I / F 509 for data communication using the communication network 100, a keyboard 511 having a plurality of keys for inputting characters, numerical values, various instructions, etc., a mouse 512 for selecting and executing various instructions, selecting a processing target, moving a cursor, etc., a CD-RW drive 514 that controls the reading or writing of various data to and from a CD-RW (Compact Disc-ReWritable) 513 as an example of a removable recording medium, and a bus line 510 such as an address bus and a data bus for electrically connecting the above-described components as shown in FIG. 11.

[0055] <Hardware Configuration of Smart Phone> Next, the hardware of the smart phone will be described with reference to FIG. 12. FIG. 12 is a hardware configuration diagram of the smart phone. As shown in FIG. 12, the smart phone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, a CMOS sensor 905, an acceleration / azimuth sensor 906, a media I / F 908, and a GPS receiver 909.

[0056] Among these, the CPU 901 controls the operation of the entire smartphone 9. The ROM 902 stores programs used for driving the CPU 901 such as the IPL. The RAM 903 is used as a work area for the CPU 901. The EEPROM 904 reads or writes various data such as smartphone programs according to the control of the CPU 901. The CMOS (Complementary Metal Oxide Semiconductor) sensor 905 captures a subject (mainly a self-portrait) according to the control of the CPU 901 to obtain image data. The acceleration / azimuth sensor 906 is various sensors such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor. The media I / F 908 controls the reading or writing (storage) of data to / from the recording medium 907 such as a flash memory. The GPS receiver 909 receives GPS signals from GPS satellites.

[0057] Also, the smartphone 9 includes a long-distance communication circuit 911, a camera 912, an image sensor I / F 913, a microphone 914, a speaker 915, an audio input / output I / F 916, a display 917, an external device connection I / F 918, a short-distance communication circuit 919, an antenna 919a for the short-distance communication circuit 919, and a touch panel 921.

[0058] Among these, the long-distance communication circuit 911 is a circuit that communicates with other devices via the communication network 100. The camera 912 is a type of built-in imaging means that captures a subject according to the control of the CPU 901 to obtain image data. The imaging element I / F 913 is a circuit that controls the driving of the camera 912. The microphone 914 is a type of built-in sound collection means that inputs sound. The audio input / output I / F 916 is a circuit that processes the input and output of sound signals between the microphone 914 and the speaker 915 according to the control of the CPU 901. The display 917 is a type of display means such as a liquid crystal or an organic EL that displays images of subjects, various icons, etc. The external device connection I / F 918 is an interface for connecting various external devices. The short-distance communication circuit 919 is a communication circuit such as NFC or Bluetooth. The touch panel 921 is a type of input means for operating the smartphone 9 when the user presses the display 917.

[0059] Also, the smartphone 9 includes a bus line 910. The bus line 910 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 901.

[0060] Note that a recording medium such as a CD-ROM in which each of the above programs is stored, and an HD in which these programs are stored can both be provided as a program product, either domestically or abroad.

[0061] <<Functional Configuration of Embodiment>> Next, with reference to FIGS. 13 to 18, the functional configuration of this embodiment will be described. FIG. 13 is a functional block diagram of the photographing apparatuses 1a and 1b, the video conference terminal 3, the communication management system 5, the PC 7, and the smartphone 9, which constitute a part of the image communication system of this embodiment.

[0062] <Functional Configuration of Photographing Apparatus 1a> As shown in FIG. 13, the imaging device 1a has a reception unit 12a, an imaging unit 13a, a sound collection unit 14a, a communication unit 18a, and a storage / reading unit 19a. Each of these units is a function or means realized by any of the components shown in FIG. 9 operating according to instructions from the CPU 111 in accordance with a program for the imaging device developed from the SRAM 113 onto the DRAM 114.

[0063] In addition, the imaging device 1 has a storage unit 1000a constructed by the ROM 112, SRAM 113, and DRAM 114 shown in FIG. 9. The GUID (Globally Unique Identifier) of the device itself is stored in the storage unit 1000a.

[0064] (Functional configurations of the imaging device 1a) Next, with reference to FIGS. 9 and 13, the functional configurations of the imaging device 1a will be described in more detail.

[0065] The reception unit 12a of the imaging device 1a is mainly realized by the operation unit 115 and the processing of the CPU 111 shown in FIG. 9, and receives operation inputs from the user.

[0066] The imaging unit 13a is mainly realized by the imaging unit 101, the image processing unit 104, the imaging control unit 105 shown in FIG. 9, and the processing of the CPU 111, and captures a landscape or the like to obtain captured image data.

[0067] The sound collection unit 14a is realized by the microphone 108 and the sound processing unit 109 shown in FIG. 9, and the processing of the CPU 111, and collects the sounds around the imaging device 1.

[0068] The communication unit 18a is mainly realized by the processing of the CPU 111, and can communicate with the communication unit 38 of the video conferencing terminal 3 by short-range wireless communication technologies such as the NFC standard, Bluetooth, and WiFi.

[0069] The memory / readout unit 19a is mainly realized by the processing of the CPU 111 shown in FIG. 9, and stores various data (or information) in the storage unit 1000a or reads out various data (or information) from the storage unit 1000a.

[0070] <Functional configurations of the imaging device 1b> The imaging device 1b includes a reception unit 12b, an imaging unit 13b, a sound collection unit 14b, a communication unit 18b, and a memory / readout unit 19b. Since these respectively have the same functions as the reception unit 12a, the imaging unit 13a, the sound collection unit 14a, the communication unit 18a, and the memory / readout unit 19a in the imaging device 1a, the description thereof is omitted. Further, the imaging device 1 has a storage unit 1000b constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG. 9. The GUID of the device itself is stored in the storage unit 1000b.

[0071] <Functional configurations of the video conferencing terminal 3> As shown in FIG. 13, the video conferencing terminal 3 includes a transmission / reception unit 31, a reception unit 32, an image / sound processing unit 33, a display control unit 34, a determination unit 35, a creation unit 36, a detection unit 37, a communication unit 38, and a memory / readout unit 39. Each of these units is a function or means realized by any of the components shown in FIG. 10 operating according to an instruction from the CPU 301 in accordance with the program for the video conferencing terminal 3 expanded from the flash memory 304 onto the RAM 303.

[0072] Further, the video conferencing terminal 3 has a storage unit 3000 constructed by a ROM 302, a RAM 303, and a flash memory 304 shown in FIG. 10. In this storage unit 3000, an image type management DB 3001, a photographing device management DB 3002, and a non-verbal communication management DB 3003 are constructed. Among these, the image type management DB 3001 is constituted by an image type management table shown in FIG. 14. The photographing device management DB 3002 is constituted by a photographing device management table shown in FIG. 15. The non-verbal communication management DB 3003 is constituted by a non-verbal communication management table shown in FIG. 16.

[0073] (Image type management table) FIG. 14 is a conceptual diagram showing an image type management table. In this image type management table, an image data ID, an IP address which is an example of the destination of the transmission source terminal, and a source name are stored and managed in association with each other. Among these, the image data ID is an example of image data identification information for identifying image data during video communication. The same image data ID is added to the image data transmitted from the same transmission source terminal. Thereby, the destination terminal (the communication terminal on the receiving side) can specify the transmission source terminal of the received image data. The IP address of the transmission source terminal indicates the IP address of the communication terminal that transmits the image data indicated by the associated image data ID. The source name is a name for specifying the photographing device that outputs the image data indicated by the associated image data ID, and is an example of image type information. This source name is a name created by each communication terminal such as the video conferencing terminal 3 according to a naming rule of a predetermined name.

[0074] For example, it is shown that three communication terminals with IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1.3" respectively are transmitting image data indicated by image data IDs "RS001", "RS002", and "RS003". Further, the types of images indicated by the source names of each communication terminal are "Video_Theta", "Video", "Video_Theta", which in turn indicate that the image types are "special image", "general image", "special image". Here, the special image is a panoramic image.

[0075] Note that data other than image data may also be managed in association with the image data ID. Data other than image data is, for example, audio data or material data during screen sharing.

[0076] (Shooting device management table) FIG. 15 is a conceptual diagram showing a shooting device management table. In this shooting device management table, the vendor ID and product ID among the GUIDs of shooting devices that can obtain two hemispherical images that are the basis of panoramic images are stored and managed. As the GUID, for example, the vendor ID (VID) and product ID (PID) used in USB devices can be used. This vendor ID and product ID are stored since the factory shipment of communication terminals such as video conference terminals, but may also be additionally stored after factory shipment.

[0077] (Non-verbal communication management table) FIG. 16 is a conceptual diagram showing a non-verbal communication management table. In this non-verbal communication management table, the content of a person's non-verbal communication and the icon representing this non-verbal communication content are associated and stored for management. "Non-verbal communication" refers to communication that does not rely on language. In this embodiment, for a person, it includes at least one of facial expressions, facial colors, eye gazes, body movements, hand gestures, and body postures.

[0078] (Functional configurations of video conference terminal 3) Next, with reference to FIGS. 10 and 13, each functional configuration of the video conferencing terminal 3 will be described in more detail.

[0079] The transmission / reception unit 31 of the video conferencing terminal 3 is mainly realized by the network I / F 311 and the processing of the CPU 301 shown in FIG. 10, and transmits and receives various data (or information) with the communication management system 5 via the communication network 100.

[0080] The reception unit 32 is mainly realized by the operation buttons 308 and the processing by the CPU 301, and receives various selections or inputs from the user. In addition to the operation buttons 308, a touch panel or the like may be used as other input means.

[0081] The image / sound processing unit 33 is realized by the command from the CPU 301 shown in FIG. 10, and performs image processing on the image data obtained by the camera 312 imaging the subject. In addition, after the user's voice is converted into a voice signal by the microphone 314, the image / sound processing unit 33 performs voice processing on the sound data related to this voice signal.

[0082] Furthermore, for the display control unit 34 to display an image on the display 4, the image / sound processing unit 33 performs image processing on the image data received from other communication terminals based on the image type information such as the source name. Specifically, when the image type information indicates that it is a special image, the image / sound processing unit 33 creates omnidirectional image data by converting the image data (for example, the data of each hemisphere image as shown in FIGS. 3(a) and (b)) into omnidirectional image data as shown in FIG. 4(b), and further creates a predetermined area image as shown in FIG. 6(b). In addition, the image / sound processing unit 33 outputs the voice signal related to the sound data received from other communication terminals via the communication management system 5 to the speaker 315, and causes the speaker 315 to output voice.

[0083] The display control unit 34 is mainly realized by the processing of the display I / F 317 and the CPU 301, and performs control to display various images, characters, etc. on the display 4.

[0084] The determination unit 35 is mainly realized by the processing of the CPU 301, and determines, for example, the image type of the image data received from the imaging device 1a.

[0085] The creation unit 36 is mainly realized by the processing of the CPU 301. Based on the result determined by the determination unit 35 as a general image or a special image (here, an omnidirectional image), it creates a source name, which is an example of image type information, according to the above naming rule. For example, when the determination unit 35 determines that it is a general image, the creation unit 36 creates a source name "Video" indicating that it is a general image. On the other hand, when the determination unit 35 determines that it is a special image, the creation unit 36 creates a source name "Video_Theta" indicating that it is a special image.

[0086] The detection unit 37 has a trained machine learning model for searching for images of non-verbal communication of a person from an image, and detects when an image of non-verbal communication is included in the omnidirectional image. In this case, the search for a person in the omnidirectional image can be realized by the technology shown in Reference 1. Also, the determination of what kind of expression (for example, being puzzled) the person has can be realized by the technology shown in Reference 2. Furthermore, the determination of what kind of action (for example, clapping hands) the person has can be realized by the technique shown in Reference 3. Note that when detecting a movement such as clapping hands, the detection unit 37 needs to recognize it in a moving image for a certain period of time including the front and rear image data (frame data). <Reference 1>YOLO (https: / / docs.ultralytics.com / ) <Reference 2>deepface (https: / / github.com / serengil / deepface) <Reference 3>PoseNet (https: / / www.tensorflow.org / lite / examples / pose_estimation / overview?hl=ja) The communication unit 38 is mainly realized by the processing of the short-range communication circuit 319, the antenna 318a, and the CPU 301, and can communicate with the communication unit 18a of the imaging device 1a by short-range wireless technologies such as NFC, Bluetooth, and WiFi. Although the communication unit 38 and the transmission / reception unit 31 are described as having separate communication units, they may also have a shared configuration.

[0087] The storage / reading unit 39 is mainly realized by the processing of the CPU 301 shown in FIG. 10, and stores various data (or information) in the storage unit 3000 or reads out various data (or information) from the storage unit 3000.

[0088] <Functional Configuration of the Communication Management System> Next, with reference to FIGS. 11 and 13, each functional configuration of the communication management system 5 will be described in detail. The communication management system 5 includes a transmission / reception unit 51, a determination unit 55, a generation unit 56, and a storage / reading unit 59. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating according to instructions from the CPU 501 according to the program for the communication management system 5 expanded from the HD 504 onto the RAM 503.

[0089] The communication management system 5 also has a storage unit 5000 constructed by the RAM 503 and the HD 504 shown in FIG. 11. In this storage unit 5000, a session management DB 5001 and an image type management DB 5002 are constructed. Among these, the session management DB 5001 is composed of the session management table shown in FIG. 17. The image type management DB 5002 is composed of the image type management table shown in FIG. 18.

[0090] (Session Management Table) FIG. 17 is a conceptual diagram showing a session management table. In this session management table, a session ID and the IP addresses of participating communication terminals are stored and managed in association with each other. Among these, the session ID is an example of session identification information for identifying a communication session that realizes a video call, and is generated for each virtual conference room. The session ID is also managed by each communication terminal such as the video conference terminal 3, and is used when each communication terminal selects a communication session. The IP address of the participating communication terminal indicates the IP address of the communication terminal that has participated in the virtual conference room indicated by the associated session ID.

[0091] (Image type management table) FIG. 18 is a conceptual diagram showing an image type management table. The image type management table shown in FIG. 18 manages, in addition to each piece of information managed by the image type management table shown in FIG. 14, the same session ID as the session ID managed by the session management table in association with it. Here, it is shown that three communication terminals with IP addresses "1.2.1.3", "1.2.2.3", and "1.3.1.3" have participated in the virtual conference room indicated by the same session ID "se101". In the communication management system 5, the reason for managing the same image data ID, the IP address of the source terminal, and the image type information in communication terminals such as the video conference terminal 3 is to transmit the image type information and the like to the communication terminal that is already in a video call and the newly participating communication terminal when a new communication terminal enters a virtual conference room or the like. Thereby, it is not necessary to perform transmission and reception of image type information and the like between the communication terminal that is already in a video call and the newly participating communication terminal.

[0092] (Functional configurations of the communication management system) Next, with reference to FIGS. 11 and 13, each functional configuration of the communication management system 5 will be described in detail.

[0093] The transmission / reception unit 51 of the communication management system 5 is mainly realized by the processing of the network I / F 509 and the CPU 501 shown in FIG. 11, and transmits and receives various data (or information) to / from the video conferencing terminal 3 or the PC 7 via the communication network 100.

[0094] The determination unit 55 is mainly realized by the processing of the CPU 501, and performs various determinations.

[0095] The generation unit 56 is mainly realized by the processing of the CPU 501, and generates an image data ID.

[0096] The storage / reading unit 59 is mainly realized by the processing of the HDD 505 shown in FIG. 11 and the CPU 501, and stores various data (or information) in the storage unit 5000 or reads out various data (or information) from the storage unit 5000.

[0097] <Functional configuration of the PC> Next, with reference to FIGS. 11 and 13, the functional configuration of the PC 7 will be described in detail. The PC 7 basically has the same functions as the video conferencing terminal 3. That is, as shown in FIG. 13, the PC 7 has a transmission / reception unit 71, a reception unit 72, an image / audio processing unit 73, a display control unit 74, a determination unit 75, a creation unit 76, a detection unit 77, a communication unit 78, and a storage / reading unit 79. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating according to an instruction from the CPU 501 according to the program for the PC 7 expanded from the HD 504 onto the RAM 503.

[0098] In addition, the PC 7 has a storage unit 7000 constructed by the ROM 502, RAM 503, and HD 504 shown in FIG. 11. In this storage unit 7000, an image type management DB 7001, a photographing device management DB 7002, and a non-verbal communication management DB 7003 are constructed. Note that since the image type management DB 7001, the photographing device management DB 7002, and the non-verbal communication management DB 7003 have the same configuration as the image type management DB 3001, the photographing device management DB 3002, and the non-verbal communication management DB 3003, respectively, their descriptions are omitted.

[0099] (Functional configurations of the PC) The transmission / reception unit 71 of the PC 7 is mainly realized by the network I / F 509 and the processing of the CPU 501 shown in FIG. 11, and realizes the same functions as the transmission / reception unit 31.

[0100] The reception unit 72 is mainly realized by the keyboard 511, the mouse 512, and the processing of the CPU 501, and realizes the same functions as the reception unit 32.

[0101] The image / audio processing unit 73 is mainly realized by commands from the CPU 501, and realizes the same functions as the image / audio processing unit 33.

[0102] The display control unit 74 is mainly realized by the processing of the CPU 501, and realizes the same functions as the display control unit 34.

[0103] The determination unit 75 is mainly realized by the processing of the CPU 501, and realizes the same functions as the determination unit 35.

[0104] The creation unit 76 is mainly realized by the processing of the CPU 501, and realizes the same functions as the creation unit 36.

[0105] The detection unit 77 is mainly realized by the processing of the CPU 501, and realizes the same functions as the creation unit 37.

[0106] The communication unit 78 is mainly realized by the processing of the CPU 501 and realizes the same functions as the communication unit 38.

[0107] The memory / reading unit 79 is realized by the processing of the CPU 501, and stores various data (or information) in the memory unit 7000 or reads out various data (or information) from the memory unit 7000.

[0108] <Functional Configuration of Smartphone> Next, with reference to FIGS. 12 and 13, the functional configuration of the smartphone 9 will be described in detail. The smartphone 9 basically has the same functions as the video conferencing terminal 3. That is, as shown in FIG. 13, the smartphone 9 includes a transmission / reception unit 91, a reception unit 92, an image / audio processing unit 93, a display control unit 94, a determination unit 95, a detection unit 97, a creation unit 96, a communication unit 98, and a memory / reading unit 99. Each of these units is a function or means realized by any of the components shown in FIG. 12 operating according to instructions from the CPU 901 in accordance with the smartphone 9 program expanded from the EEPROM 904 onto the RAM 903.

[0109] In addition, the smartphone 9 has a memory unit 9000 constructed by the ROM 902, RAM 903, and EEPROM 904 shown in FIG. 12. In this memory unit 9000, an image type management DB 9001, a photographing device management DB 9002, and a non-verbal communication management DB 9003 are constructed. Note that since the image type management DB 9001, the photographing device management DB 9002, and the non-verbal communication management DB 9003 have the same configurations as the image type management DB 3001, the photographing device management DB 3002, and the non-verbal communication management DB 3003, respectively, the description thereof is omitted.

[0110] (Functional Configurations of Smartphone) The transmission / reception unit 91 of the smartphone 9 is mainly realized by the long-distance communication circuit 911 shown in FIG. 12 and the processing of the CPU 901, and realizes the same functions as the transmission / reception unit 31.

[0111] The reception unit 92 is mainly implemented by the touch panel 921 and the processing of the CPU 901, and realizes the same functions as the reception unit 32.

[0112] The image and audio processing unit 93 is mainly implemented by commands from the CPU 901, and realizes the same functions as the image and audio processing unit 33.

[0113] The display control unit 94 is mainly implemented by the processing of the CPU 901, and realizes the same functions as the display control unit 34.

[0114] The determination unit 95 is mainly implemented by the processing of the CPU 901, and realizes the same functions as the determination unit 35.

[0115] The creation unit 96 is mainly implemented by the processing of the CPU 901, and realizes the same functions as the creation unit 36.

[0116] The detection unit 97 is mainly implemented by the processing of the CPU 901, and realizes the same functions as the creation unit 37.

[0117] The communication unit 98 is mainly implemented by the processing of the CPU 901, and realizes the same functions as the communication unit 38.

[0118] The storage and reading unit 99 is implemented by the processing of the CPU 901, and stores various data (or information) in the storage unit 9000 or reads out various data (or information) from the storage unit 9000.

[0119] <<Processing or operation of the embodiment>> <Processing of participation> Subsequently, with reference to FIGS. 19 to 23, the processing or operation of this embodiment will be described. First, with reference to FIGS. 19 and 20, the participation processing for a specific communication session will be described. FIG. 19 is a sequence diagram showing the participation processing for a specific communication session. FIG. 20 is a diagram showing the selection screen of the communication session (virtual conference room).

[0120] First, when a user at base A (for example, user A1) operates the video conferencing terminal 3 to display a selection screen for a communication session (virtual meeting room), the reception unit 32 receives the operation to display the selection screen, and the display control unit 34 displays on the display 4 a selection screen as shown in FIG. 20 (step S21). This selection screen displays selection buttons b1, b2, b3, etc. indicating each virtual meeting room R1, R2, R3, etc. to be selected. Also, each session ID is associated with each selection button b1, etc.

[0121] Here, when user A1 selects a desired selection button for the virtual meeting room (here, selection button b1), the reception unit 32 receives the selection of the communication session (step S22). Then, the transmission / reception unit 31 transmits a participation request to the virtual meeting room to the communication management system 5 (step S23). This participation request includes the session ID indicating the communication session selected in step S22 and the IP address of the video conferencing terminal 3 which is the request source terminal. Thereby, the transmission / reception unit 51 of the communication management system 5 receives the participation request.

[0122] Next, the storage / reading unit 99 performs the participation process for the communication session by adding the IP address received in step S23 to the field of the participating terminal IP address in the record of the session ID same as the session ID received in step S23 in the session management DB5001 (see FIG. 17) (step S24). Then, the transmission / reception unit 51 transmits a participation request response to the video conferencing terminal 3 (step S25). This participation request response includes the session ID received in step S23 and the participation process result. Thereby, the transmission / reception unit 31 of the video conferencing terminal 3 receives the participation request response. Hereinafter, the case where the participation process is successful will be described.

[0123] <Management Process of Image Type Information> Subsequently, with reference to FIG. 21, the management process of the image type information will be described. FIG. 21 is a sequence diagram showing the management process of the image type information.

[0124] First, when a user at base A (for example, user A1) connects the USB cable of cradle 2a with imaging device 1a attached to video conferencing terminal 3, the storage / reading unit 19a of imaging device 1a reads the GUID of its own device (imaging device 1a) stored in storage unit 1000a, and communication unit 18a transmits the GUID of its own device to communication unit 38 of video conferencing terminal 3 (step S51). As a result, communication unit 38 of video conferencing terminal 3 receives the GUID of imaging device 1a.

[0125] Next, the determination unit 35 of video conferencing terminal 3 determines the image type by determining whether the vendor ID and product ID in the GUID received in step S51 are managed in the imaging device management DB3002 (see FIG. 15) (step S52). Specifically, in the imaging device management DB3002, if the same vendor ID and product ID are managed, the determination unit 35 determines that imaging device 1a is an imaging device that captures special images (here, omnidirectional images). On the other hand, in the imaging device management DB3002, if the same vendor ID and product ID are not managed, the determination unit 35 determines that imaging device 1a is an imaging device that captures general images.

[0126] Next, the storage / reading unit 39 stores, in association with the imaging type management DB3001 (see FIG. 14), the IP address of its own terminal (video conferencing terminal 3), which is the source terminal, and the imaging type information, which is the determination result determined in step S52 (step S53). In this state, the image data ID is not associated. The imaging type information is, for example, a source name determined according to a predetermined naming rule or the imaging type (general image, special image).

[0127] Next, the transmission / reception unit 31 transmits a request for adding image type information to the communication management system 5 (step S54). This request for adding image type information includes the IP address of the own terminal, which is the transmission source terminal stored in step S53, and the image type information. Thereby, the transmission / reception unit 51 of the communication management system 5 receives the request for adding image type information.

[0128] Next, the storage / reading unit 59 of the communication management system 5 searches the session management DB5001 (see FIG. 17) using the IP address of the transmission source terminal received in step S54 as a search key, and reads out the corresponding session ID (step S55).

[0129] Next, the generation unit 56 generates a unique image data ID (step S56). Then, the storage / reading unit 59 stores, as a new record in the image type management DB5002 (see FIG. 18), the session ID read out in step S55, the image data ID generated in step S56, and the IP address and image type information of the transmission source terminal received in step S54 in an associated manner (step S57). Then, the transmission / reception unit 51 transmits the image data ID generated in step S56 to the video conference terminal 3. Thereby, the transmission / reception unit 31 of the video conference terminal 3 receives the image data ID (step S58).

[0130] Next, the storage / reading unit 39 of the video conference terminal 3 stores the image data ID received in step S58 in association with the IP address and image type information of the own terminal (video conference terminal 3), which is the transmission source terminal stored in step S53, in the image type management DB3001 (see FIG. 14) (step S59).

[0131] On the one hand, the transceiver unit 51 of the communication management system 5 transmits an additional notification of image type information to the smartphone 9, which is another communication terminal (step S60). This additional notification of image type information includes the image data ID generated in step S56, as well as the IP address and image type information of the own terminal (video conferencing terminal 3), which is the transmission source terminal, stored in step S53. As a result, the transceiver unit 91 of the smartphone 9 receives the additional notification of image type information. Note that the transmission destination of the transceiver unit 51 is another IP address associated with the same session ID as the IP address of the video conferencing terminal 3 in the session management DB5001 (see FIG. 17). That is, the transmission destination is another communication terminal that has entered the same virtual conference room as the video conferencing terminal 3.

[0132] Next, the storage / reading unit 99 of the smartphone 9 stores, as a new record in the image type management DB9001 (see FIG. 14), the image data ID, the IP address of the transmission source terminal, and the image type information received in step S60 in an associated manner (step S61). Similarly, an additional communication of image type information is also transmitted to the PC7, which is another communication terminal, and the PC7 also stores it in the image type management DB7001 (see FIG. 14). As described above, each communication terminal can share the same information in each of the image type management DBs 3001, 7001, and 9001.

[0133] <Communication Processing of Image Data> The communication processing of image data in a video call will be described with reference to FIGS. 22 to 30. FIG. 22 is a sequence diagram showing the communication processing of image data in a video call.

[0134] First, the communication unit 18a of the imaging device 1a transmits image data obtained by photographing a subject, a landscape, etc. to the communication unit 38 of the video conferencing terminal 3 (step S101). In this case, since the imaging device 1a is a device that can obtain two hemispherical images that are the basis of a panoramic image, as shown in FIGS. 3(a) and (b), the image data is composed of the data of the two hemispherical images. As a result, the communication unit 38 of the video conferencing terminal 3 receives the image data.

[0135] Next, the transmission / reception unit 31 of the video conferencing terminal 3 transmits the image data sent from the imaging device 1a to the communication management system 5 (step S102). This transmission includes an image data ID for identifying the image data to be transmitted. As a result, the transmission / reception unit 51 of the communication management system 5 receives the image data and the image data ID.

[0136] Next, the transmission / reception unit 51 of the communication management system 5 transmits the image data sent from the video conferencing terminal 3 to the smartphone 9 (step S103). This transmission includes an image data ID for identifying the image data to be transmitted. As a result, the transmission / reception unit 51 of the smartphone 9 receives the image data and the image data ID.

[0137] Next, in the smartphone 9, the storage / reading unit 99 searches the image type management DB9001 (see FIG. 14) using the image data ID received in step S103 as a search key, and reads out the corresponding image type information (source name) (step S104). Then, when the image type information indicates that it is a special image (here, an omnidirectional image) (Video_Theta), the image / audio processing unit 93 creates an omnidirectional image from the image data received in step S103, and further creates a predetermined region image (step S105). This process will be described in detail later with reference to FIGS. 24 to 30.

[0138] Next, the display control unit 94 displays a predetermined region image including the icon c1 and the like on the display 917 of the smartphone 9. When the image type information indicates that it is a general image (Video), the image / audio processing unit 93 does not create an omnidirectional image from the image data received in step S103, and the display control unit 94 displays a general image that does not include the icon c1.

[0139] Next, the state of the video call will be described with reference to FIG. 23. FIG. 23 is an image diagram showing the state of the video call. Among these, FIG. 23(a) shows the case where the imaging device 1a is not used, and FIG. 23(b) shows the case where the imaging device 1a is used.

[0140] First, as shown in FIG. 23(a), when using the camera 312 (see FIG. 10) provided in the video conference terminal 3 without using the imaging device 1a, since the viewing angle is 125 degrees horizontally and 70 degrees vertically, the video conference terminal 3 must be placed at the end of the desk where each user A1 etc. can be reflected. For this reason, each user A1 etc. must talk facing the video conference terminal 3 side. Also, since each user A1 etc. faces the video conference terminal 3, the display 4 will also be placed on the side of the video conference terminal 3. As a result, users A2, A4 who are away from the video conference terminal 3 are far from the microphone 314 (see FIG. 10), so they have to talk in a relatively loud voice, and it is difficult to see the display content of the display 4.

[0141] On the other hand, as shown in FIG. 23(b), when using the imaging device 1a, since two hemispherical images that are the basis of the omnidirectional image can be obtained, the video conference terminal 3 and the display 4 can be placed relatively in the center of the desk. As a result, each user A1 etc. can talk in a relatively quiet voice because they are close to the microphone 314, and the display content of the display 4 is also easier to see.

[0142] (Creation process of the predetermined area image) Next, the creation process of the predetermined area image at base B will be described with reference to FIGS. 24 to 30.

[0143] As shown in FIG. 24, the image / sound processing unit 93 decodes the image data of the omnidirectional image received in process S103 and generates the frame data at that time (step S131).

[0144] Next, the detection unit 37 searches for non-verbal communication from the images of people included in the omnidirectional image, which is the decoded frame data (step S132). Here, this step S132 will be described with reference to FIGS. 25 and 26. FIG. 25 is a diagram showing the omnidirectional image as an orthographic cylindrical projection image. FIG. 26 is a diagram showing the state in which each user is further detected with respect to the diagram shown in FIG. 25. The orthographic cylindrical projection image of the omnidirectional image transmitted by the video conferencing terminal 3 (an example of another communication terminal) at the base A shown in FIG. 8 is represented as shown in FIG. 25. Here, as shown in FIG. 26, the detection unit 37 detects each user A to D as shown in the detection frames a1 to a4 from the orthographic cylindrical projection image (step S133). Further, in step S133, the detection unit 37 searches for the state of non-verbal communication of each user A to D in each detection frame a1 to a4. If non-verbal communication cannot be detected in step S133 (NO), the process shown in FIG. 24 ends.

[0145] On the other hand, if non-verbal communication is detected in step S132 (YES), the detection unit 37 determines whether the detected area is within a predetermined area to be displayed (step S134). If the detected area is not within the predetermined area to be displayed in step S134 (NO), the process shown in FIG. 24 ends. Note that the image / audio processing unit 33 may perform the process of step S134.

[0146] On the other hand, if the detected area is within the predetermined area to be displayed in step S134 (YES), the storage / reading unit 39 searches the non-verbal communication management DB9003 using the content of the non-verbal communication detected by the detection unit 37 as a search key and reads out the corresponding icon (step S135).

[0147] Finally, the image / sound processing unit 33 adds the icon read by the storage / reading unit 39 at a predetermined position within the predetermined region image (step S136). Note that the display control unit 34 may perform the process of step S136. Here, with reference to FIGS. 27 to 30, the process of step S136 will be described in more detail. FIG. 27 is a diagram showing the distance between the user A3 and the periphery of the predetermined region with respect to the diagram shown in FIG. 26. FIGS. 28 to 30 are diagrams showing display examples of the display at the base B.

[0148] As shown in FIG. 27, when the predetermined region image displayed on the display 917 includes the users A1 and A2 and does not include the users A3 and A4, the periphery of this predetermined region can be represented by the left frame TL, the upper frame TU, the right frame TR, and the lower frame TD. Also, in FIG. 27, the upper and lower extension lines EL of the left frame TL and the upper and lower extension lines ER of the right frame TR are shown. Note that these frames and extension lines are not actually displayed on the display 917.

[0149] In this case, the image / sound processing unit 33 derives a predetermined position that is the position of the shortest distance from a predetermined part of the body of the user A3 among the periphery of the predetermined region. The predetermined part is between the two eyes of a person or the nose of a person, etc. In the case of FIG. 27, the distance (the distance around in a circle) from the predetermined part of the user A3 to the right frame TR is L1, and the distance from the predetermined part of the user A3 to the left frame TL is L2. And in FIG. 27, L2 < L1. Thereby, the image / sound processing unit 33 determines that the position of the user A3 is close to the left side of the predetermined region, and this is also the shortest distance. Therefore, as shown in FIG. 28, the image / sound processing unit 33 adds the icon c1 corresponding to the content of the non-verbal communication of the user A3 on the left side (left end) of the predetermined region image displayed on the display 917 and on the right extension line of the predetermined part of the user A3. Similarly, the image / sound processing unit 33 adds the icon c2 corresponding to the content of the non-verbal communication of the user A4 on the right side (right end) of the predetermined region image displayed on the display 917 and on the left extension line of the predetermined part of the user A4.

[0150] As described above, in step S106, the display control unit 34 can display a predetermined area image as shown in FIG. 28 on the display 917. As a result, users B1 and B2 at base B can intuitively grasp that in the omnidirectional image, there is no person with a face slightly below the face of user A1 on the left side of user A1, but the person seems to be troubled by something, and there is no person with a face slightly above the face of user A2 on the right side of user A2, but the person is clapping.

[0151] Note that the user can change the predetermined area related to the predetermined area image in the same omnidirectional image. That is, users B1 and B2 touch and move a finger on the touch panel 921 of the smartphone 9. In response to the reception unit 92 receiving the movement of the finger, the display control unit 94 changes the virtual viewpoint of the virtual camera IC shown in FIG. 7, so that the predetermined area image can be shifted, rotated, reduced, or enlarged. As a result, as shown in FIG. 28, even when only some users A1 and A2 at base A are initially displayed (default), the predetermined area image can be shifted to display users A3 and A4.

[0152] Therefore, by changing the virtual viewpoint of the predetermined area image shown in FIG. 28 to the right, users B1 and B2 at base B can more easily ask what user A3 is troubled about in the video conference. In this case, for example, the display control unit 34 displays user A3 on the left side and user A1 on the right side within the predetermined area image.

[0153] In Fig. 27, if user A3 stands at the position shown in Fig. 27 and there is no right frame TR on the left extension line of a predetermined part of user A3, then L1 is the distance between the predetermined part of user A3 and the extension line ER. Similarly, if user A3 stands and there is no left frame TL on the right extension line of a predetermined part of user A3, then L2 is the distance between the predetermined part of user A3 and the extension line EL. As a result, as shown in Fig. 29, the icon c3 is added as a predetermined position on the left end of the predetermined area image. Consequently, the distance between the position of the left end of this predetermined area image and the predetermined part of user A3 is the shortest distance.

[0154] Also, in Fig. 27, if user A3 stands directly behind user A1, then the shortest distance from the predetermined part of user A3 to the upper frame TU is shorter than the shortest distance (the distance around in a circle) from the predetermined part of user A3 to the lower frame TD. As a result, as shown in Fig. 30, the icon c4 is added as a predetermined position directly above user A1 in the predetermined area image (the upper end of the predetermined area image). Consequently, the distance between the position of the upper end of this predetermined area image and the predetermined part of user A3 is the shortest distance.

[0155] <<Main effects of this embodiment>> As described above, according to this embodiment, there is an effect that it is possible to make it easier for a user on the receiving side of the omnidirectional image (an example of a wide-field image) to grasp the non-verbal communication of a person on the transmitting side included in an image outside the predetermined area within the omnidirectional image.

[0156] 〔Other examples〕 Using Fig. 31, other examples of the management process of image type information will be described. Fig. 31 is a sequence diagram showing other examples of the management process of image type information.

[0157] First, when a user at base A (e.g., user A1) connects the USB cable of cradle 2a with imaging device 1a attached to video conferencing terminal 3, the storage / readout unit 19a of imaging device 1a reads out the GUID of its own device (imaging device 1a) stored in storage unit 1000a, and communication unit 18a transmits the GUID of its own device to communication unit 38 of video conferencing terminal 3 (step S151). As a result, communication unit 38 of video conferencing terminal 3 receives the GUID of imaging device 1a.

[0158] Next, the transceiver unit 31 of video conferencing terminal 3 transmits a request for an image data ID to communication management system 5 (step S152). This request includes the IP address of video conferencing terminal 3, which is the source terminal (own terminal). As a result, the transceiver unit 51 of communication management system 5 receives the request for an image data ID.

[0159] Next, the generation unit 56 generates a unique image data ID (step S153). Then, communication management system 5 transmits a notification of the image data ID to smartphone 9 (step S154). This notification includes the image data ID created in step S153 and the IP address of video conferencing terminal 3 as the source terminal received in step S152. As a result, the transceiver unit 91 of smartphone 9 receives the notification of the image data ID. And in smartphone 9, the storage / readout unit 99 stores, as a new record in the image type management DB9001 (see FIG. 14), the image data ID received in step S154 and the IP address of the source terminal in an associated manner (step S155). Note that at this point, the image type information is not stored in an associated manner.

[0160] On the other hand, in order to respond to the request for the image data ID in step S152, the transmission / reception unit 51 of the communication management system 5 transmits the image data ID to the video conference terminal 3 (step S156). As a result, the transmission / reception unit 31 of the video conference terminal 3 receives the image data ID. Then, in the video conference terminal 3, the storage / readout unit 39 stores, as a new record, the image data ID received in step S156 and the IP address of the transmission source terminal (own terminal) managed by the video conference terminal 3 in the image type management DB3001 (see FIG. 14) in an associated manner (step S157). At this point, the image type information is not stored in an associated manner.

[0161] Next, the determination unit 35 of the video conference terminal 3 determines the image type by determining whether the vendor ID and product ID of the GUID received in step S151 are managed in the imaging device management DB3002 (see FIG. 15) (step S158). This determination is the same as the process in step S52 described above.

[0162] Next, the storage / readout unit 39 stores, in the image type management DB3001 (see FIG. 14), the IP address of the own terminal (video conference terminal 3) which is the transmission source terminal and the image type information which is the determination result determined in step S158 in an associated manner with respect to the image data ID already stored in step S157 (step S159). Then, the transmission / reception unit 31 transmits a notification of the image type information to the communication management system 5 (step S160). The notification of this image type information includes the image data ID stored in step S157, and the IP address of the own terminal which is the transmission source terminal and the image type information stored in step S159. As a result, the transmission / reception unit 51 of the communication management system 5 receives the notification of the image type information.

[0163] Next, the storage / reading unit 59 of the communication management system 5 searches the session management DB5001 (see FIG. 17) using the IP address of the source terminal received in step S160 as a search key, and reads out the corresponding session ID (step S161).

[0164] Next, the storage / reading unit 59 stores, as a new record in the image type management DB5002 (see FIG. 18), the session ID read in step S161, as well as the IP address of the source terminal, the image data ID, and the image type information received in step S160 in an associated manner (step S162). Then, the communication management system 5 transmits a notification of the image type information to the smartphone 9, which is another communication terminal (step S163). This notification of the image type information includes the image data ID and the image type information stored in step S162. As a result, the transmission / reception unit 91 of the smartphone 9 receives the notification of the image type information.

[0165] Next, the storage / reading unit 99 of the smartphone 9 additionally stores, in the image type management DB9001 (see FIG. 14), the image type information received in step S163 in an associated manner with the same image data ID as the image data ID received in step S163 (step S164). Thereby, the management process of the image type information is completed.

Explanation of Reference Numerals

[0166] 1a Imaging device 1b Imaging device 3 Video conferencing terminal (an example of a communication terminal, an example of a first communication terminal) 5 Communication management system 7 PC (an example of a communication terminal) 8 Imaging device 9 Smartphone (an example of a communication terminal, an example of a second communication terminal) 31 Transmission / reception unit (an example of a transmission unit, an example of a reception unit) 33 Image / audio processing unit 34 Display control unit 51 Transmission / reception unit (an example of a transmission unit, an example of a reception unit) 53 Image and Audio Processing Unit 54 Display Control Unit 91 Transmission / Reception Unit (an example of a transmission unit and an example of a reception unit) 93 Image and Audio Processing Unit 94 Display Control Unit

Prior Art Documents

Patent Documents

[0167]

Patent Document 1

Patent Document 2

Claims

1. A communication terminal for causing a display unit to display a predetermined region image that is a predetermined region of a wide-view image, comprising: a receiving unit that receives the wide-view image transmitted by another communication terminal; a display control unit that causes a predetermined icon corresponding to a predetermined non-verbal communication of a person included in an image outside the predetermined region in the wide-view image received by the receiving unit to be displayed at a predetermined position in the predetermined region image; A communication terminal having the above.

2. The communication terminal according to claim 1, wherein the predetermined non-verbal communication includes at least one of a facial expression, a facial color, a gaze, a body movement, a hand movement, and a body posture of the person.

3. The communication terminal according to claim 1, wherein the predetermined position is a position having the shortest distance from a predetermined part of the person's body among the surroundings of the predetermined region.

4. The communication terminal according to claim 3, wherein the predetermined part is between the person's two eyes or the person's nose.

5. A communication terminal according to any one of claims 1 to 4, comprising: a detection unit that detects a predetermined non-verbal communication of the person included in an image outside the predetermined region in the wide-view image; The display control unit causes the predetermined icon corresponding to the predetermined non-verbal communication detected by the detection unit to be displayed at the predetermined position in the predetermined region image.

6. The communication terminal according to claim 5, comprising: a storage unit that stores in association with each other the content of the person's non-verbal communication and an icon representing the content of the non-verbal communication; The display control unit causes the predetermined icon corresponding to the predetermined non-verbal communication detected by the detection unit to be displayed at the predetermined position in the predetermined region image in the storage unit.

7. A display method executed by a communication terminal for causing a display unit to display a predetermined region image that is a predetermined region of a wide-view image, comprising: The communication terminal performs: a reception process of receiving the wide-view image transmitted by another communication terminal; a display control process of causing a predetermined icon corresponding to a predetermined non-verbal communication of a person included in an image outside the predetermined region in the wide-view image received by the reception process to be displayed at a predetermined position in the predetermined region image; A display method to be executed.

8. A program for causing a computer to execute the method according to claim 7.

Images