Display terminal, display method, and program

Abstract

To enable a user to understand at which position along a capture device's past movement path video being played back was captured when playing back the video obtained by the capture device recording while moving.SOLUTION: A display terminal displays video obtained from past capture and a map indicating the location of the capture and has a display control unit that displays the past movement path on the map on the basis of the location information during the capture of a capturing device that performed the capture.SELECTED DRAWING: Figure 26

Description

[Technical Field]

[0001] The present disclosure relates to a display terminal, a display method, and a program. [Background technology]

[0002] In recent years, wide-field-of-view images (hereinafter referred to as "wide-field-of-view images") with a wide viewing angle, such as a 360-degree image (also called a celestial sphere image, omnidirectional image, or omnidirectional image) capturing an entire 360-degree surrounding area, have become known as an imaging range that includes areas that cannot be fully confirmed with a normal angle of view. When attempting to display such a wide-field-of-view image in its entirety on a display terminal, the wide-field-of-view image is curved and difficult to see, so a predetermined area image showing a predetermined area in the wide-field-of-view image is displayed on the display terminal, and the user views the predetermined area image.

[0003] Furthermore, it has been proposed to select and play a predetermined image from a plurality of wide-field images that have already been recorded (see Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0004] However, at construction sites, etc., there are cases where images are taken while the imaging device is moving. In this way, when images taken while the imaging device is moving are recorded and then a user plays back and views the recorded images, there is a need to be able to grasp that the displayed image is a predetermined position on the movement route at the construction site, etc.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to enable a user, when playing back a video obtained by recording with a moving imaging device, to understand at what position along the imaging device's past movement path the video being played back was captured. [Means for solving the problem]

[0006] The disclosure of claim 1 is a display terminal that displays video obtained from past shooting and a map showing the location of the shooting, and has a display control unit that displays past travel routes on the map based on location information during shooting of the shooting device that took the shot. [Effects of the Invention]

[0007] As described above, according to the present disclosure, when playing back a video obtained by recording while the imaging device was moving, the user can understand at what position along the imaging device's past movement path the video being played back was obtained by filming it. [Brief explanation of the drawings]

[0008] [Figure 1] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. 1C is a plan view of the imaging device. [Figure 2] FIG. 10 is an image diagram of the imaging device in use. [Figure 3] (a) is a hemispherical image (before) taken with the imaging device, (b) is a hemispherical image (after) taken with the imaging device, and (c) is an image represented by the Mercator projection. [Figure 4] (a) A conceptual diagram showing how a sphere is covered with a Mercator image, and (b) a diagram showing a spherical image. [Figure 5] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. [Figure 6] (a) is a three-dimensional oblique view of Figure 5, (b) is a diagram showing the predetermined area image in the state of (a) displayed on the display, (c) is a diagram showing the predetermined area after changing the viewpoint of the virtual camera IC in (a), and (d) is a diagram showing the predetermined area image in the state of (c) displayed on the display. [Figure 7] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 8] FIG. 10 is a conceptual diagram showing the relationship between a predetermined area and a point of interest. [Figure 9]1 is a schematic diagram of a communication system according to an embodiment. [Figure 10] FIG. 2 is a diagram illustrating a hardware configuration of the imaging device. [Figure 11] FIG. 2 is a hardware configuration diagram of a relay device. [Figure 12] FIG. 2 is a hardware configuration diagram of a communication control system and a communication terminal. [Figure 13] 1 is a diagram illustrating the functional configuration of a communication system according to a first embodiment. [Figure 14] FIG. 10 is a conceptual diagram of a user device management table. [Figure 15] FIG. 10 is a conceptual diagram of a virtual room management table. [Figure 16] FIG. 2 is a conceptual diagram of a location information management table. [Figure 17] FIG. 10 is a sequence diagram showing a communication process of content data in the communication system. [Figure 18] FIG. 10 is a sequence diagram showing a process for starting video and audio recording in a communication system. [Figure 19] FIG. 10 is a sequence diagram showing a process for stopping video and audio recording in a communication system. [Figure 20] FIG. 10 is a sequence diagram showing the process of recording and playing back audio in a communication system. [Figure 21] FIG. 10 is a diagram showing a recording data selection screen. [Figure 22] 10 is a flowchart showing a playback process. [Figure 23] FIG. 10 shows a first display example on a communication terminal 9a, illustrating a video selection screen. [Figure 24] FIG. 10 shows a first display example on a communication terminal 9a, illustrating a video selection screen. [Figure 25] FIG. 10 shows a first display example on a communication terminal 9a, showing a map and video playback screen. [Figure 26] FIG. 10 shows a first display example on a communication terminal 9a, showing a map and video playback screen. [Figure 27] FIG. 10 shows a first display example on a communication terminal 9a, showing a map and video playback screen. [Figure 28] FIG. 10 shows a first display example on a communication terminal 9a, showing a map and video playback screen. [Figure 29] FIG. 10 shows a second display example on the communication terminal 9a, showing a map and video playback screen. [Figure 30] FIG. 10 shows a second display example on the communication terminal 9a, showing a map and video playback screen. [Figure 31] FIG. 10 shows a second display example on the communication terminal 9a, showing a map and video playback screen. [Figure 32] FIG. 10 shows a third display example on the communication terminal 9a, illustrating a moving image selection screen. [Figure 33] FIG. 10 shows a third display example on the communication terminal 9a, showing a map and video playback screen. [Figure 34] FIG. 10 shows a fourth display example on the communication terminal 9a, illustrating a moving image selection screen. [Figure 35] FIG. 10 shows a fourth display example on the communication terminal 9a, showing a map and video playback screen. [Figure 36] FIG. 10 shows a fourth display example on the communication terminal 9a, showing a map and video playback screen. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] [Overview of spherical images] A method for generating a spherical image will be described with reference to Fig. 1 to Fig. 8. A spherical image is also called a spherical panoramic image or a 360° panoramic image, and is an example of a wide-field video with a wide viewing angle. Wide-field images also include simple panoramic images of about 180°.

[0011] First, the appearance of the imaging device 10 will be described using Fig. 1. The imaging device 10 is a digital camera for obtaining captured images that are the basis for creating a spherical image. Fig. 1(a) is a left side view of the imaging device, Fig. 1(b) is a front view of the imaging device, and Fig. 1(c) is a plan view of the imaging device.

[0012] As shown in Fig. 1(a), the photographing device 10 is small enough to be held in one hand. As shown in Figs. 1(a), 1(b), and 1(c), an image sensor 103a is provided on the front side (front side) of the upper portion of the photographing device 10, and an image sensor 103b is provided on the rear side (rear side). As shown in Fig. 1(b), an operation unit 115 such as a shutter button is provided on the front side of the photographing device 10.

[0013] Next, a usage situation of the imaging device 10 will be described with reference to FIG. 2. Note that FIG. 2 is an image diagram of the imaging device in use. As shown in FIG. 2, the imaging device 10 is communicably connected to a relay device 3 installed on some kind of stand 2, and is used to capture surrounding subjects, scenery, and the like. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the imaging element 103a and the imaging element 103b shown in FIG. 1. Note that if the omnidirectional image captured by the imaging device 10 is not to be transmitted to other communication terminals or systems, the relay device 3 is not necessary.

[0014] Next, an outline of processing until a celestial sphere image is created from an image captured by the image capturing device 10 will be described with reference to Figs. 3 and 4. Fig. 3(a) is a diagram showing a hemispherical image (front side) captured by the image capturing device, Fig. 3(b) is a diagram showing a hemispherical image (rear side) captured by the image capturing device, and Fig. 3(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). An image expressed by Mercator projection or the like (hereinafter referred to as "Mercator image") may also be used. Fig. 4(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 4(b) is a diagram showing a celestial sphere image. The "equirectangular projection image" is an equirectangular celestial sphere image as an example of the wide-field-of-view image described above.

[0015] As shown in Fig. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a wide-angle lens 102a such as a fisheye lens, which will be described later. Also, as shown in Fig. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a wide-angle lens 102b such as a fisheye lens, which will be described later. Then, the image capturing device 10 combines the hemispherical image (front side) with a hemispherical image (rear side) flipped 180 degrees to create an equirectangular projection image EC as shown in Fig. 3(c).

[0016] The image capturing device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to apply an equirectangular projection image EC to cover the spherical surface as shown in FIG. 3( a), thereby creating a celestial sphere image CE as shown in FIG. 3( b). In this way, the celestial sphere image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (two-dimensional) and 3D (three-dimensional) data. OpenGL ES is merely an example of software that performs image processing, and the celestial sphere image CE may be created by other software. The celestial sphere image CE may be a still image or a video. While the image capturing device 10 has been described as generating a celestial sphere image, the communication control system 5 or the communication terminals 7 and 9 may perform similar image processing or some of the image processing steps.

[0017] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted onto the surface of a sphere as shown in Fig. 4(a), creating a spherical image as shown in Fig. 4(b). In this way, the spherical image is represented as an image in which the Mercator image faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data.

[0018] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people when they view it. Therefore, the communication terminals 7 and 9 can display a predetermined region (hereinafter referred to as a "predetermined region image"), which is a partial region of the spherical image, as a flat image with little curvature (distortion), thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to FIGS. 5 to 8.

[0019] Fig. 5 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the position of a virtual viewpoint of a user viewing a celestial sphere image CE displayed as a three-dimensional sphere. In Fig. 6, (a) is a three-dimensional perspective view of Fig. 5, (b) is a diagram showing the predetermined area image in the state of (a) displayed on a display, (c) is a diagram showing the predetermined area after the viewpoint of the virtual camera IC in (a) is changed, and (d) is a diagram showing the predetermined area image in the state of (c) displayed on a display.

[0020] If the celestial sphere image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is located inside the celestial sphere image CE as shown in Fig. 5. A predetermined area T in the celestial sphere image CE is an imaging area of ​​the virtual camera IC, and is specified by angle-of-view information (also referred to as "area information") that indicates the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE.

[0021] Furthermore, zooming of the predetermined region T can also be expressed by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined region image Q is an image of the predetermined region T in the celestial sphere image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the celestial sphere image CE.

[0022] Furthermore, when the virtual viewpoint of the virtual camera IC is moved (also referred to as "changed") from the state of Fig. 6(a) to the right (left as one faces the drawing) as shown in Fig. 6(c), the predetermined area T in the omnidirectional image CE is accordingly moved to a predetermined area T', and the predetermined area image Q displayed on the predetermined display is changed to the predetermined area image Q'. As a result, the image shown in Fig. 6(b) is changed to the image shown in Fig. 6(d) and displayed on the predetermined display.

[0023] Next, the relationship between the angle of view information and the image of the predetermined area T will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a diagram showing points in a three-dimensional Euclidean space using spherical coordinates. Fig. 8 is a conceptual diagram showing the relationship between the predetermined area and a point of interest (center point).

[0024] Here, the coordinates of an arbitrary position when the center point CP shown in Fig. 7 is expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius vector, polar angle, and azimuth angle, respectively. The radius vector r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to an arbitrary point (the center point CP in Fig. 8), and is equal to the distance f shown in Fig. 8.

[0025] Furthermore, as shown in FIG. 8, when the center of a predetermined area T, which is the imaging area of ​​the virtual camera IC, is considered to be the center point CP in FIG. 7, the trigonometric function shown in the following (Equation 1) generally holds. (L / f) = tan(α / 2) (Equation 1) Here, f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). α is the angle of view. In this case, the angle of view information for identifying the predetermined area T can be expressed by pan(θ), tilt(φ), and fov(α). Note that zooming of the predetermined area T can be expressed by widening or narrowing the range (arc) of the angle of view α.

[0026] [Communication system overview] Next, an overview of the communication system 1 according to the embodiment will be described with reference to Fig. 9. Fig. 9 is a schematic diagram of the communication system according to the embodiment.

[0027] 9, the communication system 1 of this embodiment is configured with an imaging device 10, a relay device 3, a communication terminal 7, and communication terminals 9a and 9b. The communication terminals 9a and 9b are collectively referred to as "communication terminal 9." The communication terminals 7 and 9 may also be referred to as "display terminals" that display images, etc.

[0028] Of these, the image capturing device 10 is a digital camera for obtaining a wide-field image (such as a spherical image), as described above. The relay device 3 functions as a cradle for charging the image capturing device 10 and transmitting and receiving data. The relay device 3 can perform data communication with the image capturing device 10 via a contact point, and can also perform data communication with the communication control system 5 via the communication network 100. The communication network 100 includes, for example, the Internet, a LAN (Local Area Network), a (wireless) router, etc.

[0029] Furthermore, the communication control system 5 is, for example, a computer, and can perform data communication with the relay device 3 and the communication terminals 7 and 9 via the communication network 100. Note that the communication control system 5 can also be referred to as an "information management system" because it manages information such as angle of view.

[0030] The communication terminals 7 and 9 are, for example, computers such as notebook PCs (Personal Computers), and can perform data communication with the communication control system 5 via the communication network 100. OpenGL ES is installed in the communication terminals 7 and 9, and the communication terminals 7 and 9 create a predetermined area image (see FIG. 6 ) from the omnidirectional image received from the communication control system 5. The communication control system 5 may be configured by a single computer or by multiple computers.

[0031] Furthermore, the imaging device 10 and the relay device 3 are installed at predetermined positions by an organizer X or the like at a site Sa such as a construction site, exhibition hall, educational site, medical site, etc. The communication terminal 7 is operated by the organizer X. The communication terminal 9a is operated by a participant A such as a viewer who is in a remote location from the site Sa. Similarly, the communication terminal 9b is operated by a participant B such as a viewer who is in a remote location from the site Sa. Participant A and participant B may be in the same location or in different locations.

[0032] The communication control system 5 transmits (distributes) the wide-field images obtained from the imaging device 10 via the relay device 3 to the communication terminals 7 and 9. The communication control system 5 also transmits (distributes) the planar images obtained from each communication terminal 7 to the communication terminals 7 and 9. The wide-field images may be either moving images (wide-field moving images) or still images (wide-field still images).

[0033] [Hardware configuration] Next, the hardware configurations of the image capturing device 10, relay device 3, and communication terminals 7 and 9 of this embodiment will be described in detail with reference to FIGS.

[0034] <Hardware configuration of the imaging device> Fig. 10 is a hardware configuration diagram of the photographing device 10. As shown in Fig. 10, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0035] Of these, the imaging unit 101 is equipped with wide-angle lenses 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) that are each capable of capturing an image with a field of view of 180° or more to form a hemispherical image, and two imaging elements 103a and 103b that are provided corresponding to the lenses 102a and 102b, respectively.

[0036] Furthermore, the imaging elements 103a, 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts optical images captured by the lenses 102a, 102b, etc. into image data in the form of electrical signals and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example, and the imaging unit 101 may have only one, or three or more.

[0037] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus).

[0038] The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to a CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a network I / F 121, and the like.

[0039] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create data for an equirectangular projection image (an example of a wide-field image), which will be described later.

[0040] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0041] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function for displaying moving images on a display (for example, a display of an external terminal such as a smartphone that performs short-range communication with the imaging device 10 using the short-range communication circuit 117). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0042] As will be described later, the imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b. Although the imaging device 10 is not provided with a display unit in this embodiment, a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 receives the sound data output from the microphone 108 via an I / F bus and performs predetermined processing on the sound data.

[0043] The CPU 111 controls the overall operation of the image capturing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of processed equirectangular projection images.

[0044] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. A user operates the operation unit 115 to input various imaging modes, imaging conditions, etc.

[0045] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as needed.

[0046] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the image capturing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0047] The electronic compass 118 calculates the direction of the image capturing device 10 from the Earth's magnetism and outputs the direction information. This direction information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data such as the image capture date and time and the data size of the image data.

[0048] The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) that accompany the movement of the image capturing device 10. The changes in angle are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

[0049] The acceleration sensor 120 is a sensor that detects acceleration in three axial directions.

[0050] The photographing device 10 can also calculate the attitude (angle with respect to the direction of gravity) of the own device (the photographing device 10) by using the electronic compass 118, the acceleration sensor 120, etc. Furthermore, by providing the photographing device 10 with the acceleration sensor 120, the accuracy of image correction is improved.

[0051] The network I / F 121 is an interface for performing data communication using a communication network 100 such as the Internet via a router or the like. The hardware configuration of the image capturing device 10 is not limited to that shown here, and any hardware configuration may be used as long as it can realize the functional configuration of the image capturing device 10. At least a part of the hardware configuration may be present on the relay device 3 or the communication network 100.

[0052] <Hardware configuration of relay device> Fig. 11 is a diagram showing the hardware configuration of the relay device 3. Note that Fig. 11 shows the hardware configuration when the relay device 3 is a cradle having a wireless communication function.

[0053] As shown in FIG. 11, the relay device 3 includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a CMOS sensor 305, a bus line 310, a communication unit 313, an antenna 313a, a positioning unit 314, and an input / output I / F 316.

[0054] Of these, the CPU 301 controls the overall operation of the relay device 3. The ROM 302 stores programs such as an IPL (Initial Program Loader) used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301.

[0055] An EEPROM (Electrically Erasable and Programmable ROM) 304 reads or writes data under the control of the CPU 301. The EEPROM 304 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

[0056] The CMOS (Complementary Metal Oxide Semiconductor) sensor 305 is a solid-state image sensor that captures an image of a subject under the control of the CPU 301 and obtains image data.

[0057] The communication unit 313 uses an antenna 313a to communicate with the communication network 100 by wireless communication signals.

[0058] The positioning unit 314 receives a positioning signal including the position information (latitude, longitude, and altitude) of the relay device 3 from a GNSS (Global Navigation Satellite System) satellite such as a GPS (Global Positioning Systems) satellite, or an IMES (Indoor Messaging System) as an indoor GPS.

[0059] The input / output I / F 316 is an interface circuit (such as a USB I / F) electrically connected to the input / output I / F 116 of the image capturing apparatus 10. The input / output I / F 316 may be wireless or wired.

[0060] The bus line 310 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 301.

[0061] <Communication control system and communication terminal hardware configuration> 12 shows the hardware configuration of the communication control system 5. The hardware configuration of the communication terminals 7 and 9 is the same as that of the communication control system 5, and therefore a description thereof will be omitted.

[0062] As shown in FIG. 4, the communication control system 5 is a computer and includes a CPU 501, a ROM 502, a RAM 503, an SSD 504, an external device connection I / F 505, a network I / F 506, a display 507, an operation unit 508, a media I / F 509, a bus line 510, a CMOS sensor 511, a speaker 512, and a positioning unit 514.

[0063] Of these, the CPU 501 controls the overall operation of the communication control system 5. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0064] The SSD 504 reads or writes various data under the control of the CPU 501. If the communication terminals 7 and 9 are smartphones or the like, the SSD 504 may not be provided. Alternatively, a hard disk drive (HDD) may be provided instead of the SSD 504.

[0065] The external device connection I / F 505 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0066] The network I / F 506 is an interface for performing data communication via the communication network 100 .

[0067] The display 507 is a type of display unit such as a liquid crystal display or organic electroluminescence (EL) display that displays various images.

[0068] An operation unit 508 is an input means for selecting and executing various instructions such as various operation buttons, a power switch, a shutter button, and a touch panel, selecting a processing target, moving a cursor, and the like.

[0069] The media I / F 509 controls reading and writing (storing) of data from and to a recording medium 509m such as a flash memory, etc. The recording medium 509m includes DVDs, Blu-ray Discs (registered trademarks), etc.

[0070] The CMOS sensor 511 is a type of imaging means that captures an image of a subject and obtains image data under the control of the CPU 501. A CCD sensor may be used instead of a CMOS sensor.

[0071] The speaker 512 is a circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice.

[0072] The positioning unit 314 receives a positioning signal including position information (latitude, longitude, and altitude) of the communication terminals 7 and 9 from a GNSS satellite such as a GPS satellite, or from IMES as an indoor GPS.

[0073] The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501.

[0074] [Functional configuration of the embodiment] Next, the functional configuration of the first embodiment will be described with reference to FIGS.

[0075] <Functional configuration of the imaging device> 13, the photographing device 10 has a reception unit 12, a detection unit 13, an imaging unit 16, a sound collection unit 17, a connection unit 18, and a memory / readout unit 19. Each of these units is a function or means realized by any of the components shown in FIG. 10 operating in response to an instruction from the CPU 111 in accordance with a photographing / storage program loaded from the SRAM 113 onto the DRAM 114.

[0076] The image capturing device 1 also has a storage unit 1000 constructed by a ROM 112, an SRAM 113, and a DRAM 114 shown in FIG.

[0077] (Functional configuration of the imaging device) The reception unit 12 of the image capturing device 1 is realized by the processing of the operation unit 115 for the CPU 111, and receives operation input from the user.

[0078] The detection unit 13 is mainly realized by processing from the CPU 111 to the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, etc., and obtains attitude information by detecting the attitude of the image capturing device 10.

[0079] The imaging section 16 is mainly realized by processing from the CPU 111 to the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111, and captures images of scenery and the like to obtain captured images.

[0080] The sound collection unit 17 is mainly realized by processing of the sound processing unit 109 from the CPU 111, and collects sounds around the image capturing device .

[0081] The connection unit 18 is mainly realized by processing from the CPU 111 to the input / output I / F 116, and performs data communication with the relay device 3.

[0082] The storage / readout unit 19 is mainly realized by the processing of the CPU 111 , and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000 .

[0083] <Functional configuration of relay device> 13, the relay device 3 has a communication unit 31 and a connection unit 38. These units are functions or means realized when any of the components shown in FIG. 11 operates in response to an instruction from the CPU 301 in accordance with the program for the relay device 3 loaded from the EEPROM 304 onto the RAM 303.

[0084] (Functional configuration of relay device 3) The communication unit 31 of the relay device 3 is mainly realized by processing from the CPU 301 shown in FIG. 11 to the communication unit 313, and performs data communication between the image capturing device 10 and the communication control system 5 via the communication network 100.

[0085] The connection unit 38 is mainly realized by processing from the CPU 301 to the input / output I / F 316, and performs data communication with the image capturing device 10.

[0086] <Functional configuration of the communication control system> Next, each functional configuration of the communication control system 5 will be described in detail with reference to Fig. 13. The communication control system 5 has a communication unit 51, a reception unit 52, a creation unit 53, an authentication unit 55, and a storage / readout unit 59. Each of these units is a function or means realized when any of the components shown in Fig. 12 operates in response to an instruction from the CPU 501 in accordance with the program for the communication control system 5 loaded from the SSD 504 onto the RAM 503.

[0087] The communication control system 5 also has a storage unit 5000 constructed by the RAM 503 and HD 504 shown in Fig. 12. The storage unit 5000 contains a user device management DB 5001, a virtual room management DB 5002, and a location information management DB 5003.

[0088] (User / Device Management DB) Figure 14 is a conceptual diagram of a user device management table. The user device management DB 5001 is composed of the user device management table shown in Figure 14. In the user device table, user IDs (or device IDs), passwords, names, user images, and IP addresses are stored and managed in association with each other.

[0089] Among these, the user ID is an example of user identification information for identifying a user (organizer X, participants A, B, etc.). The device ID is an example of device identification information for identifying a device such as the image capturing device 10. Note that if a head-mounted display or the like is used in addition to the image capturing device 10, the head-mounted display or the like is also treated as a device.

[0090] The name is the name of a user or a device. Each user name may be the name of the communication terminal of each user.

[0091] The user image is a schematic image of each user's face, a photograph of the user's face, or the like, and is registered in advance by each user.

[0092] The IP address is an example of destination identification information for devices such as the communication terminals 7 and 9 and the image capturing device 10 used by the user.

[0093] (Virtual room management DB) Fig. 15 is a conceptual diagram of a virtual room management table. The virtual room management DB 5002 is configured by the virtual room management table shown in Fig. 15. In the virtual room management table, a virtual room ID, a virtual room name, a device ID, a host ID, a participant ID, a content ID, and a content URL (information on the storage location of image and sound content data) are stored and managed in association with each other.

[0094] Of these, the virtual room ID is an example of virtual room identification information for identifying a virtual room.

[0095] The virtual room name is the name of the virtual room and is given by the user or the like.

[0096] The device ID is the same as the device ID in FIG. 14, and is the ID of the device that has participated in the virtual room indicated by the virtual room ID of the same record.

[0097] The host ID is an example of host identification information for identifying the host ID among the user IDs in FIG. 14, and is the ID of the host who participated in the virtual room indicated by the virtual room ID of the same record.

[0098] The participant ID is an example of participant identification information for identifying the participant ID among the user IDs in FIG. 14, and is the ID of a participant who has participated in the virtual room indicated by the virtual room ID of the same record.

[0099] The content ID is an example of content identification information for identifying image and sound content data. In this case, the image is a wide-field image obtained during shooting, and the sound is sound (including voice) obtained during the same shooting.

[0100] The content URL is an example of content storage location information that indicates where the content (wide-field image, sound information) data is stored. The content URL also stores the time when the image was captured (video recorded) and sound was collected (recorded) along with the content data. This time indicates the start and end dates and times of the image capture (video recording) and sound collection (recording).

[0101] (Location information management DB) FIG. 16 is a conceptual diagram of a location information management table. The location information management DB 5003 is configured with the location information management table shown in FIG. 16. In the location information management table, the "shooting and sound collection" date and time and the device location are stored and managed in association with each content ID. If sound collection is not performed, the shooting date and time is displayed. The device location indicates the location at the time the camera device 10 or the communication terminals 7 and 9 took the image. The location of the camera device 10 is measured by the positioning unit 314 of the relay device 3 to which the camera device 10 is attached. The locations of the communication terminals 7 and 9 are measured by the positioning unit 514 of the communication terminals 7 and 9. The camera device 10 may be provided with a positioning unit similar to the positioning unit 314, and the location of the camera device 10 may be measured by this positioning unit.

[0102] The content ID shown in FIG. 16 is the same as the content ID shown in FIG.

[0103] The photographing and sound collection dates and times indicate the dates and times when the photographing device 10 or the communication terminal 7 photographed and collected the sound.

[0104] The device position indicates the position (absolute position on the Earth) of the image capturing device 10 or the communication terminal 7 at the time of image capturing and sound collection.

[0105] (Functional configuration of the communication control system) Next, each functional configuration of the communication control system 5 will be described in detail with reference to FIG.

[0106] The communication unit 51 of the communication control system 5 is mainly realized by processing from the CPU 501 shown in Figure 11 to the network I / F 505, and performs data communication with other devices (relay device 3, communication terminals 7 and 9) via the communication network 100.

[0107] The reception unit 52 is realized by the processing of the operation unit 508 for the CPU 501, and receives operation input from a user (here, a system administrator or the like).

[0108] The creation unit 53 is mainly realized by the processing of the CPU 501, and creates a screen to be sent to each of the communication terminals 7 and 9 using data stored in the storage unit 5000 and the like.

[0109] The authentication unit 55 authenticates whether each user is authorized to use the virtual room.

[0110] The storage / readout unit 59 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000 .

[0111] <Functional configuration of communication terminal 7> Next, the functional configuration of the communication terminal 7 will be described in detail with reference to Fig. 13. The communication terminal 7 has a communication unit 71, a reception unit 72, a display control unit 74, a sound input / output control unit 75, a creation unit 76, a connection unit 78, and a storage / readout unit 79. Each of these units is a function or means realized when any of the components shown in Fig. 12 operates in response to an instruction from the CPU 501 in accordance with the program for the communication terminal 7 loaded from the SSD 504 onto the RAM 503.

[0112] The communication unit 71 of the communication terminal 7 is mainly realized by processing from the CPU 501 shown in FIG. 11 to the network I / F 505, and performs data communication with other devices (communication control system 5) via the communication network 100.

[0113] The reception unit 72 is mainly realized by the processing of the operation unit 508 on the CPU 501, and receives operation input from a user (here, organizer X). The reception unit 72 also functions as an acquisition unit, and when it receives from the user a display of a predetermined area in a wide-field image, it acquires viewpoint information (field angle information) for identifying this predetermined area.

[0114] The display control unit 74 is mainly realized by the processing of the CPU 501, and performs control to display various images on the display 507 of the communication terminal 7 or an external display connected to the external device connection I / F 505.

[0115] The sound input / output control unit 75 is mainly realized by the processing of the CPU 501 of the communication terminal 7, and performs control to collect sound from an external microphone connected to the external device connection I / F 505. If the communication terminal 7 has a built-in microphone, the sound input / output control unit 75 performs control to collect sound from the microphone. The sound input / output control unit 75 also performs control to output sound to the speaker 512 of the communication terminal 7 or an external speaker connected to the external device connection I / F 505.

[0116] The creation unit 76 is mainly realized by the processing of the CPU 501, and adds narration, subtitles, etc. to the content data recorded and sounded by the communication terminal 7 to create content data for use as teaching materials, etc.

[0117] The storage / readout unit 79 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 7000 and reads out various data (or information) from the storage unit 7000 .

[0118] <Functional configuration of communication terminal 9> Next, each functional configuration of the communication terminal 9 will be described in detail with reference to FIG.

[0119] The communication terminal 9 has a communication unit 91, a reception unit 92, a display control unit 94, a sound input / output control unit 95, a connection unit 98, and a storage / readout unit 99. Each of these units is a function or means realized when any of the components shown in Fig. 12 operates in response to an instruction from the CPU 501 in accordance with a program for the communication terminal 9 that has been loaded from the SSD 504 onto the RAM 503.

[0120] The communication terminal 9 also includes a storage unit 9000 constructed by the RAM 503 and the SSD 504 shown in FIG.

[0121] The communication unit 91 of the communication terminal 9 is mainly realized by processing from the CPU 501 to the network I / F 505, and performs data communication with other devices (communication control system 5) via the communication network 100.

[0122] The reception unit 92 is mainly realized by the processing of the operation unit 508 on the CPU 501, and receives operation input from a user (here, a participant). The reception unit 92 also serves as an acquisition unit, and when it receives a display of a predetermined area in a wide-field image from a user, it acquires viewpoint information (field angle information) for identifying this predetermined area.

[0123] The display control unit 94 is mainly realized by the processing of the CPU 501, and performs control to display various images on the display 507 of the communication terminal 9 or an external display connected to the external device connection I / F 505.

[0124] The sound input / output control unit 95 is mainly realized by the processing of the CPU 501 of the communication terminal 9, and performs control to collect sound from an external microphone connected to the external device connection I / F 505. If the communication terminal 7 has a built-in microphone, the sound input / output control unit 95 performs control to collect sound from the microphone. The sound input / output control unit 95 also performs control to output sound to the speaker 512 of the communication terminal 9 or an external speaker connected to the external device connection I / F 505.

[0125] The connection unit 98 is mainly realized by processing from the CPU 501 to the external device connection I / F 505, and performs data communication with external devices connected by wire or wirelessly.

[0126] The storage / readout unit 99 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000 .

[0127] [Processing or Operation of the Embodiment] Next, the processing or operation of the embodiment will be described with reference to Figures 17 to 36. Note that the following processing is performed after the image capturing device 10 and the communication terminals 7 and 9 have already joined the same virtual room.

[0128] <Communication processing of content data in a communication system> First, the communication processing of content data in the communication system 1 will be described with reference to FIG. 17. FIG. 17 is a sequence diagram showing the communication processing of a wide-field image and each piece of field angle information in the communication system. In this embodiment, a case will be described in which the image capturing device 10, the communication terminal 7 of the organizer X, the communication terminal 9a of the participant A, and the communication terminal 9b of the participant B are in the same virtual room. Furthermore, when the virtual room is constructed, the storage / readout unit 79 adds one record to the virtual room management DB (see FIG. 15) and manages the virtual room ID, virtual room name, device ID, organizer ID, and participant ID in association with each other. The content ID, content URL, and field angle information URL will be stored later. Note that the processes S11 to S15 in FIG. 17 are repeated, for example, 30 or 60 times per second.

[0129] S11: In the image capturing device 10, the imaging unit 16 captures spherical images of the area Sa and collects sound to obtain content (wide-field image, sound information) data, and then the connection unit 18 sends the content data to the relay device 3. In this case, the connection unit 18 also sends a virtual room ID for identifying the virtual room in which the image capturing device 10 is participating and a device ID for identifying the image capturing device 10. As a result, the connection unit 38 of the relay device 3 obtains the content data, the virtual room ID, and the device ID.

[0130] S12: In the relay device 3, the communication unit 31 transmits the content data, the virtual room ID, and the device ID acquired by the connection unit 38 in process S11 to the communication control system 5 via the communication network 100. As a result, in the communication control system 5, the communication unit 51 receives the content data, the virtual room, and the device ID.

[0131] The image capturing device 10 may transmit the content data, the virtual room ID, and the device ID to the communication terminal 7 instead of the relay device 3 (S11d). In this case, the communication terminal 7 transmits the content data, the virtual room ID, and the device ID to the communication control system 5 (S12d).

[0132] S13: In the communication control system 5, the storage / readout unit 59 searches the virtual room management DB 5002 based on the virtual room ID received in step S12 to read out the user IDs (organizer ID and participant IDs) of users participating in the same virtual room as the image capturing device 10. The storage / readout unit 59 also searches the user / device management DB 5001 based on the readout organizer ID and participant ID to read out the corresponding user image of the organizer X and the IP address of the communication terminal 7, as well as the user images of the corresponding participants A and B and the IP addresses of the communication terminals 9a and 9b. The communication unit 51 then refers to the IP address of the communication terminal 7 and transmits the content data received in step S12 to the communication terminal 7. The communication unit 71 of the communication terminal 7 receives the content data. At this time, the communication unit 51 may transmit the user images and user IDs of the users participating in the same virtual room in an associated state to the communication terminal 7.

[0133] S14: The communication unit 51 of the communication control system 5 references the IP address of the communication terminal 9a and transmits the content data received in step S12 to the communication terminal 9a. As a result, the communication unit 91 of the communication terminal 9a receives the content data. At this time, the communication unit 51 may transmit the content data to the communication terminal 9a in a state in which the user images and user IDs of the users participating in the same virtual room are associated with each other.

[0134] S15: Similarly, the communication unit 51 of the communication control system 5 references the IP address of the communication terminal 9b and transmits the content data received in step S12 to the communication terminal 9b. As a result, the communication unit 91 of the communication terminal 9b receives the content data. At this time, the communication unit 51 may transmit the content data to the communication terminal 9b in a state in which the user images and user IDs of the users participating in the same virtual room are associated with each other.

[0135] Through the above processing, for example, in communication terminal 9a, the display control unit 94 displays a predetermined area image (see FIG. 6(b)) showing a predetermined area (see FIG. 6(a)) of the wide-field image received in processing S14, and the sound input / output control unit 95 outputs sound based on the sound information received in processing S14. Furthermore, when the reception unit 92 receives a screen operation from participant A, the display control unit 94 changes the predetermined area T (see FIG. 6(a)) and displays a predetermined area image (see FIG. 6(d)) showing a predetermined area T' (see FIG. 6(c)) in which an object of interest to participant A is displayed.

[0136] <Process for starting video and audio recording in a communication system> Next, the process of starting video and audio recording in the communication system 1 will be described with reference to Fig. 18. Fig. 18 is a sequence diagram showing the process of starting video and audio recording in the communication system.

[0137] S31: First, in the communication terminal 7 of the organizer X, the reception unit 72 receives an operation from the organizer X to start video and audio recording.

[0138] S32: Before starting video and audio recording in the communication terminal 7, the communication unit 71 transmits a command to share the field of view information to the communication control system 5. This command includes the virtual room ID of the virtual room in which the communication terminal 7 is participating and the device ID of the image capturing device 10. As a result, the communication unit 51 of the communication control system 5 receives the command to share the field of view information.

[0139] S33: In the communication control system 5, the storage / readout unit 59 sets the content URL and the field of view information URL in the virtual room management DB (see FIG. 15). Then, the communication unit 51 transmits to the communication terminal 7 an instruction to start recording and a request to upload the field of view information. This instruction includes information indicating a content URL that indicates a location where the communication terminal 7 will save the content data after recording. This request also includes information indicating a field of view information URL for maintaining the field of view information. As a result, in the communication terminal 7, the communication unit 71 receives the instruction to start recording and the request to upload the field of view information.

[0140] S34: The communication unit 51 also transmits a request to upload the angle of view information to the communication terminal 9a. This request includes information about a URL for saving the angle of view information. As a result, the communication unit 91 of the communication terminal 9a receives the request to upload the angle of view information.

[0141] S35: Similarly, the communication unit 51 transmits a request to upload the angle of view information to the communication terminal 9b. This request includes information on a URL for saving the angle of view information. As a result, the communication unit 91 of the communication terminal 9b receives the request to upload the angle of view information.

[0142] S36: Next, in the communications terminal 7, the storage / readout unit 79 functions as a video recording unit and a sound recording unit, and starts recording and sound recording of the content data received in process S13 shown in Fig. 17. Note that in the case of process S12d shown in Fig. 17, the communications terminal 7 may start recording and sound recording of the content data received from the imaging device 10 in process S11d, instead of the content data received from the communications control system 5 in process S13.

[0143] S37: In the communication terminal 7, for example, while displaying a predetermined area image (see FIG. 6(b)), which is a predetermined area (see FIG. 6(a)) of the wide-field image received in process S13, if the reception unit 72 receives a request to change the angle of view from the organizer X, the display control unit 74 displays a predetermined area image (see FIG. 6(d)), which is a changed predetermined area (see FIG. 6(c)) of the same wide-field image. In this case, the reception unit 72 also functions as an acquisition unit, and when receiving a request to display a predetermined area in the wide-field image from a user (here, organizer X), acquires angle-of-view information (pan, tilt, fov) for specifying the predetermined area in the wide-field image to be displayed on the display 507. Then, the communication unit 71 transmits the angle-of-view information for specifying the changed predetermined area to the image information URL (communication control system 5) received in process S33. This angle-of-view information includes the user ID of the organizer X of the communication terminal 7, which is the sender. As a result, in the communication control system 5, the communication unit 51 receives the angle-of-view information. Then, the storage / readout unit 79 stores the user ID, the IP address of the sender, the angle of view information, and a timestamp in the angle of view information management DB (see FIG. 16). This timestamp indicates the time when the angle of view information was received in step S37.

[0144] S38: A process similar to the process S37 is performed independently of the process S37 in the communication terminal 9a and the communication control system 5. The user ID transmitted in this case is the user ID of participant A.

[0145] S39: A process similar to the process S37 is performed independently of the processes S37 and S38 in the communication terminal 9b and the communication control system 5. Note that the user ID sent in this case is the user ID of participant B.

[0146] The processes S37 to S39 may be executed together with the communication control system 5 when the recording is completed.

[0147] <Processing for stopping video and audio recording in a communication system> Next, a process for stopping video and audio recording in the communication system 1 will be described with reference to Fig. 19. Fig. 19 is a sequence diagram showing a process for stopping video and audio recording in the communication system.

[0148] S51: First, in the communication terminal 7 of the organizer X, the reception unit 72 receives an operation from the organizer X to stop video and audio recording.

[0149] S52: The storage / readout unit 79 stops recording the content data.

[0150] S53: The communication unit 71 uploads (transmits) the video and audio recording content data to the specified content URL (communication control system 5) received in step S33. This content data includes the time (timestamp) from the start to the end of the video and audio recording. As a result, in the communication control system 5, the communication unit 51 receives the content data.

[0151] S54: In the communication control system 5, the storage and reading unit 59 stores the content data together with the timestamp in a predetermined content URL. Furthermore, the storage and reading unit 59 converts the timestamp managed in the field of view information management DB (see FIG. 16) into an elapsed playback time in accordance with the total recording time of the content data whose recording has been stopped.

[0152] S55: The communication unit 51 transmits a video and audio end notification to the communication terminal 7. The end notification includes information indicating a specific content URL. As a result, the communication unit 71 of the communication terminal 7 receives the video and audio end notification.

[0153] S56: Similarly, the communication unit 51 transmits a video and audio end notification to the communication terminal 9a. The end notification includes information indicating a specific content URL. As a result, the communication unit 91 of the communication terminal 9a receives the video and audio end notification.

[0154] S57: Similarly, the communication unit 51 transmits a video and audio end notification to the communication terminal 9b. The end notification includes information indicating a specific content URL. As a result, the communication unit 91 of the communication terminal 9b receives the video and audio end notification.

[0155] In the case of process S55, the completion notification does not need to include the predetermined content URL.

[0156] <Recording and playback of recorded data in a communication system> Next, the process of recording and playing back audio data in a communication system will be described with reference to Fig. 20 to Fig. 26. Fig. 20 is a sequence diagram showing the process of recording and playing back audio data in a communication system. Fig. 21 is a diagram showing a recording data selection screen. Here, participant A uses communication terminal 9a to play back the recorded content data.

[0157] S71: First, when the reception unit 92 of the communication terminal 9a receives a login operation from user A by inputting a user ID, password, etc., the communication unit 91 transmits a login request to the communication control system 5. This request includes user A's user ID and password. As a result, in the communication control system 5, the communication unit 51 receives the login request, and the authentication unit 55 performs authentication by referring to the user device management DB (see FIG. 14). The following explanation will be given assuming that user A is determined to be a legitimate accessing user through login authentication.

[0158] S72: In the communication control system 5, the creation unit 53 creates a recording data selection screen 940 as shown in FIG. 21. In this case, the storage / reading unit 59 searches the virtual room management DB (see FIG. 15) using the user ID received in process S71 as a search key, thereby reading out all corresponding virtual room IDs, virtual room names, and content URLs. The creation unit 53 then creates thumbnails 941, 942, and 943 using images from each piece of content data (with timestamp) stored in the content URL. As a result, the creation unit 53 adds, for each thumbnail, a virtual room name (such as "Construction Site α") and a recording time (such as "2022 / 10 / 31 15:00") indicating a predetermined time in the timestamp (for example, the recording start time).

[0159] S73: The communication unit 51 transmits the selection screen data created in step S72 to the communication terminal 9a. In this selection screen data, each thumbnail includes a content ID for identifying the wide-field image from which each thumbnail was created. As a result, the communication unit 91 of the communication terminal 9a receives the selection screen data.

[0160] S74: In the communication terminal 9a, the display control unit 94 displays a recording data selection screen as shown in Fig. 21 on the display 507 of the communication terminal 9a. Then, the reception unit 92 receives a designation (selection) of a predetermined thumbnail from the participant A. Here, the description will continue assuming that the thumbnail 941 is designated (selected).

[0161] S75: The communication unit 71 transmits a download request to the communication control system 5 for the content data that is the creation source of the selected thumbnail 941. This request includes the content ID associated with the thumbnail 941. As a result, the communication unit 51 of the communication control system 5 receives the download request for the content data.

[0162] S76: In the communication control system 5, the storage and reading unit 59 searches the virtual room management DB (see FIG. 15) using the content ID received in process S75 as a search key, and reads out content data from the corresponding content URL. This content data also includes a map, which will be described later. The storage and reading unit 59 also searches the location information management DB (see FIG. 16) using the content ID received in process S75 as a search key, and reads out the corresponding shooting and sound collection dates and times and the location of the shooting device. The storage and reading unit 59 also searches the virtual room management DB (see FIG. 15) using the content ID received in process S75 as a search key, and reads out the corresponding device ID or user ID. The storage and reading unit 59 then searches the user device management DB (see FIG. 14) using the read device ID or user ID as a search key, and reads out the corresponding name. This name is the name of the photographing device 10, which is the photographing device, and the name of the photographer who took the photograph using the communication terminal 7.

[0163] The communication unit 51 then transmits the requested content data, along with the date and time of shooting and sound collection, the location of the shooting device, and the name of the shooting device (or photographer), to the communication terminal 9a. As a result, the communication unit 91 of the communication terminal 9a receives the information on the content data, the date and time of shooting and sound collection, the location of the shooting device, and the name of the shooting device (or photographer).

[0164] S77: The communication terminal 9a performs a playback process. That is, the display control unit 94 of the communication terminal 9a displays a screen including the recorded image on the display 507 of the communication terminal 9a, and the sound input / output control unit 95 outputs sound.

[0165] <Details of the regeneration process> Next, the process of displaying a screen including an image, which is part of the playback process of step S77, will be described in detail with reference to Fig. 22 to Fig. 36. Fig. 22 is a flowchart showing the playback process in the communication terminal 9a. Fig. 23 to Fig. 36 show examples of display on the communication terminal 9a.

[0166] (First display example) First, a first display example will be described with reference to Fig. 22 to Fig. 28. Fig. 22 is a flowchart showing playback processing. Fig. 23 and Fig. 24 show the first display example on the communication terminal 9a, and are diagrams showing a video selection screen. Fig. 25 to Fig. 28 show the first display example on the communication terminal 9a, and are diagrams showing a map and video playback screen.

[0167] S111: First, the reception unit 72 receives a request to display a video selection screen from participant A. As a result, the display control unit 94 displays a video selection screen 600 shown in Fig. 23 on the display 507 of the communication terminal 9a. As shown in Fig. 23, the video selection screen 600 displays a "video selection" menu 601 for selecting a video, a map 607, and a close button 609 for the video selection screen 600. Note that the video selection screen 600 displays a map 607 corresponding to the recorded data selected on the recorded data selection screen 940 shown in Fig. 21.

[0168] S112: When participant A presses the "video selection" menu 601 with cursor c1, the display control unit 94 displays a pull-down menu 610 for selecting a video, as shown in FIG. 24. The pull-down menu 610 displays the shooting date (e.g., 2023 / 11 / 11), shooting start time (e.g., 13:55:34), and shooting end time (e.g., 14:10:00) for each "device or photographer" that shot the video. The pull-down menu 610 also displays check boxes 611, 612, 613, and 614 for accepting selections for each device or photographer that shot the video. The pull-down menu 610 also displays an "OK" button 619 that is pressed to confirm the selection. Here, when user A selects a predetermined check box 611 and presses the "OK" button 619, the reception unit 92 accepts the selection of a predetermined video (2023 / 11 / 11 13:55:34-14:10:00) that was previously captured by a predetermined imaging device (imaging device α).

[0169] S113: The display control unit 94 displays a map and video playback screen 650 including a past movement path (also referred to as a "trajectory") 701, as shown in FIG. 25, based on the position (position information) of the imaging device during imaging acquired in process S76. FIG. 25 shows a first display example of the communication terminal 9a, illustrating the map and video playback screen. The map and video playback screen 650 displays a "video selection" menu 601 and a close button 659 for the map and video playback selection screen 650. The map and video playback screen 650 also includes a map display area 700 and a video display area 710. The map display area 700 displays a past movement path 701 traveled by the specific imaging device selected in process S112 during imaging. Note that an "S" is displayed at the start of the movement path 701, and a "G" is displayed at the goal of the movement path 701. The movement path 701 also displays multiple arrows indicating the direction of past movement. Furthermore, below the map display area 700, the name of the selected imaging device (here, imaging device α), a seek bar (also called a "playback bar") 651, and a slider s1 in the seek bar 651 are displayed. The seek bar 651 indicates the total playback time of the video being played in the video display area 710. The slider s1 in the seek bar 651 indicates the elapsed playback time of the video being played in the video display area 710. Conversely, when the position of the slider s1 in the seek bar 651 is changed, the elapsed playback time of the video being played in the video display area 710 also changes in accordance with this position.

[0170] Also, a mark m1 is displayed inside the moving image display area 710. The mark m1 indicates that the user (here, participant A) can change the predetermined area image displayed in the moving image display area 710 (see FIGS. 6(b) to 6(d)) by changing the predetermined area in the wide-field image (see FIGS. 6(a) to 6(c)).

[0171] Furthermore, the same name as the selected imaging device (imaging device α) is also displayed at the top of the moving image display area 710. At the bottom of the moving image display area 710, a play button 655 for starting playback of the moving image in the moving image display area 710 and a play stop button 656 for pausing playback of this moving image are displayed.

[0172] In FIG. 25, the seek bar 651 indicates the initial state of the action playback (the playback time is 0 seconds).

[0173] S114: When participant A presses the play button 655 shown in FIG. 25 with the cursor c1, the accepting unit 92 of the communication terminal 9a accepts the start of playback of the video, and the display control unit 94 displays the video captured by the imaging device (here, imaging device α) in the video display area 710. Then, as shown in FIG. 26, the display control unit 94 displays a thumbnail t1 related to the video on the movement path 701 in the map display area 700 in accordance with (according to) the elapsed playback time of the video. Note that the display control unit 94 may display the user image shown in FIG. 4 instead of the thumbnail t1 related to the video. In this case, the communication terminal 9a also receives data of the user image in process S76 shown in FIG. 20. The display control unit 94 also changes the position (length) of the slider s1 in accordance with the elapsed playback time of the video.

[0174] This allows participant A to associate and grasp the video captured by the imaging device, the positions where the video was captured by the imaging device along the previous movement route, and the playback progress position on the seek bar 651.

[0175] S115: In Fig. 27, when user A moves the cursor from the position of cursor c1' to the position of cursor c1 in the video display area 710, the reception unit 92 receives a change (movement) of the virtual viewpoint (see Fig. 6(c)). As a result, the display control unit 94 can change the image of the predetermined area in the video display area 710 by moving the virtual viewpoint in accordance with the movement of cursor c1 (see Fig. 6(d)).

[0176] S116: As the video continues to be played, the display control unit 94 displays a video of a predetermined elapsed playback time in the video display area 710, as shown in FIG. 28. The display control unit 94 also moves the thumbnail t1 in the map display area 700 in accordance with the position of the imaging device at the predetermined elapsed playback time. In this case, the thumbnail t1 has been changed to content related to the video currently being displayed. Furthermore, the display control unit 94 moves (changes) the slider s1 in the seek bar 651 to the position at the predetermined elapsed playback time.

[0177] As described above, the display control unit 94 can display, in conjunction with (synchronized with) each other, the thumbnail t1 indicating the position of a predetermined image capturing device at a predetermined elapsed playback time along the movement path 701 that the predetermined image capturing device moved during image capturing, the seek bar 651 visually indicating the predetermined elapsed playback time, and the display of the video at the predetermined elapsed playback time. This allows a user such as participant A to know at what position along the past movement path of the image capturing device the image being played back was captured.

[0178] (Second display example) Next, a second display example will be described with reference to Fig. 29 to Fig. 31. Fig. 29 to Fig. 31 show the second display example on the communication terminal 9a, and are diagrams showing a map and video playback screen.

[0179] In FIG. 25 (process S113), when participant A moves the cursor c1 to a predetermined position of interest in the map display area 700 and clicks (specifies) it, as shown in FIG. 29, the reception unit 92 accepts the specification of the predetermined position in the map display area 700. As a result, as shown in FIG. 30, the display control unit 94 displays in white the portion of the seek bar 651 corresponding to the predetermined movement path within a predetermined range based on the predetermined position in the movement path 701, and in black the portion of the seek bar 651 corresponding to other movement paths. "Within a predetermined range based on the predetermined position" refers to, for example, a circular area with a radius of 3 m centered on the predetermined position, a square area with sides of 5 m centered on the predetermined position, etc. White indicates the range in which the slider s1 can move, and black indicates the range in which the slider s1 cannot move. Note that using different colors is an example of changing the display format. Changing the display format also includes changing the shape, pattern, lighting timing, etc. in addition to color.

[0180] Here, when playback of the video starts, in process S116, the display control unit 94 moves the slider s1 within its movable range, and synchronizes the displays of the video display area 710 and the map display area 700 with the movement of the slider s1, as shown in Fig. 31. In Fig. 31, the portion of the seek bar 651 to which the slider s1 has moved is displayed in gray.

[0181] As described above, when participant A specifies a predetermined position on the map, the display control unit 94 can narrow down the videos displayed to those around the specified position in the seek bar 651, and disable display of any other videos. This allows participant A to view videos that are narrowed down to those around the position that interests him / her, and therefore he / she can efficiently search for and view videos of noteworthy locations such as damaged areas.

[0182] (Third display example) Next, a third display example will be described with reference to Fig. 32 and Fig. 33. Fig. 32 shows the third display example on the communication terminal 9a, and is a diagram showing a video selection screen. Fig. 33 shows the third display example on the communication terminal 9a, and is a diagram showing a map and video playback screen.

[0183] 24 (process S112), when participant A selects check box 612 in addition to selecting check box 611 in pull-down menu 610 and presses "OK" button 619 as shown in FIG. 32, the reception unit 92 accepts the selection of a predetermined video (2023 / 11 / 11 13:55:40-14:10:00) captured by a predetermined photographer (organizer x). Note that, as shown in FIG. 32, check box 612 of a video whose recording time overlaps with the recording time (from the start to the end of recording) of an already selected video can be selected, and check boxes 613 and 614 of videos that do not overlap cannot be selected.

[0184] As a result, the display control unit 94 displays a map display area 700 and action display areas 710s and 720s on the map / video playback screen 650, as shown in FIG. 33. The video display area 710s is a smaller version of the video display area 710 shown in FIG. 25, and displays a video captured by the camera α (an example of a first camera device). The video display area 720s is an area of ​​the same size as the video display area 710s, and displays a video captured by the communication terminal 7 of the organizer x (an example of a second camera device). However, unlike the camera 10, the communication terminal 7 cannot capture a wide-field image by shooting, and therefore cannot move the virtual viewpoint. Therefore, the mark m1 is not displayed in the video display area 720s. Note that by pressing the play button 655, the video display areas 710s and 720s can be played back (displayed) simultaneously and synchronized. Similarly, pressing the pause button 656 allows the playback of the moving image display areas 710s and 720s to be paused simultaneously and in synchronization.

[0185] The map display area 700 also displays a movement route 701 (solid line portion; an example of a first movement route) traveled by the image capturing device α selected in Fig. 32 while capturing (recording), and a movement route 702 (dashed line portion; an example of a second movement route) traveled by the communication terminal 7 of the organizer x selected in Fig. 32 while capturing (recording). A thumbnail t1 related to the video displayed in the video display area 710s is displayed on the movement route 701, and a thumbnail t2 related to the video displayed in the video display area 720s is displayed on the movement route 702. In this case, the thumbnail t2 may also display a user image shown in Fig. 14 instead of one related to the video.

[0186] Furthermore, the name of the selected imaging device (here, imaging device α), a seek bar 651, and a slider s1 are displayed below the map display area 700. Similarly, the selected photographer (here, organizer x), a seek bar 655, and a slider s2 are displayed below the map display area 700. The seek bar 652 and slider s2 have the same display form and function as the seek bar 651 and slider s1, respectively. That is, the seek bar 652 indicates the total playback time of the video played in the video display area 720s. The slider s2 in the seek bar 652 indicates the elapsed playback time of the video played in the video display area 720s. Conversely, when the position of the slider s2 is changed in the seek bar 652, the elapsed playback time of the video played in the video display area 720s also changes accordingly.

[0187] The number of video display areas may be three or more. In this case, three or more travel routes and thumbnails are displayed in the map display area 700, and three or more seek bars are also displayed.

[0188] As described above, the display control unit 94 simultaneously and synchronously displays the multiple movement paths 701, 702, the multiple thumbnails t1, t2, the multiple seek bars 651, 652, and the multiple video display areas 710s, 720s, allowing users such as participant A to view and compare the videos, etc. This reduces the risk of overlooking video of problematic locations such as damaged areas.

[0189] (Fourth display example) Next, a fourth display example will be described with reference to Fig. 34 to Fig. 36. Fig. 34 shows the fourth display example on the communication terminal 9a, and is a diagram showing a video selection screen. Fig. 35 and Fig. 36 show the fourth display example on the communication terminal 9a, and are diagrams showing a map and video playback screen.

[0190] In addition to the video selection screen 600 shown in FIG. 23, a "select object" menu 602 is displayed on the video selection screen 800 shown in FIG.

[0191] Here, in process S112, when participant A presses the “Object Selection” menu 602 with cursor c1, the display control unit 94 displays a pull-down menu 620 for selecting an object that will appear in the video (image), as shown in FIG. 34. The pull-down menu 620 displays the names of objects (object names) that may appear in the video. Furthermore, check boxes 621, 622, 623, and 624 for accepting the selection are displayed for each object name. The pull-down menu 620 also displays an “OK” button 629 that is pressed to confirm the selection. The selectable object names may be automatically extracted in advance by the communication control system 5 by performing object recognition on the recorded video, or may be extracted and manually set by a person who views the recorded video. Here, when user A selects the check box 621 and presses the “OK” button 629, the accepting unit 92 accepts the selection of a predetermined object (e.g., a barricade).

[0192] Then, in process S113, the display control unit 94 displays a portion 701a of the movement path 701 in which the predetermined object selected in the selected video is likely to appear thicker than the other portions, as shown in Fig. 35. Note that varying the thickness is an example of varying the display form. Differentiating the display form also includes, in addition to the thickness, varying the shape other than the thickness (such as a solid line and a dashed line of the same thickness), color, pattern, lighting timing, etc.

[0193] The reason why it is said that "there is a possibility that it may be displayed" is that even if it is not displayed in the video display area 710, if the selected specific object is included in the wide-field image, participant A can change the virtual viewpoint as in process S115 and display it in the video display area 710.

[0194] In this case, the display control unit 94 includes an object recognition function, which uses techniques such as those disclosed in the following reference documents 1 to 3. <Reference 1>YOLO (https: / / docs.ultralytics.com / ) <Reference 2> deepface (https: / / github.com / serengil / deepface) <Reference 3>PoseNet (https: / / www.tensorflow.org / lite / examples / pose_estimation / overview?hl=ja) Here, the portion 701a is displayed thicker than the other portions of the movement path 701, but the shape and color may be changed.

[0195] 35, the display control unit 94 displays in white the seek bars 651 corresponding to the predetermined paths along which the selected object may be displayed, and displays the rest in black on the movement path 701. White indicates the range within which the slider s1 can move, and black indicates the range within which the slider s1 cannot move.

[0196] Here, when playback of the video starts, in process S116, the display control unit 94 moves the slider s1 within its movable range, and interlocks the displays of the video display area 710 and the map display area 700 with the movement of the slider s1, as shown in Fig. 31. In Fig. 36, the portion of the seek bar 651 to which the slider s1 has moved is displayed in gray.

[0197] As described above, when participant A selects an object, the display control unit 94 can narrow down the videos displayed in the seek bar 651 to those that may show the selected object, and disables the display of any other videos. This allows participant A to narrow down the videos he or she views to the object that interests him or her, and therefore he or she can efficiently search for and view videos of objects that require attention, such as damaged areas.

[0198] [Major Effects of the Embodiments] As described above, according to this embodiment, when playing back a video obtained by a moving imaging device, a user such as participant A can understand at what position on the imaging device's past movement path the video being played back was obtained by shooting the video.

[0199] 〔supplement〕 Although the embodiments have been described above, the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the scope of the present invention.

[0200] (1) Each function of each of the above embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each of the above-described functions.

[0201] (2) A (non-transitory) recording medium such as a DVD-ROM on which each of the above programs is stored can be provided domestically or internationally as a program product.

[0202] (3) There may be multiple CPUs 111, 301, 501, and 801 as hardware processors. [Explanation of symbols]

[0203] 1. Communication Systems 3. Relay Device 5. Communication control system (also referred to as information management system or image management system) 7. Communication terminal (example of display terminal) 9, 9a, 9b Communication terminal (an example of a display terminal) 10 Imaging equipment 51 Communication unit (an example of a transmission unit, an example of a reception unit) 52 Reception 53 Creation Department 71 Communication unit (an example of a transmission unit, an example of a reception unit) 72 Reception unit (also referred to as acquisition unit) 74 Display control unit 75 Sound input / output control section 76 Creation Department 78 Connection 91 Communication unit (an example of a transmission unit, an example of a reception unit) 92 Reception unit (also referred to as acquisition unit) 94 Display control unit 95 Sound input / output control section 98 Connection 507 Display (an example of a display unit) 807 Display (Example of display unit) 5001 User Device Management DB 5002 Virtual Room Management DB 5003 Location information management DB [Prior art documents] [Patent documents]

[0204] [Patent Document 1] Japanese Patent Publication No. 2022-46997

Claims

1. A display terminal that displays a video obtained by past shooting and a map showing the location of the shooting, A display terminal having a display control unit that displays a past travel route on the map based on position information during shooting of the shooting device that took the image.

2. The display terminal according to claim 1 , wherein the display control unit moves and displays thumbnails relating to the video on the movement route in accordance with an elapsed playback time of the video.

3. The display terminal according to claim 2 , wherein the display control unit displays a seek bar indicating the elapsed playback time.

4. 2. The display terminal according to claim 1, a reception unit that receives a designation of a predetermined position on the map; The display control unit displays a narrowed-down video image obtained by the imaging device moving within a predetermined range based on the predetermined position during the imaging process.

5. The display terminal according to claim 4 , wherein the display control unit displays a seek bar in a different display format for the elapsed playback time of the narrowed-down video and the elapsed playback time of a video other than the narrowed-down video.

6. The display terminal according to claim 1, wherein the display control unit displays on the map a first movement route taken by a first photographing device as the photographing device that took the photograph, and a second movement route taken by a second photographing device as the photographing device that took the photograph.

7. The display terminal of claim 6, wherein the display control unit moves and displays a first thumbnail related to the first video on the first movement path in accordance with the elapsed playback time of the first video obtained by shooting with the first imaging device, and moves and displays a second thumbnail related to the second video on the second movement path in synchronization with the playback of the first video in accordance with the elapsed playback time of the second video obtained by shooting with the second imaging device.

8. 2. The display terminal according to claim 1, a receiving unit that receives a selection of a predetermined object that can be displayed in the video; The display control unit narrows down the videos to be displayed to those with a playback elapsed time that can show the specified object.

9. The display terminal according to claim 8, wherein the display control unit displays a predetermined route indicating a movement route of the imaging device when the narrowed-down video was obtained by the imaging in a different display form from routes other than the predetermined route.

10. The display terminal according to claim 1 , wherein the moving image is a predetermined area image that shows a predetermined area of ​​a wide-field image.

11. A display method executed by a display terminal that displays a video captured in the past and a map showing the location of the video, comprising: A display method including a process of displaying a past travel route on the map based on position information of the imaging device that took the image during imaging.

12. A program that causes a computer to display a video obtained by past photography and a map showing the location of the photography, A program that causes the computer to execute a process of displaying a past travel route on the map based on position information of the imaging device that took the image.

Images