Information processing apparatus, screen creation method, program, and information processing system

Abstract

To improve convenience for a user by complementing a photographed image obtained through photographing.SOLUTION: An information processing apparatus has a creation unit that creates a screen including a first photographed image display area in which a first predetermined area image is displayed which is a first predetermined area in a first photographed image obtained by photographing an object at a first photographing position by using a photographing device, and a three-dimensional image display area in which at least part of a three-dimensional image aligned to the first photographed image is displayed and a photographing position image is displayed which indicates a position corresponding to a second photographing position on the predetermined photographing date. The creation unit includes, in the screen, a second photographed image display area in which a second predetermined area image and a specific image are displayed in a superimposed manner, the second predetermined area image being stored in association with the second photographing position and being a second predetermined area in a second photographed image at the predetermined photographing date associated with the second photographing position, the specific image indicating, on the basis of a specific area specified from the second photographed image, the position of the specific area in the second predetermined area image.SELECTED DRAWING: Figure 42

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a screen creation method, a program, and an information processing system. [Background technology]

[0002] In recent years, wide-field-of-view images (hereinafter referred to as "wide-field-of-view images") with a wide viewing angle, such as a 360-degree image (also called a celestial sphere image, omnidirectional image, or panoramic image) capturing an entire 360-degree surrounding area, have become known as an imaging range that includes areas that cannot be fully confirmed with a normal angle of view. When attempting to display such a wide-field-of-view image in its entirety on a display terminal, the wide-field-of-view image is curved and difficult to see, so a predetermined area image showing a predetermined area in the wide-field-of-view image is displayed on the display terminal, and the user views the predetermined area image (see Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0003] However, it is difficult for a user to understand what the predetermined area image was taken of and where it was taken just by viewing the predetermined area image within the displayed range.

[0004] In addition, the camera captures an object from a 360-degree angle around the camera in one shot from the camera's position, but it cannot capture the object from all directions. Therefore, even if the object is within the shooting range, it cannot capture the back side of the object, so the user cannot grasp the state of the back side of the object.

[0005] The above-mentioned problems also exist when the captured image is not a curved image such as a wide-field image.

[0006] The present disclosure aims to improve user convenience by complementing captured images obtained by photography. [Means for solving the problem]

[0007] The disclosure of claim 1 is an information processing device having a creation unit that creates a screen including a first captured image display area that displays a first predetermined area image, which is a first predetermined area in a first captured image obtained by capturing an object at a first capturing position using a capturing device, and a three-dimensional image display area that displays at least a portion of a three-dimensional image aligned with the first captured image and displays a capturing position image that indicates a position corresponding to a second capturing position at a predetermined capturing date and time, and the creation unit includes on the screen a second captured image display area that displays a specific image that indicates the position of the specific area superimposed on the second predetermined area image, based on a second predetermined area image that is a second predetermined area in a second captured image at the predetermined capturing date and time that is associated with the second capturing position and that is stored in a memory unit in association with the second capturing position, and a specific area identified from the second captured image. [Effects of the Invention]

[0008] According to the present disclosure, an effect is achieved in that convenience for a user can be improved by creating an image that complements a captured image obtained by photography. [Brief explanation of the drawings]

[0009] [Figure 1] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. 1C is a plan view of the imaging device. [Figure 2] FIG. 10 is an image diagram of the imaging device in use. [Figure 3] (a) is a hemispherical image (before) taken with the imaging device, (b) is a hemispherical image (after) taken with the imaging device, and (c) is an image represented by the Mercator projection. [Figure 4] (a) A conceptual diagram showing how a sphere is covered with a Mercator image, and (b) a diagram showing a spherical image. [Figure 5] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. [Figure 6](a) is a three-dimensional oblique view of Figure 5, (b) is a diagram showing the predetermined area image in the state of (a) displayed on the display, (c) is a diagram showing the predetermined area after changing the viewpoint of the virtual camera IC in (a), and (d) is a diagram showing the predetermined area image in the state of (c) displayed on the display. [Figure 7] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 8] FIG. 10 is a conceptual diagram showing the relationship between a predetermined area and a point of interest. [Figure 9] 1 is a schematic diagram of a communication system according to an embodiment. [Figure 10] FIG. 2 is a diagram illustrating a hardware configuration of the imaging device. [Figure 11] FIG. 2 is a hardware configuration diagram of a relay device. [Figure 12] FIG. 2 is a hardware configuration diagram of a communication control device and a communication terminal. [Figure 13] FIG. 2 is a diagram illustrating the functional configuration of the communication system according to the embodiment. [Figure 14] FIG. 10 is a conceptual diagram of a user device management table. [Figure 15] FIG. 10 is a conceptual diagram of a virtual room management table. [Figure 16] FIG. 10 is a conceptual diagram of a three-dimensional image management table (part 1). [Figure 17] FIG. 10 is a conceptual diagram of a three-dimensional image management table (part 2). [Figure 18] FIG. 10 is a conceptual diagram of a movement history management table. [Figure 19] FIG. 10 is a sequence diagram showing a communication process of content data in the communication system. [Figure 20] FIG. 10 is a sequence diagram showing a process for starting video and audio recording in a communication system. [Figure 21] FIG. 10 is a sequence diagram showing a process for stopping video and audio recording in a communication system. [Figure 22] FIG. 10 is a sequence diagram showing the process of recording and playing back audio in a communication system. [Figure 23] FIG. 10 is a diagram showing a recording data selection screen. [Figure 24] 10 is a flowchart showing a process executed by the communication control device in a screen display process. [Figure 25] 10 is a flowchart showing a process executed by the communication control device in a screen display process. [Figure 26] FIG. 10 is a diagram showing an example of an initial display of a screen displayed on a communication terminal 9a. [Figure 27] FIG. 10 is a diagram showing an example of a screen displayed on a communication terminal 9a in which a specific area has been identified. [Figure 28] 13 is a diagram showing a screen on which a corresponding predetermined area, an icon of the virtual camera IC1, and a line diagram showing the line of sight of the virtual camera IC1 are displayed in the display area 620. FIG. [Figure 29] FIG. 10 is a diagram showing an example of a screen displayed on a communication terminal 9a in which a specific area has been identified. [Figure 30] FIG. 10 is a diagram showing a screen on which a display area for displaying text information is displayed. [Figure 31] FIG. 10 is a diagram showing an example of a screen displayed on a communication terminal 9a in which a specific area has been identified. [Figure 32] FIG. 10 is a diagram showing an example of a screen displayed on a communication terminal 9a in which a specific area has been identified. [Figure 33] FIG. 10 is a diagram showing another state in which the icon of the virtual camera IC1, the corresponding display area, and the line of sight diagram are superimposed on the communication terminal 9a. [Figure 34] FIG. 10 is a diagram showing an example of a screen displayed on a communication terminal 9a in which a plurality of specific regions are specified. [Figure 35] FIG. 10 is a diagram showing an example of a screen displayed on a communication terminal 9a in which a plurality of specific regions are specified. [Figure 36] FIG. 10 is a conceptual diagram of a movement history management table. [Figure 37] FIG. 10 is a diagram showing an example of an initial display of a screen displayed on a communication terminal 9a. [Figure 38] 13 is a diagram showing a screen on which a corresponding predetermined area, an icon of the virtual camera IC1, and a line diagram showing the line of sight of the virtual camera IC1 are displayed in the display area 620. FIG. [Figure 39]FIG. 10 is a diagram showing an example of a screen displayed on a communication terminal 9a in which a specific area has been identified. [Figure 40] 10 is a flowchart showing a process executed by the communication control device in a screen display process after registration of a photographing date and time. [Figure 41] 10 is a flowchart showing a process executed by the communication control device in a screen display process after registration of a photographing date and time. [Figure 42] FIG. 10 is a diagram showing a screen on the communication terminal 9a when the current predetermined area image is displayed after the photographing date and time is registered. [Figure 43] FIG. 10 is a diagram showing a screen on the communication terminal 9a when the current predetermined area image is displayed after the photographing date and time is registered. [Figure 44] 10 is a flowchart showing a process executed by the communication control device in a screen display process after registration of a photographing date and time. [Figure 45] 10 is a flowchart showing a process executed by the communication control device in a screen display process after registration of a photographing date and time. [Figure 46] FIG. 10 is a diagram showing a screen on the communication terminal 9a when the current predetermined area image is displayed after the photographing date and time is registered. [Figure 47] FIG. 10 is a diagram showing a screen on the communication terminal 9a when the current predetermined area image is displayed after the photographing date and time is registered. [Figure 48] FIG. 10 is a diagram showing a screen on the communication terminal 9a when the current predetermined area image is displayed after the photographing date and time is registered. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [Overview of spherical images] A method for generating a spherical image will be described with reference to Fig. 1 to Fig. 8. A spherical image is also called a spherical panoramic image or a 360° panoramic image, and is an example of a wide-field video with a wide viewing angle. Wide-field images also include simple panoramic images of about 180°.

[0012] First, the appearance of the imaging device 10 will be described using Fig. 1. The imaging device 10 is a digital camera for obtaining captured images that are the basis for creating a spherical image. Fig. 1(a) is a left side view of the imaging device, Fig. 1(b) is a front view of the imaging device, and Fig. 1(c) is a plan view of the imaging device.

[0013] As shown in Fig. 1(a), the photographing device 10 is small enough to be held in one hand. As shown in Figs. 1(a), 1(b), and 1(c), an image sensor 103a is provided on the front side (front side) of the upper portion of the photographing device 10, and an image sensor 103b is provided on the rear side (rear side). As shown in Fig. 1(b), an operation unit 115 such as a shutter button is provided on the front side of the photographing device 10.

[0014] Next, a usage situation of the imaging device 10 will be described with reference to FIG. 2. Note that FIG. 2 is an image diagram of the imaging device in use. As shown in FIG. 2, the imaging device 10 is communicably connected to a relay device 3 installed on some kind of stand 2, and is used to capture surrounding subjects, scenery, and the like. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the imaging element 103a and the imaging element 103b shown in FIG. 1. Note that if the omnidirectional image captured by the imaging device 10 is not to be transmitted to other communication terminals or systems, the relay device 3 is not necessary.

[0015] Next, an outline of processing until a celestial sphere image is created from an image captured by the image capturing device 10 will be described with reference to Figs. 3 and 4. Fig. 3(a) is a diagram showing a hemispherical image (front side) captured by the image capturing device, Fig. 3(b) is a diagram showing a hemispherical image (rear side) captured by the image capturing device, and Fig. 3(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). An image expressed by Mercator projection or the like (hereinafter referred to as "Mercator image") may also be used. Fig. 4(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 4(b) is a diagram showing a celestial sphere image. The "equirectangular projection image" is an equirectangular celestial sphere image as an example of the wide-field-of-view image described above.

[0016] As shown in Fig. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a wide-angle lens 102a such as a fisheye lens, which will be described later. Also, as shown in Fig. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a wide-angle lens 102b such as a fisheye lens, which will be described later. Then, the image capturing device 10 combines the hemispherical image (front side) with a hemispherical image (rear side) flipped 180 degrees to create an equirectangular projection image EC as shown in Fig. 3(c).

[0017] The image capturing device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to apply an equirectangular projection image EC to cover the spherical surface as shown in FIG. 4( a), thereby creating a celestial sphere image CE as shown in FIG. 4( b). In this way, the celestial sphere image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (two-dimensional) and 3D (three-dimensional) data. OpenGL ES is merely an example of software that performs image processing, and the celestial sphere image CE may be created by other software. The celestial sphere image CE may be a still image or a video. While the image capturing device 10 has been described as generating a celestial sphere image, the communication control device 5 or the communication terminals 7 and 9 may perform similar image processing or some of the image processing steps.

[0018] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted onto the surface of a sphere as shown in Fig. 4(a), creating a spherical image as shown in Fig. 4(b). In this way, the spherical image is represented as an image in which the Mercator image faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data.

[0019] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people when they view it. Therefore, the communication terminals 7 and 9 can display a predetermined region (hereinafter referred to as a "predetermined region image"), which is a partial region of the spherical image, as a flat image with little curvature (distortion), thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to FIGS. 5 to 8.

[0020] Fig. 5 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. The virtual camera IC1 corresponds to the position of a virtual viewpoint of a user viewing the celestial sphere image CE displayed as a three-dimensional sphere. In Fig. 6, (a) is a three-dimensional perspective view of Fig. 5, (b) is a diagram showing the predetermined area image in the state of (a) displayed on the display, (c) is a diagram showing the predetermined area after the viewpoint of the virtual camera IC1 in (a) is changed, and (d) is a diagram showing the predetermined area image in the state of (c) displayed on the display.

[0021] If the celestial sphere image CE generated in this manner is a three-dimensional sphere CS, the virtual camera IC1 is located inside the celestial sphere image CE as shown in Fig. 5. A predetermined area T in the celestial sphere image CE is an imaging area of ​​the virtual camera IC1, and is specified by angle-of-view information (also referred to as "area information") that indicates the imaging direction and angle of view of the virtual camera IC1 in a three-dimensional virtual space including the celestial sphere image CE.

[0022] Zooming of the predetermined region T can also be expressed by moving the virtual camera IC1 closer to or farther away from the celestial sphere image CE. The predetermined region image Q is an image of the predetermined region T in the celestial sphere image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC1 to the celestial sphere image CE.

[0023] Furthermore, when the virtual viewpoint of the virtual camera IC1 is moved (also referred to as "changed") to the right (left as one faces the drawing) from the state of Fig. 6(a) as shown in Fig. 6(c), the predetermined area T in the omnidirectional image CE is moved to the predetermined area T' accordingly, and the predetermined area image Q displayed on the predetermined display is changed to the predetermined area image Q'. As a result, the image shown in Fig. 6(b) is changed to the image shown in Fig. 6(d) and displayed on the predetermined display.

[0024] Next, the relationship between the angle of view information and the image of the predetermined area T will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a diagram showing points in a three-dimensional Euclidean space using spherical coordinates. Fig. 8 is a conceptual diagram showing the relationship between the predetermined area and a point of interest (center point).

[0025] Here, the coordinates of an arbitrary position when the center point CP shown in Fig. 7 is expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius vector, polar angle, and azimuth angle, respectively. The radius vector r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to an arbitrary point (the center point CP in Fig. 8), and is equal to the distance f shown in Fig. 8.

[0026] Furthermore, as shown in FIG. 8, when the center of a predetermined area T, which is the imaging area of ​​the virtual camera IC1, is considered to be the center point CP in FIG. 7, the trigonometric function shown in the following (Equation 1) generally holds. (L / f) = tan(α / 2) (Equation 1) Note that f is the distance from the virtual camera IC1 to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). α is the angle of view. In this case, the angle of view information for identifying the predetermined area T can be expressed by pan(θ), tilt(φ), and fov(α). Note that zooming of the predetermined area T can be expressed by widening or narrowing the range (arc) of the angle of view α.

[0027] [Communication system overview] Next, an overview of the communication system 1 according to the embodiment will be described with reference to Fig. 9. Fig. 9 is a schematic diagram of the communication system according to the embodiment.

[0028] As shown in Fig. 9, the communication system 1 of this embodiment is composed of a communication control device 5, an imaging device 10, a relay device 3, a communication terminal 7, and communication terminals 9a and 9b. The communication terminals 9a and 9b are collectively referred to as "communication terminal 9." The communication control device 5, the communication terminal 7, and the communication terminal 9 are also examples of information processing devices. The communication terminals 7 and 9 may also be referred to as "display terminals" that display images, etc.

[0029] Of these, the image capturing device 10 is a digital camera for obtaining a wide-field image (such as a spherical image), as described above. The relay device 3 functions as a cradle for charging the image capturing device 10 and transmitting and receiving data. The relay device 3 can perform data communication with the image capturing device 10 via a contact point, and can also perform data communication with the communication control device 5 via the communication network 100. The communication network 100 includes, for example, the Internet, a LAN (Local Area Network), a (wireless) router, etc.

[0030] The communication control device 5 is, for example, a computer, and can perform data communication with the relay device 3 and the communication terminals 7 and 9 via the communication network 100. The communication control device 5 can also be referred to as an "information management device" because it manages information such as angle of view.

[0031] The communication terminals 7 and 9 are, for example, computers such as notebook PCs (Personal Computers), and can perform data communication with the communication control device 5 via the communication network 100. OpenGL ES is installed in the communication terminals 7 and 9, and the communication terminals 7 and 9 create a predetermined area image (see FIG. 6 ) from the omnidirectional image received from the communication control device 5. The communication control device 5 may be configured by a single computer or by multiple computers.

[0032] Furthermore, the imaging device 10 and the relay device 3 are installed at predetermined positions by an organizer X or the like at a site Sa such as a construction site, exhibition hall, educational site, medical site, etc. The communication terminal 7 is operated by the organizer X. The communication terminal 9a is operated by a participant A such as a viewer who is in a remote location from the site Sa. Similarly, the communication terminal 9b is operated by a participant B such as a viewer who is in a remote location from the site Sa. Participant A and participant B may be in the same location or in different locations.

[0033] The communication control device 5 transmits (distributes) wide-field images obtained from the image capturing device 10 via the relay device 3 to the communication terminals 7 and 9. The communication control device 5 also transmits (distributes) captured images obtained from each communication terminal 7 to the communication terminals 7 and 9. Note that the captured images sent from the image capturing device 10 via the relay device 3 are wide-field images, but if a single-lens reflex camera is used instead of the image capturing device 10, the captured images will be ordinary narrow-field images. The captured images may be either moving images or still images.

[0034] [Hardware configuration] Next, the hardware configurations of the image capturing device 10, relay device 3, and communication terminals 7 and 9 of this embodiment will be described in detail with reference to FIGS.

[0035] <Hardware configuration of the imaging device> Fig. 10 is a hardware configuration diagram of the photographing device 10. As shown in Fig. 10, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0036] Of these, the imaging unit 101 is equipped with wide-angle lenses 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) that are each capable of capturing an image with a field of view of 180° or more to form a hemispherical image, and two imaging elements 103a and 103b that are provided corresponding to the lenses 102a and 102b, respectively.

[0037] Furthermore, the imaging elements 103a, 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts optical images captured by the lenses 102a, 102b, etc. into image data in the form of electrical signals and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example, and the imaging unit 101 may have only one, or three or more.

[0038] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus).

[0039] The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to a CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a network I / F 121, and the like.

[0040] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create data for an equirectangular projection image (an example of a wide-field image), which will be described later.

[0041] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0042] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function for displaying moving images on a display (for example, a display of an external terminal such as a smartphone that performs short-range communication with the imaging device 10 using the short-range communication circuit 117). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0043] As will be described later, the imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b. Although the imaging device 10 is not provided with a display unit in this embodiment, a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 receives the sound data output from the microphone 108 via an I / F bus and performs predetermined processing on the sound data.

[0044] The CPU 111 controls the overall operation of the image capturing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of processed equirectangular projection images.

[0045] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. A user operates the operation unit 115 to input various imaging modes, imaging conditions, etc.

[0046] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as needed.

[0047] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the image capturing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0048] The electronic compass 118 calculates the direction of the image capturing device 10 from the Earth's magnetism and outputs the direction information. This direction information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data such as the image capture date and time and the data size of the image data.

[0049] The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) that accompany the movement of the image capturing device 10. The changes in angle are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

[0050] The acceleration sensor 120 is a sensor that detects acceleration in three axial directions.

[0051] The photographing device 10 can also calculate the attitude (angle with respect to the direction of gravity) of the own device (the photographing device 10) by using the electronic compass 118, the acceleration sensor 120, etc. Furthermore, by providing the photographing device 10 with the acceleration sensor 120, the accuracy of image correction is improved.

[0052] The network I / F 121 is an interface for performing data communication using a communication network 100 such as the Internet via a router or the like. The hardware configuration of the image capturing device 10 is not limited to that shown here, and any hardware configuration may be used as long as it can realize the functional configuration of the image capturing device 10. At least a part of the hardware configuration may be present on the relay device 3 or the communication network 100.

[0053] <Hardware configuration of relay device> Fig. 11 is a diagram showing the hardware configuration of the relay device 3. Note that Fig. 11 shows the hardware configuration when the relay device 3 is a cradle having a wireless communication function.

[0054] As shown in FIG. 11, the relay device 3 includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a CMOS sensor 305, a bus line 310, a communication unit 313, an antenna 313a, a positioning unit 314, and an input / output I / F 316.

[0055] Of these, the CPU 301 controls the overall operation of the relay device 3. The ROM 302 stores programs such as an IPL (Initial Program Loader) used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301.

[0056] An EEPROM (Electrically Erasable and Programmable ROM) 304 reads or writes data under the control of the CPU 301. The EEPROM 304 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

[0057] The CMOS (Complementary Metal Oxide Semiconductor) sensor 305 is a solid-state image sensor that captures an image of a subject under the control of the CPU 301 and obtains image data.

[0058] The communication unit 313 uses an antenna 313a to communicate with the communication network 100 by wireless communication signals.

[0059] The positioning unit 314 receives a positioning signal including the position information (latitude, longitude, and altitude) of the relay device 3 from a GNSS (Global Navigation Satellite System) satellite such as a GPS (Global Positioning Systems) satellite, or an IMES (Indoor Messaging System) as an indoor GPS.

[0060] The input / output I / F 316 is an interface circuit (such as a USB I / F) electrically connected to the input / output I / F 116 of the image capturing apparatus 10. The input / output I / F 316 may be wireless or wired.

[0061] The bus line 310 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 301.

[0062] <Hardware configuration of communication control device and communication terminal> 12 shows the hardware configuration of the communication control device 5. The hardware configuration of the communication terminals 7 and 9 is the same as that of the communication control device 5, and therefore a description thereof will be omitted.

[0063] As shown in FIG. 12, the communication control device 5 is a computer and includes a CPU 501, a ROM 502, a RAM 503, an SSD 504, an external device connection I / F 505, a network I / F 506, a display 507, an operation unit 508, a media I / F 509, a bus line 510, a CMOS sensor 511, a speaker 512, and a positioning unit 514.

[0064] Of these, the CPU 501 controls the overall operation of the communication control device 5. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0065] The SSD 504 reads or writes various data under the control of the CPU 501. If the communication terminals 7 and 9 are smartphones or the like, the SSD 504 may not be provided. Alternatively, a hard disk drive (HDD) may be provided instead of the SSD 504.

[0066] The external device connection I / F 505 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0067] The network I / F 506 is an interface for performing data communication via the communication network 100 .

[0068] The display 507 is a type of display unit such as a liquid crystal display or organic electroluminescence (EL) display that displays various images.

[0069] An operation unit 508 is an input means for selecting and executing various instructions such as various operation buttons, a power switch, a shutter button, and a touch panel, selecting a processing target, moving a cursor, and the like.

[0070] The media I / F 509 controls reading and writing (storing) of data from and to a recording medium 509m such as a flash memory, etc. The recording medium 509m includes DVDs, Blu-ray Discs (registered trademarks), etc.

[0071] The CMOS sensor 511 is a type of imaging means that captures an image of a subject and obtains image data under the control of the CPU 501. A CCD sensor may be used instead of a CMOS sensor.

[0072] The speaker 512 is a circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice.

[0073] The positioning unit 314 receives a positioning signal including position information (latitude, longitude, and altitude) of the communication terminals 7 and 9 from a GNSS satellite such as a GPS satellite, or from IMES as an indoor GPS.

[0074] The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501.

[0075] [Functional configuration of the embodiment] Next, the functional configuration of the first embodiment will be described with reference to FIGS.

[0076] <Functional configuration of the imaging device> 13, the photographing device 10 has a reception unit 12, a detection unit 13, an imaging unit 16, a sound collection unit 17, a connection unit 18, and a memory / readout unit 19. Each of these units is a function or means realized by any of the components shown in FIG. 10 operating in response to an instruction from the CPU 111 in accordance with a photographing / storage program loaded from the SRAM 113 onto the DRAM 114.

[0077] The image capturing device 10 also includes a storage unit 1000 configured by a ROM 112, an SRAM 113, and a DRAM 114 shown in FIG.

[0078] (Functional configuration of the imaging device) The reception unit 12 of the image capturing device 10 is realized by the processing of the operation unit 115 for the CPU 111, and receives operation input from the user.

[0079] The detection unit 13 is mainly realized by processing from the CPU 111 to the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, etc., and obtains attitude information by detecting the attitude of the image capturing device 10.

[0080] The imaging section 16 is mainly realized by processing from the CPU 111 to the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111, and captures images of scenery and the like to obtain captured images.

[0081] The sound collection unit 17 is mainly realized by processing of the sound processing unit 109 from the CPU 111, and collects sounds around the image capturing device .

[0082] The connection unit 18 is mainly realized by processing from the CPU 111 to the input / output I / F 116, and performs data communication with the relay device 3.

[0083] The storage / readout unit 19 is mainly realized by the processing of the CPU 111 , and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000 .

[0084] <Functional configuration of relay device> 13, the relay device 3 has a communication unit 31 and a connection unit 38. These units are functions or means realized when any of the components shown in FIG. 11 operates in response to an instruction from the CPU 301 in accordance with the program for the relay device 3 loaded from the EEPROM 304 onto the RAM 303.

[0085] (Functional configuration of relay device 3) The communication unit 31 of the relay device 3 is mainly realized by processing from the CPU 301 shown in FIG. 11 to the communication unit 313, and performs data communication between the image capturing device 10 and the communication control device 5 via the communication network 100.

[0086] The connection unit 38 is mainly realized by processing from the CPU 301 to the input / output I / F 316, and performs data communication with the image capturing device 10.

[0087] <Functional configuration of communication control device> Next, each functional configuration of the communication control device 5 will be described in detail with reference to Fig. 13. The communication control device 5 has a communication unit 51, a reception unit 52, a creation unit 53, a processing unit 54, an authentication unit 55, a text generation unit 56, a specific processing unit 57, and a storage / readout unit 59. Each of these units is a function or means realized when any of the components shown in Fig. 12 operates in response to an instruction from the CPU 501 in accordance with the program for the communication control device 5 loaded from the SSD 504 onto the RAM 503.

[0088] The communication control device 5 also has a storage unit 5000 constructed by a RAM 503 and an HD 504 shown in Fig. 12. The storage unit 5000 contains a user device management DB 5001, a virtual room management DB 5002, a three-dimensional image management DB 5003, and a movement history management DB 5004.

[0089] (User / Device Management DB) Figure 14 is a conceptual diagram of a user device management table. The user device management DB 5001 is composed of the user device management table shown in Figure 14. In the user device table, user IDs (or device IDs), passwords, names, user images, and IP addresses are stored and managed in association with each other.

[0090] Among these, the user ID is an example of user identification information for identifying a user (organizer X, participants A, B, etc.). The device ID is an example of device identification information for identifying a device such as the image capturing device 10. Note that if a head-mounted display or the like is used in addition to the image capturing device 10, the head-mounted display or the like is also treated as a device.

[0091] The name is the name of a user or a device. Each user name may be the name of the communication terminal of each user.

[0092] The user image is a schematic image of each user's face, a photograph of the user's face, or the like, and is registered in advance by each user.

[0093] The IP address is an example of destination identification information for devices such as the communication terminals 7 and 9 and the image capturing device 10 used by the user.

[0094] (Virtual room management DB) Fig. 15 is a conceptual diagram of a virtual room management table. The virtual room management DB 5002 is configured by the virtual room management table shown in Fig. 15. In the virtual room management table, a virtual room ID, a virtual room name, a device ID, a host ID, a participant ID, a content ID, a content URL (information on the storage location of content data of images and sounds (audio)), and a three-dimensional image ID are stored and managed in association with each other.

[0095] Of these, the virtual room ID is an example of virtual room identification information for identifying a virtual room.

[0096] The virtual room name is the name of the virtual room and is given by the user or the like.

[0097] The device ID is the same as the device ID in FIG. 14, and is the ID of the device that has participated in the virtual room indicated by the virtual room ID of the same record.

[0098] The host ID is an example of host identification information for identifying the host ID in particular among the user IDs in FIG. 14, and is the ID of the host who participated in the virtual room indicated by the virtual room ID of the same record.

[0099] The participant ID is an example of participant identification information for identifying the participant ID among the user IDs in FIG. 14, and is the ID of a participant who has participated in the virtual room indicated by the virtual room ID of the same record.

[0100] The content ID is an example of content identification information for identifying image and sound content data. In this case, the image is a wide-field image obtained during shooting, and the sound is sound (including voice) obtained during the same shooting.

[0101] The content URL is an example of content storage location information that indicates where the content (wide-field image, sound information) data is stored. The content URL also stores the time of shooting (video recording) and sound collection (recording) as well as the shooting location (absolute location on Earth) in association with the content data.

[0102] The three-dimensional image ID is an example of three-dimensional image identification information that identifies a three-dimensional image to be displayed in a display area 620, which will be described later.

[0103] (3D image management DB) (Part 1) Fig. 16 is a conceptual diagram of a three-dimensional image management table (part 1). The three-dimensional image management DB 5003 is configured by the three-dimensional image management table shown in Fig. 16. In the three-dimensional image management table shown in Fig. 16, a model ID and position information are associated and managed for each three-dimensional image ID.

[0104] The model ID is an example of three-dimensional model identification information for identifying a three-dimensional model, and the three-dimensional model is generated based on a point cloud acquired by, for example, a TOF (Time-of-Flight) camera or the like.

[0105] The position information is information that indicates the position of the three-dimensional model in the three-dimensional virtual space using three-dimensional coordinates of XYZ, and is indicated, for example, by the three-dimensional coordinates of eight points that define the rectangular parallelepiped space occupied by the three-dimensional model.

[0106] (Part 2) Fig. 17 is a conceptual diagram of a three-dimensional image management table (part 2). The three-dimensional image management DB 5003 is configured by the three-dimensional image management table shown in Fig. 17. The three-dimensional image management table shown in Fig. 17 is a table for managing attribute information indicating the attributes of parts (three-dimensional models) that make up a structure included in a three-dimensional virtual space. In the three-dimensional image management table shown in Fig. 17, the attribute information, such as part number, member information, dimension information, color information, material information, position information, and construction date information, is managed in association with each three-dimensional image ID.

[0107] The component information is information that identifies components such as walls, floors, ceilings, windows, pipes, and doors that are made up of parts.

[0108] The dimension information is information that identifies the dimensions of a part in virtual space, and is expressed, for example, by numerical values ​​in the three axial directions of X, Y, and Z.

[0109] The color information is information for identifying the color of the part, and the material information is information for identifying the material of the part.

[0110] The position information is information for identifying the position of a part in virtual space, and is indicated by coordinates in three axes, X, Y, and Z, for example. This makes it possible to identify whether or not each of a plurality of parts is adjacent to another.

[0111] The construction date information indicates the planned date on which a component will be constructed in the real world, making it possible to identify a structure at a given point in time, excluding components that have not yet been constructed.

[0112] As described above, the position information in Figures 16 and 17 is managed in association with absolute positions on the Earth. As an example, by associating the origin (X=0, Y=0, Z=0) of the position information in Figures 16 and 17 with absolute positions on the Earth (latitude, longitude, altitude), all coordinates in the 3D images including the 3D model and parts can be associated with absolute positions on the Earth.

[0113] Furthermore, the three-dimensional image management DB 5003 may manage three-dimensional point groups, mesh objects, textured mesh objects, etc., instead of three-dimensional models.

[0114] (Movement history management DB) FIG. 18 is a conceptual diagram of a movement history management table. The movement history management DB 5004 is configured with the movement history management table shown in FIG. 18. In the movement history management table, the "shooting and sound collection" date and time, shooting location, angle of view information, text information, and specific area information are stored and managed in association with each content ID. The position of the camera device 10 is measured by the positioning unit 314 of the relay device 3 to which the camera device 10 is attached. Note that the camera device 10 may be provided with a positioning unit similar to the positioning unit 314, and the position of the camera device 10 may be measured by this positioning unit.

[0115] The content ID shown in FIG. 18 is the same as the content ID shown in FIG.

[0116] The photographing and sound collection dates and times indicate the dates and times when the photographing device 10 photographed and collected the sound.

[0117] The image capturing position indicates the position (absolute position on the earth) when the image was captured by the image capturing device 10. When sound is collected, the image capturing position is also the sound collection position.

[0118] The field of view information is information for identifying a predetermined area indicating a predetermined area image displayed on the communication terminal 9 (see FIG. 8). When managing the field of view information for a plurality of communication terminals (communication terminals 7, 9, etc.), the field of view information is managed for each communication terminal.

[0119] The text registered in the "Text Information" column is data obtained by converting the voice recorded on the recording date and time of the same record into text.

[0120] The specific area information is information indicating the position of a specific area identified from a captured image. The information indicating the position of the specific area may be information indicated by two-dimensional coordinates or information indicated by a three-dimensional image. Details of the specific area identified from a captured image will be described later.

[0121] Furthermore, the movement history management table has a registration field for registering the display position of the icon of the virtual camera IC1 for each shooting date and time. By registering a flag in the registration field, the creation unit 53 can display an icon 622a of the virtual camera IC1, etc., at the shooting position in a display area 620, which will be described later. Also, by registering a flag in the registration field, the creation unit 53 can associate the shooting position in the display area 620, which will be described later, with a specific area 611, which will be described later and which is identified from the captured image.

[0122] (Functional configuration of communication control device) Next, each functional configuration of the communication control device 5 will be described in detail with reference to FIG.

[0123] 12 to the network I / F 505, and performs data communication with other devices (relay device 3, communication terminals 7 and 9) via the communication network 100. The communication unit 51 also serves as an acquisition unit, acquiring instruction information indicating instructions sent from the communication terminals 7 and 9.

[0124] The reception unit 52 is realized by the processing of the operation unit 508 for the CPU 501, and receives operation input from a user (here, a system administrator or the like).

[0125] Creation unit 53 is mainly realized by processing of CPU 501, and uses data stored in storage unit 5000 to create a screen to be sent to each communication terminal 7, 9. For example, creation unit 53 creates screen 600 including display area 610 displaying a predetermined area image, which is a predetermined area in a wide-field-of-view image (an example of a captured image) captured by imaging device 10, and display area 620 displaying at least a part of a three-dimensional image in which a first position and a second position in the wide-field-of-view image are associated by processing unit 54 (see FIG. 26).

[0126] Processing unit 54 is mainly realized by processing of CPU 501, and performs processing to associate position information indicating a position in a captured image obtained by capturing an image of an object with imaging device 10 (an example of a first position) with position information indicating a position in a three-dimensional image including a three-dimensional area corresponding to the object (an example of a second position). Note that processing unit 54 can also be referred to as a "corresponding unit."

[0127] The authentication unit 55 is mainly realized by the processing of the CPU 501, and performs authentication such as whether each user is a legitimate person to use the virtual room.

[0128] The text generation unit 56 is mainly realized by the processing of the CPU 501, and generates text data from sound (audio) data of the content data.

[0129] The identification processing unit 57 is mainly realized by the processing of the CPU 501, and identifies a specific area from the captured image. The specific area is an area identified by a user's instruction through operation input in the display area 610. The specific area is also an area identified by image recognition processing using text information, which is text generated from voice collected together with the captured image or text input by the user.

[0130] The storage / readout unit 59 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000 .

[0131] <Functional configuration of communication terminal 7> Next, the functional configuration of the communication terminal 7 will be described in detail with reference to Fig. 13. The communication terminal 7 has a communication unit 71, a reception unit 72, a creation unit 73, a display control unit 74, a sound input / output control unit 75, a connection unit 78, and a storage / readout unit 79. Each of these units is a function or means realized when any of the components shown in Fig. 12 operates in response to an instruction from the CPU 501 in accordance with the program for the communication terminal 7 loaded from the SSD 504 onto the RAM 503.

[0132] The communication unit 71 of the communication terminal 7 is mainly realized by processing from the CPU 501 shown in FIG. 12 to the network I / F 505, and performs data communication with other devices (communication control device 5) via the communication network 100.

[0133] The reception unit 72 is mainly realized by the processing of the operation unit 508 on the CPU 501, and receives instructions by operation input from a user (here, organizer X). The reception unit 72 also serves as an acquisition unit, and can be said to acquire instructions given by user operation.

[0134] The creation unit 73 is mainly realized by the processing of the CPU 501, and creates a screen to be displayed on the terminal itself (communication terminal 7) using data stored in the storage unit 7000. When the creation unit 53 of the communication control device 5 creates the screen, the communication terminal 7 may or may not have the creation unit 73.

[0135] The display control unit 74 is mainly realized by the processing of the CPU 501, and performs control to display various images on the display 507 of the communication terminal 7 or an external display connected to the external device connection I / F 505.

[0136] The sound input / output control unit 75 is mainly realized by the processing of the CPU 501 of the communication terminal 7, and performs control to collect sound from an external microphone connected to the external device connection I / F 505. If the communication terminal 7 has a built-in microphone, the sound input / output control unit 75 performs control to collect sound from the microphone. The sound input / output control unit 75 also performs control to output sound to the speaker 512 of the communication terminal 7 or an external speaker connected to the external device connection I / F 505.

[0137] The creation unit 73 is mainly realized by the processing of the CPU 501, and adds narration, subtitles, etc. to the content data recorded and sounded by the communication terminal 7 to create content data for use as teaching materials, etc.

[0138] The storage / readout unit 79 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 7000 and reads out various data (or information) from the storage unit 7000 .

[0139] <Functional configuration of communication terminal 9> Next, each functional configuration of the communication terminal 9 will be described in detail with reference to FIG.

[0140] The communication terminal 9 has a communication unit 91, a reception unit 92, a creation unit 93, a display control unit 94, a sound input / output control unit 95, a connection unit 98, and a storage / readout unit 99. Each of these units is a function or means realized when any of the components shown in Fig. 12 operates in response to an instruction from the CPU 501 in accordance with the program for the communication terminal 9 loaded from the SSD 504 onto the RAM 503.

[0141] The communication terminal 9 also includes a storage unit 9000 constructed by the RAM 503 and the SSD 504 shown in FIG.

[0142] The communication unit 91 of the communication terminal 9 is mainly realized by processing from the CPU 501 to the network I / F 505, and performs data communication with other devices (communication control device 5) via the communication network 100.

[0143] The reception unit 92 is mainly realized by the processing of the operation unit 508 on the CPU 501, and receives instructions by operation input from users (participants in this case). The reception unit 92 also serves as an acquisition unit, and can be said to acquire instructions given by user operations.

[0144] The creation unit 93 is mainly realized by the processing of the CPU 501, and creates a screen to be displayed on the terminal itself (communication terminal 9) using data stored in the storage unit 9000. When the creation unit 53 of the communication control device 5 creates the screen, the communication terminal 9 may or may not have the creation unit 73.

[0145] The display control unit 94 is mainly realized by the processing of the CPU 501, and performs control to display various images on the display 507 of the communication terminal 9 or an external display connected to the external device connection I / F 505.

[0146] The sound input / output control unit 95 is mainly realized by the processing of the CPU 501 of the communication terminal 9, and performs control to collect sound from an external microphone connected to the external device connection I / F 505. If the communication terminal 7 has a built-in microphone, the sound input / output control unit 95 performs control to collect sound from the microphone. The sound input / output control unit 95 also performs control to output sound to the speaker 512 of the communication terminal 9 or an external speaker connected to the external device connection I / F 505.

[0147] The connection unit 98 is mainly realized by processing from the CPU 501 to the external device connection I / F 505, and performs data communication with external devices connected by wire or wirelessly.

[0148] The storage / readout unit 99 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000 .

[0149] [Processing or Operation of the Embodiment] Next, the processing or operation of the embodiment will be described with reference to Figures 19 to 35. Note that the following processing is performed after the image capturing device 10 and the communication terminals 7 and 9 have already joined the same virtual room.

[0150] <Communication processing of content data in a communication system> First, the communication processing of content data in the communication system 1 will be described with reference to FIG. 19. FIG. 19 is a sequence diagram showing the communication processing of a wide-field image and each angle of view information in the communication system. In this embodiment, a case will be described in which the image capturing device 10, the communication terminal 7 of the organizer X, the communication terminal 9a of the participant A, and the communication terminal 9b of the participant B are in the same virtual room. Furthermore, when the virtual room is constructed, the storage / readout unit 79 adds one record to the virtual room management DB (see FIG. 15) and manages the virtual room ID, virtual room name, device ID, organizer ID, and participant ID in association with each other. The content ID, content URL, and three-dimensional image ID will be stored later. Note that the processes S11 to S15 in FIG. 19 are repeated, for example, 30 or 60 times per second.

[0151] S11: In the camera device 10, the imaging unit 16 captures spherical images of the area Sa and collects sound to obtain content (wide-field image, sound information) data, and then the connection unit 18 sends the content data to the relay device 3. In this case, the connection unit 18 also sends a virtual room ID for identifying the virtual room in which the camera device 10 is participating and a device ID for identifying the camera device 10. Furthermore, the camera device 10 transmits information about the camera position, which is the position (absolute position) of the camera device 10, to the relay device 3 at predetermined time intervals (for example, every second). As a result, the connection unit 38 of the relay device 3 acquires the content data, the virtual room ID, the device ID, and the camera position information.

[0152] S12: In the relay device 3, the communication unit 31 transmits the content data, virtual room ID, device ID, and information on the shooting location acquired by the connection unit 38 in step S11 to the communication control device 5 via the communication network 100. As a result, in the communication control device 5, the communication unit 51 receives the content data, virtual room ID, device ID, and information on the shooting location. As a result, the shooting information (shooting and sound collection date and time, shooting location, angle of view information, and text information) is managed by the processing unit 54 in the movement history management DB 5004 for each shooting date and time (for example, every second). In this case, the text generation unit 56 uses the sound data of the content data to convert the audio portion into text and generate text. In addition, the processing unit 54 stores the content data in the content URL (see FIG. 15) in the storage unit 5000.

[0153] The photographing device 10 may transmit the content data, the virtual room ID, the device ID, and the photographing position information to the communication terminal 7 instead of the relay device 3 (S11d). In this case, the communication terminal 7 transmits the content data, the virtual room ID, the device ID, and the photographing position information to the communication control device 5 (S12d).

[0154] S13: In the communication control device 5, the storage / readout unit 59 searches the virtual room management DB 5002 based on the virtual room ID received in step S12 to read out the user IDs (host ID and participant IDs) participating in the same virtual room as the image capturing device 10. The storage / readout unit 59 also searches the user / device management DB 5001 based on the readout host ID and participant ID to read out the corresponding user image of the host X and the IP address of the communication terminal 7, as well as the user images of the corresponding participants A and B and the IP addresses of the communication terminals 9a and 9b. The communication unit 51 then references the IP address of the communication terminal 7 and transmits screen data including the content data (image data, sound data) and three-dimensional image data received in step S12 to the communication terminal 7. As a result, the communication unit 71 of the communication terminal 7 receives the content data and three-dimensional image data. The detailed process (screen display process) will be described later.

[0155] S14: The communication unit 51 of the communication control device 5 references the IP address of the communication terminal 9a and transmits the content data received in step S12 to the communication terminal 9a. As a result, the communication unit 91 of the communication terminal 9a receives the content data. Note that the detailed processing at this time (screen display processing) will be described later.

[0156] S15: Similarly, the communication unit 51 of the communication control device 5 references the IP address of the communication terminal 9b and transmits the content data received in step S12 to the communication terminal 9b. As a result, the communication unit 91 of the communication terminal 9b receives the content data. Note that the detailed processing at this time (screen display processing) will be described later.

[0157] <Process for starting video and audio recording in a communication system> Next, the process of starting video and audio recording in the communication system 1 will be described with reference to Fig. 20. Fig. 20 is a sequence diagram showing the process of starting video and audio recording in the communication system.

[0158] S31: First, in the communication terminal 7 of the organizer X, the reception unit 72 receives an operation from the organizer X to start video and audio recording.

[0159] S32: Before starting video and audio recording in the communication terminal 7, the communication unit 71 transmits a command to share the field of view information to the communication control device 5. This command includes the virtual room ID of the virtual room in which the communication terminal 7 is participating and the device ID of the image capturing device 10. As a result, the communication unit 51 of the communication control device 5 receives the command to share the field of view information.

[0160] S33: In the communication control device 5, the storage / readout unit 59 sets the content URL and the field of view information URL in the virtual room management DB (see FIG. 15). Then, the communication unit 51 transmits to the communication terminal 7 an instruction to start recording and a request to upload the field of view information. This instruction includes information indicating a content URL that indicates a location where the communication terminal 7 will save the content data after recording. This request also includes information indicating the field of view information URL for saving the field of view information. As a result, in the communication terminal 7, the communication unit 71 receives the instruction to start recording and the request to upload the field of view information.

[0161] S34: The communication unit 51 also transmits a request to upload the angle of view information to the communication terminal 9a. This request includes information about a URL for saving the angle of view information. As a result, the communication unit 91 of the communication terminal 9a receives the request to upload the angle of view information.

[0162] S35: Similarly, the communication unit 51 transmits a request to upload the angle of view information to the communication terminal 9b. This request includes information about a URL for saving the angle of view information. As a result, the communication unit 91 of the communication terminal 9b receives the request to upload the angle of view information.

[0163] S36: Next, in the communications terminal 7, the storage / readout unit 79 functions as a video recording unit and a sound recording unit, and starts recording and sound recording of the content data received in process S13 shown in Fig. 19. In the case of process S12d shown in Fig. 19, the communications terminal 7 may start recording and sound recording of the content data received from the imaging device 10 in process S11d, instead of the content data received from the communications control device 5 in process S13.

[0164] S37: In the communication terminal 7, for example, while displaying a predetermined area image (see FIG. 6(b)), which is a predetermined area (see FIG. 6(a)) of the wide-field image received in process S13, if the reception unit 72 receives a request to change the angle of view from the organizer X, the display control unit 74 displays a predetermined area image (see FIG. 6(d)), which is a changed predetermined area (see FIG. 6(c)) of the same wide-field image. In this case, the reception unit 72 also functions as an acquisition unit, and when receiving a request to display a predetermined area in the wide-field image from a user (here, organizer X), acquires angle-of-view information (pan, tilt, fov) for specifying the predetermined area in the wide-field image to be displayed on the display 507. Then, the communication unit 71 transmits the angle-of-view information for specifying the changed predetermined area to the image information URL (communication control device 5) received in process S33. This angle-of-view information includes the user ID of the organizer X of the communication terminal 7, which is the sender. As a result, in the communication control device 5, the communication unit 51 receives the angle-of-view information. The received information on the angle of view is used as information on the angle of view that the storage / readout unit 79 stores in the movement history management DB (see FIG. 18).

[0165] S38: The same process as S37 is performed independently of S37 in the communication terminal 9a and the communication control device 5. The user ID transmitted in this case is the user ID of participant A.

[0166] S39: A process similar to the process S37 is performed independently of the processes S37 and S38 in the communication terminal 9b and the communication control device 5. The user ID transmitted in this case is the user ID of participant B.

[0167] The processes S37 to S39 may be executed collectively by the communication control device 5 when the recording is completed.

[0168] <Processing for stopping video and audio recording in a communication system> Next, a process for stopping video and audio recording in the communication system 1 will be described with reference to Fig. 21. Fig. 21 is a sequence diagram showing a process for stopping video and audio recording in the communication system.

[0169] S51: First, in the communication terminal 7 of the organizer X, the reception unit 72 receives an operation from the organizer X to stop video and audio recording.

[0170] S52: The storage / readout unit 79 stops recording the content data.

[0171] S53: The communication unit 71 uploads (transmits) the video and audio content data to the specified content URL (communications control device 5) received in step S33. This content data includes the times (timestamps) from the start to the end of the video and audio recording. As a result, the communication unit 51 in the communications control device 5 receives the content data. The timestamps are the same as the shooting and audio collection dates and times shown in FIG. 18.

[0172] S54: In the communication control device 5, the storage / readout unit 59 stores the content data together with the timestamp in a predetermined content URL. Furthermore, the storage / readout unit 59 converts the timestamps (dates and times of shooting and sound collection) managed in the movement history management DB (see FIG. 18) into elapsed playback time in accordance with the total recording time of the content data whose recording has been stopped.

[0173] S55: The communication unit 51 transmits a video and audio end notification to the communication terminal 7. The end notification includes information indicating a specific content URL. As a result, the communication unit 71 of the communication terminal 7 receives the video and audio end notification.

[0174] S56: Similarly, the communication unit 51 transmits a video and audio end notification to the communication terminal 9a. The end notification includes information indicating a specific content URL. As a result, the communication unit 91 of the communication terminal 9a receives the video and audio end notification.

[0175] S57: Similarly, the communication unit 51 transmits a video and audio end notification to the communication terminal 9b. The end notification includes information indicating a specific content URL. As a result, the communication unit 91 of the communication terminal 9b receives the video and audio end notification.

[0176] In the case of process S55, the completion notification does not need to include the predetermined content URL.

[0177] <Recording and playback of recorded data in a communication system> Next, the process of recording and playing back audio data in a communication system will be described with reference to Fig. 22 to Fig. 35. Fig. 22 is a sequence diagram showing the process of recording and playing back audio data in a communication system. Fig. 23 is a diagram showing a recording data selection screen. Here, participant A uses communication terminal 9a to play back the recorded content data.

[0178] S71: First, when the reception unit 92 of the communication terminal 9a receives a login operation from user A by inputting a user ID, password, etc., the communication unit 91 transmits a login request to the communication control device 5. This request includes the user ID and password of user A. As a result, in the communication control device 5, the communication unit 51 receives the login request, and the authentication unit 55 performs authentication by referring to the user device management DB (see FIG. 14). Hereinafter, the explanation will proceed assuming that user A is determined to be a legitimate accessing person through login authentication.

[0179] S72: In the communication control device 5, the creation unit 53 creates a recording data selection screen 940 as shown in FIG. 23. In this case, the storage / reading unit 59 searches the virtual room management DB (see FIG. 15) using the user ID received in process S71 as a search key, thereby reading out all corresponding virtual room IDs, virtual room names, and content URLs. The creation unit 53 then creates thumbnails 941, 942, and 943 using images from each piece of content data (with timestamp) stored in the content URL. As a result, the creation unit 53 adds, for each thumbnail, a virtual room name (such as "Construction Site α") and a recording time (such as "2022 / 10 / 31 15:00") indicating a predetermined time in the timestamp (for example, the recording start time).

[0180] S73: The communication unit 51 transmits the selection screen data created in step S72 to the communication terminal 9a. In this selection screen data, each thumbnail includes a content ID for identifying the wide-field image from which each thumbnail was created. As a result, the communication unit 91 of the communication terminal 9a receives the selection screen data.

[0181] S74: In the communication terminal 9a, the display control unit 94 displays a recorded data selection screen 940 as shown in Fig. 23 on the display 507 of the communication terminal 9a. Then, the reception unit 92 receives a designation (selection) of a predetermined thumbnail from the participant A. Here, the description will continue assuming that the thumbnail 941 is designated (selected).

[0182] S75: The communication unit 71 transmits to the communication control device 5 a download request for the content data that is the creation source of the selected thumbnail 941. This request includes the content ID associated with the thumbnail 941. As a result, the communication unit 51 of the communication control device 5 receives the download request for the content data.

[0183] S76: In the communication control device 5, the storage / readout unit 59 searches the virtual room management DB (see FIG. 15) using the content ID received in process S75 as a search key, and reads out content data from the corresponding content URL. The content data also includes location information (shooting location) when the image capturing device 10, described below, captured and collected the image. The storage / readout unit 59 also searches the three-dimensional image management DB 5003 using the three-dimensional image ID associated with the content ID also received in process S75 as a search key, and reads out parameters of the corresponding model ID and location information (see FIG. 16), or parameters of the corresponding part number, location information, etc. (see FIG. 17).

[0184] Then, the communication unit 51 transmits the requested content data and the three-dimensional image data to the communication terminal 9a, whereby the communication unit 91 of the communication terminal 9a receives the content data and the three-dimensional image data.

[0185] S77: The communication terminal 9a performs a playback process. That is, the display control unit 94 of the communication terminal 9a displays a screen including the recorded image on the display 507 of the communication terminal 9a, and the sound input / output control unit 95 outputs sound.

[0186] <Details of screen display process (registration)> Next, process S14 of processes S13 to S15 will be described with reference to Figures 24 to 35. Here, participant A operates the communication terminal 9a, and the accepting unit 92 accepts the operation, and the communication unit 91 transmits operation information indicating the operation content to the communication control device 5. As a result, in the communication control device 5, the communication unit 51, which serves as an acquiring unit, acquires the operation information, and the creation unit 53 creates a screen 600 based on the operation content indicated by the operation information. Then, the communication unit 51 transmits data of the screen 600 to the communication terminal 9a, and the communication unit 91 of the communication terminal 9a receives the data of the screen 600, and the display control unit 94 displays the screen 600 on the display 507 or the like of the communication terminal 9a. In this case, the communication control device 5 is an example of an information processing device.

[0187] Alternatively, the communication unit 91 of the communication terminal 9a may receive each piece of data that is material for creating the screen 600 from the communication control device 5, and the reception unit 92, which functions as an acquisition unit, may acquire operation information indicating the content of the operation performed by the participant A. In this case, the creation unit 93 of the communication terminal 9a creates the screen 600. In this case, the communication terminal 9a is an example of an information processing device. The communication terminals 7 and 9b can also perform the same processes and functions as the communication terminal 9a.

[0188] Next, a process in which the communication control device 5 creates a screen 600 to be displayed on the display 507 of the communication terminal 9a will be described. In steps S13 and S15, the communication control device 5 performs the same process as step S14, except that the terminal on which the screen is displayed is different, and therefore a description of steps S13 and S15 will be omitted. Figures 24 and 25 are flowcharts showing the process executed by the communication control device in the screen display process.

[0189] First, the creation unit 53 of the communication control device 5 creates a screen 600 to be displayed by the communication terminal 9a, as shown in FIG. 26. FIG. 26 is a diagram showing an example of an initial display of a screen displayed on the communication terminal 9a. The screen 600 includes a display area 610 (an example of a captured image display area) and a display area 620 (an example of a three-dimensional image area). In FIG. 26, the display area 610 and the display area 620 are displayed simultaneously in the same size within the screen 600, but they may be displayed simultaneously in different sizes, or one of them may be displayed by switching selected by participant A. Alternatively, the display area 610 may be displayed on one of two displays of the screen 600, and the display area 620 may be displayed on the other.

[0190] The display area 610 displays a predetermined area image, which is a predetermined area in a wide-field image (an example of a captured image) obtained by capturing an object such as a desk, pillar, or window using the imaging device 10 (see Figures 6(a) and (b)).

[0191] Here, if the captured image is not a curved image such as a wide-field image, the display area 610 displays a predetermined area image whose predetermined area is the same area as the captured area of ​​the captured image.

[0192] The display area 620 includes a three-dimensional model that represents the shape of the object in three dimensions, and displays part or all of a three-dimensional image in which coordinates (an example of first coordinates) of the wide-field image are associated with coordinates (an example of second coordinates).

[0193] As explained in Figure 11, the shooting position, which is the position of the shooting device 10, is associated with an absolute position on Earth, and as explained in Figures 16 and 17, all coordinates in the three-dimensional images, including each three-dimensional model and part, managed in the three-dimensional image management DB 5003 are also associated with absolute positions on Earth.

[0194] As a result, as a positioning process, the processing unit 54 associates, for example, coordinates indicating the photographing position as a first position in the wide-field image of the content data received by the communication control device 5 in process S12 (or S12d) with coordinates indicating a second position in the three-dimensional image having the same absolute position as the coordinates. Here, the coordinates are an example of position information.

[0195] As another form of alignment processing, the processing unit 54 may associate coordinates in the wide-field image with coordinates indicating a second position in the three-dimensional image by image processing such as matching feature points in the wide-field image with feature points in the three-dimensional image, without using absolute positions on the Earth.

[0196] Then, as described above, the creation unit 53 creates a screen 600 including a display area 610 that displays a predetermined area image, which is a predetermined area in the wide-field image, and a display area 620 that displays at least a portion of a three-dimensional image in which the first position (first coordinate) and the second position (second coordinate) in the wide-field image are associated by the processing unit 54.

[0197] Here, when the table of the three-dimensional image management DB 5003 shown in Figures 16 and 17 manages three-dimensional point clouds, mesh objects, textured mesh objects, etc. instead of three-dimensional models, the display area 620 displays at least a portion of a three-dimensional image including three-dimensional point clouds, mesh objects, textured mesh objects, etc. as a three-dimensional area corresponding to the object included in the wide-field image.

[0198] Also displayed on the screen 600 is a “Register” button 680. The “Register” button 680 is a button that is pressed when displaying, in the display area 620, an icon of the virtual camera IC1 whose shooting area is the angle of view (information) that specifies the predetermined area of ​​the predetermined area image being displayed in the display area 610, at a position corresponding to the shooting position of the image capturing device 10.

[0199] The screen 600 also displays a close button 609 that is pressed to close the screen 600.

[0200] Further, the display contents of the screen 600 will be described in detail below. The virtual camera IC1 is used to specify a predetermined area related to the predetermined area image to be displayed in the display area 610 (see FIG. 8), while the virtual camera IC2 described below is used to specify a three-dimensional image to be displayed in the display area 620.

[0201] S111: First, the creation unit 53 of the communication control device 5 identifies a predetermined area image to be displayed in the display area 610, as shown in FIG. 26, from the wide-field image of the content data received by the communication unit 51, based on the virtual angle of view of the virtual camera IC1 that has been set in advance.

[0202] S112: The processing unit 54 adjusts the position of the virtual camera IC2 to the shooting position associated with the wide-field image, and adjusts the virtual angle of view (an example of the second angle of view) of the virtual camera IC2 to the preset virtual angle of view (an example of the first angle of view) of the virtual camera IC1. As a result, the creation unit 53 creates a three-dimensional image to be displayed in the display area 620.

[0203] The identification processing unit 57 identifies a specific area 611 from the captured image within the display area 610, as shown in Fig. 27. Fig. 27 is a diagram showing an example of a screen displayed on the communication terminal 9a in which a specific area has been identified. The specific area 611 is an area identified in the display area 610 by a user through an instruction input by operation. For example, the user identifies the specific area 611 through an instruction input by operation of an input device such as a mouse. The user can identify an area of ​​concern (for example, an area with a defect) as the specific area 611.

[0204] Furthermore, the specific area 611 may be an area identified by image recognition processing using text information, which is text generated from voice collected together with the captured image or text input by the user. For example, the user may speak about a location of concern, and the specific area 611 may be identified by image recognition processing using text information, which is text generated from the voice (e.g., "The top of the pillar is missing").

[0205] 27, the specific area 611 is represented by a dashed line, which is one example of a display mode of the specific area 611. This display mode includes dashed and solid lines of a predetermined color, and the thickness of the dashed and solid lines may be different from that of other parts. Furthermore, the specific area 611 may be an area to be masked.

[0206] S113: The communication unit 51 determines whether a request to register the shooting date and time has been received from the communication unit 91 as a result of the "Register" button 680 being pressed on the communication terminal 9a.

[0207] S114: If a registration request is acquired (S113; YES), the processing unit 54 registers the requested photography date and time in the movement history management DB 5004 (see FIG. 18). In this case, the processing unit 54 registers a flag corresponding to the registration request in the registration field shown in FIG. 18. In this case, the date and time when the registration request is acquired is registered as the photography date and time.

[0208] As described in FIG. 18, in the movement history management DB 5004, the shooting and sound collection date and time, shooting position, angle of view information, text information, and specific area information are stored and managed in association with each content ID.

[0209] As described with reference to FIG. 15, the virtual room management DB 5002 stores and manages content IDs and content URLs (information on storage locations of content data of wide-field images and sound) in association with each other.

[0210] As a result, the photographing position, the angle of view information, the wide-field image, the sound (audio), and the specific area are associated with the registered photographing date and time.

[0211] Here, the field of view information is information for identifying a specific area indicating the specific area image displayed on the communication terminal 9, so the shooting position, the specific area image, the sound (audio), and the specific area are associated with the registered shooting date and time.

[0212] S115: As shown in Fig. 28, the creation unit 53 superimposes an icon 622a of the virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in step S114 at a position corresponding to the shooting position in the display area 620. Fig. 28 is a diagram showing a screen on which the corresponding predetermined area, the icon of the virtual camera IC1, and a line diagram showing the line of sight of the virtual camera IC1 are displayed in the display area 620.

[0213] S116: Furthermore, the creation unit 53 superimposes a corresponding predetermined area 621a corresponding to the angle of view of the virtual camera IC1 within the display area 620, as shown in FIG. 28. FIG. 28 is a diagram showing a state in which an icon of the virtual camera IC1, a corresponding display area, and a line diagram indicating the line of sight are superimposed on the communication terminal 9a. This corresponding predetermined area 621a is an area corresponding to a predetermined area related to the predetermined area image displayed in the display area 610. Note that in FIG. 28, the corresponding predetermined area 621a is represented by a dashed line, but this is an example of a display mode of the corresponding predetermined area 621a. This display mode includes dashed lines and solid lines of a predetermined color, and the thickness of the dashed lines and solid lines may be different from that of other parts.

[0214] Furthermore, the creation unit 53 superimposes an icon 622a of the virtual camera IC1 at a position within the display area 620 that corresponds to the shooting position of the image capturing device 10, and superimposes the icon 622a so that it faces the center point CP1a of the corresponding predetermined area 621a. The icon 622a is an example of a schematic diagram (image) of the virtual camera IC1. In addition to the icon, this schematic diagram also includes characters such as "camera" or a figure including these characters.

[0215] Furthermore, the creation unit 53 superimposes a line diagram 623a indicating the line of sight from the icon 622a to the center point CP1a of the corresponding predetermined area 621a within the display area 620. This line diagram 623a may be a solid line or a dashed line, and may be displayed with a thickness or color different from that of the other lines.

[0216] S117: Furthermore, the identification processing unit 57 determines whether or not a identified area 611 is identified within the display area 610, as shown in FIG.

[0217] S118: If it is determined that a specific area 611 has been identified (S117; YES), the processing unit 54 registers the identified specific area 611 in the movement history management DB 5004 (see FIG. 18) in association with the shooting position of the captured image.

[0218] S119: Furthermore, the creation unit 53 superimposes a specific image 624 indicating the position of the specific area 611 within the display area 620, as shown in Fig. 28. This specific image 624 is an image indicating the position of the specific area 611 specified in the display area 610. Therefore, the user can refer to the three-dimensional image on which the specific image 624 indicating the position of the specific area 611 is superimposed, making it easier for the user to grasp the position and improving convenience.

[0219] 28, the specific image 624 is represented by dashed lines, which is one example of a display mode of the specific image 624. This display mode includes dashed lines and solid lines of a predetermined color, and the thickness of the dashed lines and solid lines may be different from that of other parts.

[0220] S120: After process S119 or if a registration request has not been acquired in process S113 (NO), the creation unit 53 determines whether the position or angle of view of the virtual camera IC1 in the display area 610 has been changed (see FIG. 6(c)). For example, when user A performs an operation to change the predetermined area image in the display area 610 shown in FIG. 28, the acceptance unit 92 accepts the changed angle of view, and the communication unit 91 transmits angle of view information indicating the changed angle of view to the communication control device 5. When the communication unit 51 receives the angle of view information indicating the changed angle of view, the creation unit 53 determines that the angle of view of the virtual camera IC1 has been changed. Then, when the position or angle of view of the virtual camera IC1 has been changed (S120; YES), the process returns to process S111. In this way, by repeating the processes S111 to S119, each time the “Register” button 680 is pressed at different times, the communication unit 51 acquires multiple registration requests, and the processing unit 54 registers flags corresponding to the multiple registration requests in the movement history management DB 5004. In addition, the creation unit 53 may superimpose multiple icons 622a, etc. of virtual cameras IC1 in the display area 620.

[0221] In addition, when the shooting positions of the multiple virtual cameras IC1 are the same, the creation unit 53 may superimpose an icon 622a of one virtual camera IC1 in the display area 620, and register flags corresponding to the multiple registration requests in the movement history management DB 5004 in association with the icon 622a of this one virtual camera IC1.

[0222] S121: If the reception unit 92 receives a request to end the display of the screen 600 by, for example, pressing the close button 609 by user A (YES), the display of the screen 600 ends. On the other hand, if the reception unit 92 does not receive a request to end the display of the screen 600 (NO), the process returns to step S117, and the processes in FIGS. 24 and 25 continue until the transmission of the captured image data by the communication control device 5 is completed in step S14 shown in FIG.

[0223] The timing for identifying the specific area 611 from the captured image is not limited to the timing when the screen 600 shown in FIG. 27 is displayed, but may be the timing when the screen 600 shown in FIG. 29 is displayed, for example.

[0224] The identification processing unit 57 identifies a specific area 611 from the captured image within the display area 610, as shown in Fig. 29. Fig. 29 is a diagram showing an example of a screen displayed on the communication terminal 9a in which a specific area has been identified. The specific area 611 is an area identified by a user in the display area 610 through an instruction entered by operation input. The specific area 611 may also be an area identified by image recognition processing using text information, which is text generated from voice collected together with the captured image or text input by the user.

[0225] 29, when the "Register" button 680 is pressed, the processing unit 54 performs steps S114 to S119, and stores and manages the shooting and sound collection dates and times, shooting positions, field of view information, text information, and specific areas in association with each content ID in the movement history management DB 5004 (see FIG. 18). As a result, the shooting positions, field of view information, wide-field image, sound (audio), and specific areas are associated with the registered shooting dates and times.

[0226] As shown in FIG. 28, the creation unit 53 superimposes an icon 622a of the virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in process S114 at a position corresponding to the shooting position in the display area 620.

[0227] 28, the creation unit 53 superimposes a corresponding predetermined area 621a corresponding to the angle of view of the virtual camera IC1 within the display area 620. The creation unit 53 also superimposes an icon 622a of the virtual camera IC1 within the display area 620 at a position corresponding to the shooting position of the image capturing device 10, and superimposes the icon 622a so that it faces the center point CP1a of the corresponding predetermined area 621a. The creation unit 53 also superimposes a line 623a indicating the line of sight from the icon 622a to the center point CP1a of the corresponding predetermined area 621a within the display area 620.

[0228] 28, the creation unit 53 superimposes a specific image 624 indicating the position of the specific area 611 within the display area 620. In this way, the timing at which the specific area 611 is identified from the captured image may be the timing at which the screen 600 shown in FIG. 27 is displayed before the icon 622a is displayed, or the timing at which the screen 600 shown in FIG. 29 is displayed after the icon 622a is displayed.

[0229] <Registration Variations> The above screen 600 is an example and is not limited to the above example. For example, as shown in Fig. 30, the screen 600 may display a display area 650 that displays text information. Fig. 30 is a diagram showing a screen in which a display area that displays text information is displayed.

[0230] A display area 650 is displayed below the display area 610 of the screen 600 shown in FIG. 30. The display area 650 displays text information based on audio within a predetermined time period, including the elapsed playback time of the image currently displayed in the display area 610. For example, in FIG. 18, when the shooting and audio collection date and time (elapsed playback time) is "2023 / 11 / 11 10:00:01," the creation unit 53 creates the screen 600 to be displayed in the display area 650a using the text "The top of the pillar is missing." registered during the period when the shooting position and angle of view information did not change (2023 / 11 / 11 10:00:01 to 2023 / 11 / 11 10:00:03). Note that the display area 650 may also display text information input by the user.

[0231] 30, when the "Register" button 680 is pressed, the processing unit 54 performs steps S114 to S119, and stores and manages the shooting and sound collection date and time, shooting position, field of view information, and text information "The top of the pillar is missing" in association with each content ID in the movement history management DB 5004 (see FIG. 18). As a result, the shooting position, field of view information, wide-field image, text information in the display area 650, and specific area 611 in the display area 610 are associated with the registered shooting date and time.

[0232] Furthermore, as shown on the screen 600 in Figures 31 and 32, the processing unit 54 performs processing to associate specific areas 611 of each of the multiple captured images captured at different times based on the position information (capturing position) when the photographing device 10 captured and collected sound with an icon 622a of the virtual camera IC1 based on the photographing position and angle of view associated with the photographing date and time registered in processing S114.

[0233] For example, as shown in FIG. 31, the identification processing unit 57 identifies a specific area 611a from the captured image in the display area 610. FIG. 31 is a diagram showing an example of a screen displayed on the communication terminal 9a in which a specific area has been identified. The specific area 611a is an area identified in the display area 610 by a user through an instruction input by operation. For example, the user identifies the specific area 611a through an instruction input by operation of an input device such as a mouse. Furthermore, the specific area 611a may be an area identified from the captured image in the display area 610 through image recognition processing using text information "The center space is empty" displayed in the display area 650.

[0234] 31, when the "Register" button 680 is pressed, the processing unit 54 performs steps S114 to S119, and stores and manages, for each content ID, the shooting and sound collection date and time, shooting position, angle of view information, text information "The center space is empty," and the specific area 611a in association with each other in the movement history management DB 5004 (see FIG. 18). As a result, the processing unit 54 associates the specific area 611a identified from the captured image in the display area 610 with the icon 622a of the virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in step S114.

[0235] For example, as shown in Fig. 32, the identification processing unit 57 identifies a specific area 611a from the captured image in the display area 610. Fig. 32 is a diagram showing an example of a screen displayed on the communication terminal 9a in which the specific area has been identified. The specific area 611a is an area identified in the display area 610 by a user's instruction via operation input. For example, the user identifies the specific area 611a by an instruction via operation input using an input device such as a mouse. Furthermore, the specific area 611a may be an area identified from the captured image in the display area 610 by image recognition processing using text information displayed in the display area 650, such as "There is a desk in the central space."

[0236] 32, when the "Register" button 680 is pressed, the processing unit 54 performs steps S114 to S119, and stores and manages, for each content ID, the shooting and sound collection date and time, shooting position, angle of view information, text information "There is a desk in the central space," and the specific area 611a in association with each other in the movement history management DB 5004 (see FIG. 18). As a result, the processing unit 54 associates the specific area 611a identified from the captured image in the display area 610 with the icon 622a of the virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in step S114.

[0237] The specific area 611a in Fig. 31 and the specific area 611a in Fig. 32 are examples of the specific area 611 of each of a plurality of captured images taken at different times. By pressing the "Register" button 680 in the state of the screen 600 in Fig. 31 or the screen 600 in Fig. 32, the user can associate the specific area 611 of each of the plurality of captured images taken at different times with the icon 622a of the virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in process S114.

[0238] 33 is a diagram showing another state in which the icon of virtual camera IC1, the corresponding display area, and a line of sight diagram are superimposed on communication terminal 9a. As shown in Fig. 33, when the shooting position of image capturing device 10 changes from the state in Fig. 30, the content displayed in display areas 610 and 620 also changes from the display areas 610 and 620 in Fig. 30. Display area 620 in Fig. 33 shows corresponding predetermined area 621b, icon 622b of virtual camera IC1, and line of sight diagram 623b.

[0239] 30 and 33, the positions of the icon 622a of the virtual camera IC1 and the specific area 611 in Fig. 30 are different from the positions of the icon 622b of the virtual camera IC1 and the specific area 611b in Fig. 33. The processing unit 54 performs processing to associate an image indicating the shooting position (for example, the icon 622a or 622b of the virtual camera IC1) with the specific area 611 or 611b, respectively, for each of a plurality of captured images captured at different shooting positions.

[0240] 33, for example, the identification processing unit 57 identifies an identified area 611b from the captured image in the display area 610. The identified area 611b is an area identified by a user's instruction through an operation input in the display area 610. Alternatively, the identified area 611b may be an area identified from the captured image in the display area 610 by image recognition processing using text information displayed in the display area 650, such as "The lower part of the pillar is missing."

[0241] 33, when the "Register" button 680 is pressed, the processing unit 54 performs processes S114 to S119, and stores and manages, for each content ID, the shooting and sound collection date and time, shooting position, angle of view information, text information "The bottom of the pillar is missing," and the identified area 611b in association with each other in the movement history management DB 5004 (see FIG. 18). As a result, the processing unit 54 associates the identified area 611b identified from the captured image in the display area 610 with the icon 622b of the virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in process S114.

[0242] By pressing the "Register" button 680 in the state of the screen 600 of Figure 30 or the screen 600 of Figure 32, the user can associate the specific area 611 on the screen 600 of Figure 30 with the icon 622a of the virtual camera IC1, and can also associate the specific area 611b on the screen 600 of Figure 33 with the icon 622b of the virtual camera IC1.

[0243] Furthermore, as shown on screen 600 in FIG. 34, the creation unit 53 may display on screen 600 a plurality of specific areas identified from the captured image, and allow the user to identify a specific area to be associated with the icon 622a of the virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in process S114.

[0244] 34, "Register" buttons 680a and 680b are displayed. When the "Register" button 680a is pressed in the state of FIG. 34, the processing unit 54 associates a specific area 611a identified from the captured image in the display area 610a with an icon 622a of the virtual camera IC1 based on the shooting position and angle of view. Furthermore, when the "Register" button 680b is pressed in the state of FIG. 34, the processing unit 54 associates a specific area 611b identified from the captured image in the display area 610b with an icon 622a of the virtual camera IC1 based on the shooting position and angle of view.

[0245] Furthermore, as shown on screen 600 in FIG. 35, the creation unit 53 may display on screen 600 a plurality of specific areas identified from the captured image, and allow the user to identify a specific area to be associated with icon 622a or 622b of virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in process S114.

[0246] 35, "Register" buttons 680a and 680b are displayed. When the "Register" button 680a is pressed in the state of FIG. 35, the processing unit 54 associates a specific area 611a identified from the captured image in the display area 610a with an icon 622a of the virtual camera IC1 based on the shooting position and angle of view. Furthermore, when the "Register" button 680b is pressed in the state of FIG. 35, the processing unit 54 associates a specific area 611b identified from the captured image in the display area 610b with an icon 622a or 622b of the virtual camera IC1 based on the shooting position and angle of view.

[0247] [Major Effects of the Embodiments] As described above, according to this embodiment, the processing unit 54 associates a first position in the captured image obtained by photographing the object with a second position in the three-dimensional image including a three-dimensional area corresponding to the object, and the creation unit 53 (or creation unit 73, 93) creates a screen 600 including a display area 610 that displays a predetermined area image, which is a predetermined area in the captured image obtained by photographing the object, and a display area 620 that displays at least a part of the three-dimensional image in which the first position and the second position in the captured image are associated by the processing unit 54.

[0248] This allows the user to understand what the predetermined area image is of and where it was taken by viewing the screen 600. Furthermore, even if the back side of an object cannot be photographed, the user can understand the state of the back side of the object by viewing the screen 600. Therefore, by creating an image that complements the photographed image, it is possible to achieve the effect of improving user convenience.

[0249] Furthermore, by associating the photographing date and time, the photographing location, the currently displayed angle of view, the text, and the specific area using the movement history management DB 5004 shown in FIG. 18 , when participant A presses the “Register” button 680 while the predetermined area image is being displayed in the display area 610, the processing unit 54 can register the currently displayed photographing date and time, the photographing location, the angle of view, the text, and the specific area in the movement history management DB 5004. Also, as shown in FIG. 28 , the creation unit 53 can superimpose an icon 622a or the like of a virtual camera IC1, whose photographing area is the currently displayed angle of view of the predetermined area image, at a position corresponding to the photographing location in the display area 620. Furthermore, as shown in FIG. 28 , the creation unit 53 can display a three-dimensional image on which a specific image 624 indicating the position of the specific area 611 is superimposed in the display area 620. This allows the user to refer to the three-dimensional image on which the specific image 624 indicating the position of the specific area 611 is superimposed, making it easier to grasp the location and improving convenience.

[0250] (Movement history management DB) FIG. 36 is a conceptual diagram of a movement history management table. The movement history management DB 5004 is configured with the movement history management table shown in FIG. 36. In the movement history management table, the "filming and sound collection" date and time, filming location, angle of view information, text information for the current date and time, text information for another date and time, and specific area information are stored and managed in association with each other for each content ID. The position of the camera device 10 is determined by the positioning unit 314 of the relay device 3 to which the camera device 10 is attached. Note that the camera device 10 may be provided with a positioning unit similar to the positioning unit 314, and the position of the camera device 10 may be determined by this positioning unit.

[0251] The content ID shown in FIG. 36 is the same as the content ID shown in FIG.

[0252] The photographing and sound collection dates and times indicate the dates and times when the photographing device 10 photographed and collected the sound.

[0253] The image capturing position indicates the position (absolute position on the earth) when the image was captured by the image capturing device 10. When sound is collected, the image capturing position is also the sound collection position.

[0254] The field of view information is information for identifying a predetermined area indicating a predetermined area image displayed on the communication terminal 9 (see FIG. 8). When managing the field of view information for a plurality of communication terminals (communication terminals 7, 9, etc.), the field of view information is managed for each communication terminal.

[0255] The text registered in the "Text information for current date and time" column is data converted into text from audio recorded on the same record at the recording date and time, or data associated by the processing unit 54 by pressing the link button 682 described below.

[0256] In the "Text information of another date and time" column, the current or past text being displayed in the display area 650 or 652 is registered by the processing unit 54. Note that the display areas 650 and 652 are examples of text display areas.

[0257] The specific area information is information indicating the position of a specific area identified from an image captured at the current date and time or another date and time. The image captured at the current date and time is an example of a first captured image. The image captured at another date and time is an example of a second captured image. The information indicating the position of the specific area may be information indicated by two-dimensional coordinates or information indicated by a three-dimensional image. Details of the specific area identified from an image captured at the current date and time or another date and time will be described later.

[0258] Furthermore, the movement history management table is provided with a registration field for registering the display position of the icon of the virtual camera IC1 for each shooting date and time. By registering a flag in the registration field, the creation unit 53 can display an icon 622a of the virtual camera IC1 at the shooting position in a display area 620 described below. Furthermore, by registering a flag in the registration field, the creation unit 53 can associate the shooting position in a display area 620 described below with a specific area 611a (described below) identified from an image shot at the current date and time or a specific area 631a (described below) identified from an image shot at a different date and time.

[0259] First, the creation unit 53 of the communication control device 5 creates a screen 601 to be displayed by the communication terminal 9a, as shown in FIG. 37. FIG. 37 is a diagram showing an example of an initial display of a screen displayed on the communication terminal 9a. The screen 601 includes a display area 610 (an example of a first captured image display area), a display area 620 (an example of a three-dimensional image area), and a display area 630 (an example of a second captured image display area). In FIG. 37, the display areas 610, 620, and 630 are simultaneously displayed in the same size within the screen 601, but they may be simultaneously displayed in different sizes, or one of them may be displayed by switching selected by participant A. Alternatively, the display areas 610, 620, and 630 may be displayed on each of the three displays of the screen 601.

[0260] The display area 610 displays a predetermined area image, which is a predetermined area in a current wide-field-of-view image (an example of a captured image) obtained by capturing an image of an object such as a desk, pillar, or window using the image capturing device 10 (see FIGS. 6(a) and 6(b)). Here, if the current captured image is not a curved image like a wide-field-of-view image, the display area 610 displays a predetermined area image whose predetermined area is the same as the captured area of ​​the current captured image.

[0261] The display area 620 includes a three-dimensional model that represents the shape of an object (such as a table or pillar) in three dimensions, and displays part or all of a three-dimensional image in which coordinates (an example of first coordinates) in the wide-field image are associated with coordinates (an example of second coordinates).

[0262] Display area 630 displays a predetermined area image, which is a predetermined area in a past wide-field-of-view image (an example of a past captured image) obtained by capturing an image of an object using image capturing device 10 (see FIGS. 6(a) and 6(b)). Here, if the past captured image is not a curved image like a wide-field-of-view image, display area 630 displays a predetermined area image whose predetermined area is the same as the captured area of ​​the past captured image.

[0263] As explained in Figure 11, the shooting position, which is the position of the shooting device 10, is associated with an absolute position on Earth, and as explained in Figures 16 and 17, all coordinates in the three-dimensional images, including each three-dimensional model and part, managed in the three-dimensional image management DB 5003 are also associated with absolute positions on Earth.

[0264] As a result, as a positioning process, the processing unit 54 associates, for example, coordinates indicating a photographing position as a first position in the current wide-field-of-view image of the content data received by the communication control device 5 in process S12 (or S12d), coordinates indicating a second position in the three-dimensional image having the same absolute position as the coordinates, and coordinates indicating a photographing position as a third position in the past wide-field-of-view image. Here, coordinates are an example of position information.

[0265] As another form of alignment processing, the processing unit 54 may correspond coordinates indicating a first position in the current wide-field image, coordinates indicating a second position in the three-dimensional image, and coordinates indicating a third position in the past wide-field image, for example, by image processing such as matching feature points of the current wide-field image with feature points of the three-dimensional image, without using absolute positions on the Earth.

[0266] Then, as described above, the creation unit 53 creates a screen 601 including a display area 610 that displays a predetermined area image that is a predetermined area in the current wide-field image, a second display area that displays at least a portion of a three-dimensional image in which a first position (first coordinate) and a second position (second coordinate) in the wide-field image are associated, and a display area 630 that displays a predetermined area image that is a predetermined area in a past wide-field image in which the first position (first coordinate) and a third position (third coordinate) are associated.

[0267] Here, when the table of the three-dimensional image management DB 5003 shown in Figures 16 and 17 manages three-dimensional point clouds, mesh objects, textured mesh objects, etc. instead of three-dimensional models, the display area 620 displays at least a portion of a three-dimensional image including three-dimensional point clouds, mesh objects, textured mesh objects, etc. as a three-dimensional area corresponding to the object included in the wide-field image.

[0268] Also displayed on the screen 601 are a "Register" button 680 and a "Link" button 682. The "Register" button 680 is a button that is pressed when displaying, in the display area 620, an icon of a virtual camera IC1 whose shooting area is the angle of view (information) that specifies the predetermined area of ​​the predetermined area image currently being displayed in the display area 610, at a position corresponding to the shooting position of the image capturing device 10. The "Link" button 682 is a button that is pressed when performing a process of associating a specific image that indicates the position of the specific area, which will be described later, between the current captured image and a previously captured image.

[0269] The screen 601 also displays a close button 609 that is pressed to close the screen 601.

[0270] Further, the display contents of the screen 600 will be described in detail below. The virtual camera IC1 is used to specify a predetermined area related to the predetermined area image to be displayed in the display area 610 (see FIG. 8), while the virtual camera IC2 described below is used to specify a three-dimensional image to be displayed in the display area 620.

[0271] S111: First, the creation unit 53 of the communication control device 5 identifies the current specified area image to be displayed in the display area 610, as shown in FIG. 37, from the current wide-field image of the content data received by the communication unit 51, based on the virtual angle of view of the virtual camera IC1 that has been set in advance.

[0272] S112: The processing unit 54 adjusts the position of the virtual camera IC2 to the shooting position associated with the current wide-field image, and adjusts the virtual angle of view (an example of the second angle of view) of the virtual camera IC2 to the preset virtual angle of view (an example of the first angle of view) of the virtual camera IC1. As a result, the creation unit 53 creates a three-dimensional image to be displayed in the display area 620.

[0273] S113: The communication unit 51 determines whether a request to register the shooting date and time has been received from the communication unit 91 as a result of the "Register" button 680 being pressed on the communication terminal 9a.

[0274] S114: If a registration request is acquired (S113; YES), the processing unit 54 registers the requested photography date and time in the movement history management DB 5004 (see FIG. 36). In this case, the processing unit 54 registers a flag corresponding to the registration request in the registration field shown in FIG. 36. In this case, the date and time when the registration request is acquired is registered as the photography date and time.

[0275] As described in FIG. 36, in the movement history management DB 5004, for each content ID, the shooting and sound collection date and time, shooting position, angle of view information, text information for the current date and time, text information for a different date and time, and specific area information are stored and managed in association with each other.

[0276] As described with reference to FIG. 15, the virtual room management DB 5002 stores and manages content IDs and content URLs (information on storage locations of content data of wide-field images and sound) in association with each other.

[0277] As a result, the photographing position, the angle of view information, the wide-field image, the sound (audio), and the specific area are associated with the registered photographing date and time.

[0278] Here, the field of view information is information for identifying a specific area indicating the specific area image displayed on the communication terminal 9, so the shooting position, the specific area image, the sound (audio), and the specific area are associated with the registered shooting date and time.

[0279] S115: As shown in Fig. 38, the creation unit 53 superimposes an icon 622a of the virtual camera IC1 based on the shooting position and angle of view associated with the shooting date and time registered in step S114 at a position corresponding to the shooting position in the display area 620. Fig. 38 is a diagram showing a screen on which the corresponding predetermined area, the icon of the virtual camera IC1, and a line diagram showing the line of sight of the virtual camera IC1 are displayed in the display area 620.

[0280] S116: Furthermore, the creation unit 53 superimposes a corresponding predetermined area 621a corresponding to the angle of view of the virtual camera IC1 in the display area 620, as shown in FIG. 38. FIG. 38 is a diagram showing a state in which an icon of the virtual camera IC11, a corresponding display area, and a line diagram indicating the line of sight are superimposed on the communication terminal 9a. This corresponding predetermined area 621a is an area corresponding to a predetermined area related to the predetermined area image displayed in the display area 610. Note that in FIG. 38, the corresponding predetermined area 621a is represented by a dashed line, but this is an example of a display mode of the corresponding predetermined area 621a. This display mode includes dashed lines and solid lines of a predetermined color, and the thickness of the dashed lines and solid lines may be different from that of other parts.

[0281] Furthermore, the creation unit 53 superimposes an icon 622a of the virtual camera IC1 at a position within the display area 620 that corresponds to the shooting position of the image capturing device 10, and superimposes the icon 622a so that it faces the center point CP1a of the corresponding predetermined area 621a. The icon 622a is an example of a schematic diagram (image) of the virtual camera IC1. In addition to the icon, this schematic diagram also includes characters such as "camera" or a figure including these characters.

[0282] Furthermore, the creation unit 53 superimposes, within the display area 620a, a line diagram 623a indicating a line of sight from the icon 622a to the center point CP1a of the corresponding predetermined area 621a. The line diagram 623a may be a solid line or a dashed line, and may be displayed with a thickness or color different from that of the other lines.

[0283] S117: Furthermore, the identification processing unit 57 determines whether or not a specific area 611 is identified within the display area 610.

[0284] The identification processing unit 57 identifies a specific area 611 from the current captured image in the display area 610, as shown in Fig. 39. Fig. 39 is a diagram showing an example of a screen displayed on the communication terminal 9a in which a specific area has been identified. The specific area 611 is an area identified in the display area 610 by a user through an instruction input by operation. For example, the user identifies the specific area 611 through an instruction input by operation of an input device such as a mouse. The user can identify an area of ​​concern (for example, an area with a defect) as the specific area 611.

[0285] Furthermore, the specific area 611 may be an area identified by image recognition processing using text information, which is text generated from voice collected together with the current captured image or text input by the user. For example, the user may speak about a location of concern, and the specific area 611 may be identified by image recognition processing using text information, which is text generated from the voice (e.g., "The top of the pillar is missing").

[0286] 39, the specific area 611 is represented by a dashed line, which is one example of a display mode of the specific area 611. This display mode includes dashed and solid lines of a predetermined color, and the thickness of the dashed and solid lines may be different from that of other parts. Furthermore, the specific area 611 may be an area to be masked.

[0287] S118: If it is determined that a specific area 611 has been identified (S117; YES), the processing unit 54 registers the identified specific area 611 in the movement history management DB 5004 (see FIG. 36) in association with the shooting position of the current captured image.

[0288] S119: After process S118, if it is determined in process S117 that the specific area 611 has not been specified (NO), or if a registration request has not been acquired in process S113 (NO), the creation unit 53 determines whether the position or the angle of view of the virtual camera IC1 in the display area 610 has been changed (see FIG. 6(c)). For example, when user A performs an operation to change the current predetermined area image in the display area 610 shown in FIG. 38, the reception unit 92 receives the changed angle of view, and the communication unit 91 transmits angle of view information indicating the changed angle of view to the communication control device 5. As a result, when the communication unit 51 receives the angle of view information indicating the changed angle of view, the creation unit 53 determines that the angle of view of the virtual camera IC1 has been changed. Then, if the position or the angle of view of the virtual camera IC1 has been changed (S119; YES), the process returns to process S111. In this way, by repeating the processes S111 to S118, each time the “Register” button 680 is pressed at different times, the communication unit 51 acquires multiple registration requests, and the processing unit 54 registers flags corresponding to the multiple registration requests in the movement history management DB 5004. In addition, the creation unit 53 may superimpose multiple icons 622a, etc. of virtual cameras IC1 in the display area 620.

[0289] In addition, when the shooting positions of the multiple virtual cameras IC1 are the same, the creation unit 53 may superimpose an icon 622a of one virtual camera IC1 in the display area 620, and register flags corresponding to the multiple registration requests in the movement history management DB 5004 in association with the icon 622a of this one virtual camera IC1.

[0290] S120: If the reception unit 92 receives a request to end the display of the screen 601 by, for example, pressing the close button 609 by user A (YES), the display of the screen 601 ends. On the other hand, if the reception unit 92 does not receive a request to end the display of the screen 601 (NO), the process returns to step S117, and the processes in FIGS. 24 and 25 continue until the transmission of the captured image data by the communication control device 5 is completed in step S14 shown in FIG.

[0291] The timing for identifying the specific area 611 from the captured image is not limited to the timing when the screen 601 shown in Figure 39 is displayed, but may also be the timing when the screen 601 shown in Figure 37 is displayed, for example.

[0292] The screen 601 may display a display area 650 that displays text information. The display area 650 is displayed below the display area 610 of the screen 601. The display area 650 displays text information, which is text based on audio within a predetermined time period including the elapsed playback time of the image currently displayed in the display area 610. For example, in FIG. 36 , when the shooting and audio collection date and time (elapsed playback time) is "2023 / 11 / 11 10:00:01," the creation unit 53 creates the screen 600 to be displayed in the display area 650 using the text "A pillar is missing." registered during the period when the shooting position and angle of view information did not change (2023 / 11 / 11 10:00:04 to 2023 / 11 / 11 10:00:06). Note that the display area 650 may display text information, which is text input by the user.

[0293] A display area 652 for displaying text information may be displayed on the screen 601. The display area 652 is displayed below the display area 630 of the screen 601. The display area 652 displays text information that is text based on audio within a predetermined time period that includes the elapsed playback time of the image being displayed in the display area 630.

[0294] 39, when the "Register" button 680 is pressed, the processing unit 54 performs processes S114 to S118, and stores and manages, for each content ID, the shooting and sound collection date and time, the shooting position, the field of view information, and the text information of the current date and time in association with each other, in the movement history management DB 5004 (see FIG. 36). As a result, the shooting position, the field of view information, the text information of the current date and time, and the specific area 611 of the display area 610 are associated with the registered shooting date and time.

[0295] <Details of screen display process (after registration)> Next, with reference to FIGS. 40 to 43, a screen display process after registration for process S14 among processes S13 to S15 will be described. In FIGS. 42 and 43, components with the same reference numerals as those in FIGS. 37 to 39 have the same roles or functions. Here, when participant A operates the communication terminal 9a, the accepting unit 92 accepts the operation, and the communication unit 91 transmits operation information indicating the operation content to the communication control device 5. As a result, in the communication control device 5, the communication unit 51, which serves as an acquiring unit, acquires the operation information, and the creation unit 53 creates a screen 601 based on the operation content indicated by the operation information. Then, when the communication unit 51 transmits data of the screen 601 to the communication terminal 9a, the communication unit 91 of the communication terminal 9a receives the data of the screen 601, and the display control unit 94 displays the screen 601 on the display 507 or the like of the communication terminal 9a. The communication control device 5 in this case is an example of an information processing device.

[0296] Next, a process in which the communication control device 5 creates a screen 601 to be displayed on the display 507 of the communication terminal 9a will be described. Note that in steps S13 and S15, the communication control device 5 performs the same process as step S14, and only the terminal on which the display is performed is different, so a description of steps S13 and S15 will be omitted. Figures 40 and 41 are flowcharts showing the process executed by the communication control device in the screen display process after the photography date and time is registered.

[0297] First, the creation unit 53 of the communication control device 5 creates a screen 601 to be displayed by the communication terminal 9a, as shown in FIG. 42. FIG. 42 is a diagram showing a screen displayed on the communication terminal 9a when a current predetermined area image is displayed after the image capture date and time has been registered. The screen 601 includes a display area 610 (an example of a first captured image display area), a display area 620 (an example of a three-dimensional image area), and a display area 630 (an example of a second captured image display area). In FIG. 42, the display areas 610, 620, and 630 are simultaneously displayed in the same size within the screen 601. However, they may be simultaneously displayed in different sizes, or one of them may be displayed by switching selection by participant A. Alternatively, the display areas 610, 620, and 630 may be displayed on each of the three displays of the screen 601.

[0298] The display area 610 displays a predetermined area image, which is a predetermined area in the current wide-field image (an example of the current captured image) obtained by capturing an image of an object using the imaging device 10 (see Figures 6(a) and (b)).

[0299] The display area 620 includes a three-dimensional model that represents the shape of an object (such as a table or pillar) in three dimensions, and displays part or all of a three-dimensional image in which coordinates (an example of first coordinates) in the wide-field image are associated with coordinates (an example of second coordinates).

[0300] The display area 630 displays a predetermined area image, which is a predetermined area in a past wide-field image (an example of a past captured image) obtained by capturing an object using the imaging device 10 (see Figures 6(a) and (b)).

[0301] Here, if the captured image is not a curved image such as a wide-field image, the display areas 610 and 630 display a predetermined area image whose predetermined area is the same as the captured area of ​​the captured image.

[0302] The display area 620 includes a three-dimensional model that represents the shape of the object in three dimensions, and displays part or all of a three-dimensional image in which coordinates (an example of first coordinates) of the wide-field image are associated with coordinates (an example of second coordinates).

[0303] Also here, as described above, processing unit 54 aligns display areas 610, 620, and 630. Then, as described above, creation unit 53 creates screen 601 including display area 610 displaying a predetermined area image that is a predetermined area in the current wide-field-of-view image, a second display area displaying at least a part of a three-dimensional image in which a first position (first coordinate) and a second position (second coordinate) in the wide-field-of-view image are associated, and display area 630 displaying a predetermined area image that is a predetermined area in the past wide-field-of-view image in which the first position (first coordinate) and a third position (third coordinate) are associated.

[0304] Also displayed on the screen 601 are a "Register" button 680 and a "Link" button 682. The "Register" button 680 is a button that is pressed when displaying, in the display area 620, an icon of a virtual camera IC1 whose shooting area is the angle of view (information) that specifies the predetermined area of ​​the predetermined area image currently being displayed in the display area 610, at a position corresponding to the shooting position of the image capturing device 10. The "Link" button 682 is a button that is pressed when performing processing to associate a specific image that indicates the position of the specific area 631a of a previously captured image with the current captured image.

[0305] Further, the display contents of the screen 601 will be described in detail below. Note that the virtual camera IC1 is used to identify a predetermined area related to each predetermined area image to be displayed in the display areas 610 and 630 (see FIG. 8), while the virtual camera IC2 is used to identify a three-dimensional image to be displayed in the display area 620.

[0306] S131: First, the creation unit 53 of the communication control device 5 identifies the current specified area image to be displayed in the display area 610, as shown in FIG. 42, from the current wide-field image of the content data received by the communication unit 51, based on the virtual angle of view of the virtual camera IC1 that has been set in advance.

[0307] S132: The processing unit 54 adjusts the position of the virtual camera IC2 to the shooting position associated with the current wide-field image (content data), and adjusts the virtual angle of view (an example of the second angle of view) of the virtual camera IC2 to the preset virtual angle of view (an example of the first angle of view) of the virtual camera IC1. As a result, the creation unit 53 creates a three-dimensional image to be displayed in the display area 620.

[0308] 42, pressing a "Register" button 680 performs the same processes as the above-described processes S113 to S116. This enables creation unit 53 to superimpose, in the display area 620, a corresponding predetermined area 621b and a diagram 623b indicating the line of sight of icon 622b (an example of an image indicating a first virtual camera) of virtual camera IC1 relative to center point CP1b of this corresponding predetermined area 621b, at a position corresponding to the current shooting position.

[0309] S133: The creation unit 53 determines whether there is an icon of the virtual camera IC1 relating to a past photographing position within a predetermined range from the current photographing position. The predetermined range is, for example, 3 m when an absolute position is assumed.

[0310] S134: If there is no previous shooting position within a predetermined distance from the current shooting position (S133: NO), the communication unit 51 determines whether an instruction to operate a previously captured image has been acquired. For example, the communication unit 51 determines whether an instruction to select (or specify) a specific previous virtual camera IC1 icon from among multiple previous virtual camera IC1 icons has been acquired by user A. If the communication unit 51 does not acquire an instruction to operate a previously captured image (S134; NO), the communication unit 51 performs processing S136.

[0311] S135: If there is a previous shooting position within a predetermined distance from the current shooting position (S133: YES), or if an instruction to operate on a previously captured image has been acquired (S134; YES), the creation unit 53 identifies a predetermined icon at the previous shooting position that is closest to the current shooting position (the position of the icon 622b of the virtual camera IC1) within a predetermined distance from the current shooting position. Here, the icon 622a is identified.

[0312] S136: As in process S116, the creation unit 53 superimposes, within the display area 620, a corresponding predetermined area 621b relating to the icon 622b of the current predetermined virtual camera IC1, and a line diagram 623b showing the line of sight of the icon 622b of the virtual camera IC1 relative to the center point CP1b of this corresponding predetermined area 621b.

[0313] S137: As in process S116, the creation unit 53 superimposes in the display area 620 a corresponding predetermined area 621a relating to the icon 622a of a past predetermined virtual camera IC1 and a line diagram 623a showing the line of sight of the icon 622a of the virtual camera IC1 relative to the center point CP1a of this corresponding predetermined area 621a.

[0314] S138: Then, the creation unit 53 superimposes a specific image indicating the position of a specific area 631a identified from the past captured image in a display area 630 (an example of a second captured image display area) that displays a past specific area image (an example of a second specific area image) corresponding to the icon 622a of the virtual camera IC1, as shown in Fig. 42. Furthermore, the creation unit 53 displays text information at the time of capturing the past specific area image in a display area 652 below the display area 630 on the screen 601.

[0315] S139: When the "Link" button 682 is pressed, the processing unit 54 determines, via the acquisition unit (reception unit 92, communication unit 51), whether there is an instruction for a link request to associate a specific image indicating the position of the specific area 631a identified from a previously captured image with the current captured image and the previously captured image.

[0316] S140: If there is an instruction for a linking request to associate (S139; YES), the processing unit 54 associates the specific area 631a being displayed in the display area 630 with the date and time of shooting and sound collection of the current predetermined area image (predetermined area) being displayed in the display area 610, and registers the associated information in the "specific area information" field of the movement history management DB 5004 (see FIG. 36). In addition, the processing unit 54 associates the past text (text information) being displayed in the display area 652 with the date and time of shooting and sound collection of the current predetermined area image (predetermined area) being displayed in the display area 610, and registers the associated information in the "voice text at a different date and time" field of the movement history management DB 5004 (see FIG. 36).

[0317] S141: As shown in Fig. 43, creation unit 53 creates screen 601 that displays a superimposed specific image of specific area 611a, which indicates the position of specific area 631a identified from the previously captured image currently being displayed in display area 630, within display area 610. Also, as shown in Fig. 43, creation unit 53 creates screen 601 that displays past text information currently being displayed in display area 652, in display area 650 below display area 610. Fig. 43 is a diagram showing a screen when a current predetermined area image is displayed on communication terminal 9a after the capture date and time has been registered.

[0318] According to Figures 42 and 43, a specific image showing the position of a specific area 631a identified from a previously captured image can be displayed superimposed on the specific area 611a of the current specified area image in the display area 610, making it easier for the user to identify areas of concern in the previously captured image (for example, areas with defects) in the current captured image as well.

[0319] Furthermore, according to Figures 42 and 43, past text information being displayed in display area 652 can be displayed in addition to the current text information being displayed in display area 650, so that the user can check areas of interest in past captured images not only by looking at the specific image but also by looking at the text information, making it easier for the user to understand.

[0320] <Modification of screen display process (after registration)> In addition, in process S133 of Figure 40, an example is shown in which it is determined whether there is an icon of virtual camera IC1 relating to a past shooting position within a predetermined range from the current shooting position, but it may also be determined whether there is a past shooting image similar to the current shooting image.

[0321] If it is determined that there is a previously captured image similar to the current captured image, the creation unit 53 identifies a predetermined icon at the previous capture position associated with the previously captured image similar to the current captured image in step S135. Note that the similarity of the images can be measured using, for example, an existing image similarity index.

[0322] Next, with reference to FIGS. 44 to 47, a screen display process after registration for process S14 among processes S13 to S15 will be described. In FIGS. 46 and 47, components with the same reference numerals as those in FIGS. 37 to 39 have the same roles or functions. Here, when participant A operates the communication terminal 9a, the accepting unit 92 accepts the operation, and the communication unit 91 transmits operation information indicating the operation content to the communication control device 5. As a result, in the communication control device 5, the communication unit 51, which serves as an acquiring unit, acquires the operation information, and the creation unit 53 creates a screen 601 based on the operation content indicated by the operation information. Then, when the communication unit 51 transmits data of the screen 601 to the communication terminal 9a, the communication unit 91 of the communication terminal 9a receives the data of the screen 601, and the display control unit 94 displays the screen 601 on the display 507 or the like of the communication terminal 9a. The communication control device 5 in this case is an example of an information processing device.

[0323] Next, a process in which the communication control device 5 creates a screen 601 to be displayed on the display 507 of the communication terminal 9a will be described. Note that in steps S13 and S15, the communication control device 5 performs the same process as step S14, and only the terminal on which the display is performed is different, so a description of steps S13 and S15 will be omitted. Figures 44 to 45 are flowcharts showing the process executed by the communication control device in the screen display process after the photography date and time is registered.

[0324] First, the creation unit 53 of the communication control device 5 creates a screen 601 to be displayed by the communication terminal 9a, as shown in FIG. 46. FIG. 46 is a diagram showing a screen displayed on the communication terminal 9a when a current predetermined area image is displayed after the image capture date and time has been registered. The screen 601 includes a display area 610 (an example of a first captured image display area), a display area 620 (an example of a three-dimensional image area), and a display area 630 (an example of a second captured image display area). In FIG. 46, the display areas 610, 620, and 630 are displayed simultaneously in the same size within the screen 601. However, they may be displayed simultaneously in different sizes, or one of them may be displayed by switching selection made by participant A. Alternatively, the display areas 610, 620, and 630 may be displayed on each of the three displays of the screen 601.

[0325] The display area 610 displays a predetermined area image, which is a predetermined area in the current wide-field image (an example of the current captured image) obtained by capturing an image of an object using the imaging device 10 (see Figures 6(a) and (b)).

[0326] The display area 620 includes a three-dimensional model that represents the shape of an object (such as a table or pillar) in three dimensions, and displays part or all of a three-dimensional image in which coordinates (an example of first coordinates) in the wide-field image are associated with coordinates (an example of second coordinates).

[0327] The display area 630 displays a predetermined area image, which is a predetermined area in a past wide-field image (an example of a past captured image) obtained by capturing an object using the imaging device 10 (see Figures 6(a) and (b)).

[0328] Here, if the captured image is not a curved image such as a wide-field image, the display areas 610 and 630 display a predetermined area image whose predetermined area is the same as the captured area of ​​the captured image.

[0329] The display area 620 includes a three-dimensional model that represents the shape of the object in three dimensions, and displays part or all of a three-dimensional image in which coordinates (an example of first coordinates) of the wide-field image are associated with coordinates (an example of second coordinates).

[0330] Also here, as described above, processing unit 54 aligns display areas 610, 620, and 630. Then, as described above, creation unit 53 creates screen 601 including display area 610 displaying a predetermined area image that is a predetermined area in the current wide-field-of-view image, a second display area displaying at least a part of a three-dimensional image in which a first position (first coordinate) and a second position (second coordinate) in the wide-field-of-view image are associated, and display area 630 displaying a predetermined area image that is a predetermined area in the past wide-field-of-view image in which the first position (first coordinate) and a third position (third coordinate) are associated.

[0331] Also displayed on the screen 601 are a "Register" button 680 and a "Link" button 682. The "Register" button 680 is a button that is pressed when displaying, in the display area 620, an icon of a virtual camera IC1 whose shooting area is the angle of view (information) that specifies the predetermined area of ​​the predetermined area image currently being displayed in the display area 610, at a position corresponding to the shooting position of the image capturing device 10. The "Link" button 682 is a button that is pressed when performing processing to associate a specific image that indicates the position of the specific area 611a of the current captured image with a previously captured image.

[0332] Further, the display contents of the screen 601 will be described in detail below. Note that the virtual camera IC1 is used to identify a predetermined area related to each predetermined area image to be displayed in the display areas 610 and 630 (see FIG. 8), while the virtual camera IC2 is used to identify a three-dimensional image to be displayed in the display area 620.

[0333] S151: First, the creation unit 53 of the communication control device 5 identifies the current specified area image to be displayed in the display area 610, as shown in FIG. 46, from the current wide-field image of the content data received by the communication unit 51, based on the virtual angle of view of the virtual camera IC1 that has been set in advance.

[0334] S152: The processing unit 54 adjusts the position of the virtual camera IC2 to the shooting position associated with the current wide-field image (content data), and adjusts the virtual angle of view (an example of the second angle of view) of the virtual camera IC2 to the virtual angle of view (an example of the first angle of view) of the virtual camera IC1 that has been set in advance. As a result, the creation unit 53 creates a three-dimensional image to be displayed in the display area 620.

[0335] 46, pressing a "Register" button 680 performs the same processes as the above-described processes S113 to S116. This enables creation unit 53 to superimpose, in display area 620, corresponding predetermined area 621b and a diagram 623b indicating the line of sight of icon 622b (an example of an image indicating a first virtual camera) of virtual camera IC1 relative to center point CP1b of corresponding predetermined area 621b, at a position corresponding to the current shooting position.

[0336] S153: The creation unit 53 determines whether there is an icon of the virtual camera IC1 relating to a past photographing position within a predetermined range from the current photographing position. The predetermined range is, for example, 3 m when an absolute position is assumed.

[0337] S154: If there is no previous shooting position within a predetermined distance from the current shooting position (S153: NO), the communication unit 51 determines whether an instruction to operate a previously captured image has been acquired. For example, the communication unit 51 determines whether an instruction to select (or specify) a specific past virtual camera IC1 icon from among multiple past virtual camera IC1 icons has been acquired by user A. If the communication unit 51 does not acquire an instruction to operate a previously captured image (S154; NO), the communication unit 51 performs processing S156.

[0338] S155: If there is a previous shooting position within a predetermined distance from the current shooting position (S153: YES), or if an instruction to operate on a previously captured image has been acquired (S154; YES), the creation unit 53 identifies a predetermined icon at the previous shooting position that is closest to the current shooting position (the position of the icon 622b of the virtual camera IC1) within a predetermined distance from the current shooting position. Here, the icon 622a is identified.

[0339] S156: As in process S116, the creation unit 53 superimposes in the display area 620 a corresponding predetermined area 621b relating to the icon 622b of the current predetermined virtual camera IC1 and a line diagram 623b showing the line of sight of the icon 622b of the virtual camera IC1 relative to the center point CP1b of this corresponding predetermined area 621b.

[0340] S157: As in process S116, the creation unit 53 superimposes in the display area 620 a corresponding predetermined area 621a relating to the icon 622a of a past predetermined virtual camera IC1 and a line diagram 623a showing the line of sight of the icon 622a of the virtual camera IC1 relative to the center point CP1a of this corresponding predetermined area 621a.

[0341] S158: Then, the creation unit 53 superimposes a specific image indicating the position of the specific area 611a identified from the current captured image in a display area 610 (an example of a first captured image display area) that displays a current specific area image (an example of a first specific area image) corresponding to the icon 622b of the virtual camera IC1, as shown in Fig. 46. Furthermore, the creation unit 53 displays text information at the time of capturing the current specific area image in a display area 650 below the display area 610 on the screen 601.

[0342] S159: When the "Link" button 682 is pressed, the processing unit 54 determines, via the acquisition unit (reception unit 92, communication unit 51), whether there is an instruction for a link request to associate a specific image indicating the position of the specific area 611a identified from the current captured image with a previously captured image.

[0343] S160: If there is an instruction for a link request to associate (S159; YES), the processing unit 54 associates the specific area 611a being displayed in the display area 610 with the date and time of shooting and sound collection of the past predetermined area image (predetermined area) being displayed in the display area 630, and registers the associated information in the "specific area information" field of the movement history management DB 5004 (see FIG. 36). In addition, the processing unit 54 associates the current text (text information) being displayed in the display area 650 with the date and time of shooting and sound collection of the past predetermined area image (predetermined area) being displayed in the display area 630, and registers the associated information in the "voice text at a different date and time" field of the movement history management DB 5004 (see FIG. 36).

[0344] S161: As shown in Fig. 47, creation unit 53 creates screen 601 that displays a superimposed specific image of specific area 631a, which indicates the position of specific area 611a identified from the current captured image being displayed in display area 610, within display area 630. Also, as shown in Fig. 47, creation unit 53 creates screen 601 that displays current text information being displayed in display area 650, in display area 652 at the bottom of display area 630. Fig. 47 is a diagram showing a screen when the current predetermined area image is displayed on communication terminal 9a after the capture date and time has been registered.

[0345] According to Figures 46 and 47, a specific image showing the position of a specific area 611a identified from the current captured image can be displayed superimposed on a specific area 631a of a past image of a specified area in the display area 630, making it easier for the user to identify areas of concern in the current captured image (for example, areas with defects) in the past captured image as well.

[0346] Furthermore, according to Figures 46 and 47, the current text information being displayed in display area 650 can be displayed in addition to the past text information being displayed in display area 652, so that the user can check any parts of the current captured image that interest them not only by looking at the specific image but also by looking at the text information, making it easier for the user to understand.

[0347] In addition, in process S153 of Figure 44, an example is shown in which it is determined whether there is an icon of virtual camera IC1 relating to a past shooting position within a predetermined range from the current shooting position, but it may also be determined whether there is a past shooting image similar to the current shooting image.

[0348] If it is determined that there is a previously captured image similar to the current captured image, the creation unit 53 identifies a predetermined icon at the previous capture position associated with the previously captured image similar to the current captured image in step S155. Note that the similarity of the images can be measured using, for example, an existing image similarity index.

[0349] In addition, in process S133 of Figure 40 and process S153 of Figure 44, an example is shown in which it is determined whether there is an icon of virtual camera IC1 relating to a past shooting position within a predetermined range from the current shooting position, but it may also be determined whether there is past text information similar to the current text information being displayed in the display area 650.

[0350] If it is determined that there is past text information similar to the current text information, the creation unit 53 identifies a predetermined icon at the past shooting position associated with the past text information similar to the current text information in step S135 or step S155. Note that the similarity of the text information can be measured using, for example, an existing text similarity index.

[0351] Fig. 48 is a diagram showing a screen on the communication terminal 9a when the current predetermined area image is displayed after the photographing date and time has been registered. In Fig. 48, the same reference numerals as those in Figs. 37 to 39 have the same roles or functions.

[0352] The creation unit 53 of the communication control device 5 identifies the current specified area image to be displayed in the display area 610, as shown in Figure 48, from the current wide-field image of the content data received by the communication unit 51, based on the virtual angle of view of the pre-set virtual camera IC1.

[0353] The processing unit 54 adjusts the position of the virtual camera IC2 to the shooting position associated with the current wide-field image, and adjusts the virtual angle of view of the virtual camera IC2 to the preset virtual angle of view of the virtual camera IC1. In this way, the creation unit 53 creates a three-dimensional image to be displayed in the display area 620.

[0354] Creation unit 53 determines whether there is past text information similar to the current text information being displayed in display area 650. If there is past text information similar to the current text information being displayed in display area 650, creation unit 53 identifies a predetermined icon at the past shooting position associated with the past text information similar to the current text information.

[0355] As in process S116, the creation unit 53 superimposes within the display area 620 a corresponding predetermined area 621b relating to the icon 622b of the current predetermined virtual camera IC1, and a line diagram 623b showing the line of sight of the icon 622b of the virtual camera IC1 relative to the center point CP1b of this corresponding predetermined area 621b.

[0356] In addition, in the display area 620, the creation unit 53 superimposes, as in process S116, a corresponding predetermined area 621a relating to the icon 622a of a past predetermined virtual camera IC1 and a line diagram 623a showing the line of sight of the icon 622a of the virtual camera IC1 relative to the center point CP1a of this corresponding predetermined area 621a.

[0357] 48, the creation unit 53 superimposes a specific image indicating the position of a specific area 631a identified from the past captured image in a display area 630 (an example of a second captured image display area) that displays a past predetermined area image corresponding to the icon 622a of the virtual camera IC1. Furthermore, the creation unit 53 displays text information of the past predetermined area image in a display area 652 below the display area 630 on the screen 601. When the "link" button 682 is pressed, the processing unit 54 receives an instruction for a link request to link the current captured image and the past captured image that have similar text information.

[0358] According to FIG. 48, the current captured image and the previously captured image having similar text information can be displayed on the screen 601, so that the user can easily recognize the current captured image and the previously captured image having similar text information.

[0359] It should be noted that the past captured images displayed in the display area 630 are not limited to images captured from a capture position aligned with the current captured image, but may be, for example, images captured from a capture position different from that of the current captured image (for example, images captured of another room, floor, or property with the same specifications).

[0360] [Major Effects of the Embodiments] As described above, according to this embodiment, the processing unit 54 associates a first position in the captured image obtained by photographing the object with a second position in the three-dimensional image including a three-dimensional area corresponding to the object, and the creation unit 53 (or creation unit 73, 93) creates a screen 601 including: a display area 610 that displays a predetermined area image, which is a predetermined area in the captured image obtained by photographing the object; a display area 620 that displays at least a part of the three-dimensional image in which the first position and the second position in the captured image are associated by the processing unit 54; and a display area 630 that displays a predetermined area image at a different date and time, which is a predetermined area in a wide-field-of-view image photographed by the photographing device 10.

[0361] This allows the user to understand what the predetermined area image is of and where it was taken by viewing the screen 601. Furthermore, even if the user is unable to photograph the back side of the object, the user can understand the condition of the back side of the object by viewing the screen 601. Therefore, by creating an image that complements the photographed image, it is possible to achieve the effect of improving user convenience.

[0362] Furthermore, by associating the photographing date and time, photographing location, angle of view currently being displayed, text (text information for the current date and time, text information for a different date and time), and specific area using the movement history management DB 5004 shown in FIG. 36, when participant A presses the "Register" button 680 while the predetermined area image is being displayed in the display area 610, the processing unit 54 can register the photographing date and time, photographing location, angle of view, text information, and specific area of ​​the photographed image currently being displayed on the screen 601 in the movement history management DB 5004. Also, as shown in FIG. 42, the creation unit 53 can superimpose an icon 622b of a virtual camera IC1, whose photographing area is the angle of view currently being displayed of the predetermined area image, at a position corresponding to the current photographing position in the display area 620. Furthermore, as shown in FIG. 42, the creation unit 53 can display an icon 622a of a virtual camera IC1 at a previous photographing position in the display area 620. 42, when the participant A or the like presses the icon 622a, the creation unit 53 creates a screen 601 so as to display a predetermined area image indicating the angle of view corresponding to the icon 622a and text information in the display area 630. This allows the participant A or the like to check the previously captured image associated with the position of the icon 622a of the virtual camera IC1, the specific area 631a associated with the captured image, and the text information.

[0363] 〔supplement〕 Although the embodiments have been described above, the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the scope of the present invention.

[0364] (1) Each function of each of the above embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each of the above-described functions.

[0365] (2) A (non-transitory) recording medium such as a DVD-ROM on which each of the above programs is stored can be provided domestically or internationally as a program product.

[0366] (3) There may be multiple CPUs 111, 301, and 501 as processors that are hardware.

[0367] (4) In the above embodiment, in Fig. 18, the text input by the user may be received by the receiving unit 92 without using the text generating unit 56, and the text data may be transmitted to the communication unit 51 of the communication control device 5 via the communication unit 91, and the processing unit 54 may register the text in the movement history management DB 5004. The record registered here is a record of the same shooting and sound collection date and time as the date and time when the communication unit 51 received the text data. [Explanation of symbols]

[0368] 1. Communication systems (examples of information processing systems) 3. Relay Device 5. Communication control device (an example of information processing device) 7. Communication terminal (an example of a display terminal, an example of an information processing device) 9, 9a, 9b Communication terminal (an example of a display terminal, an example of an information processing device) 10 Imaging equipment 51 Communication unit (an example of an acquisition unit) 52 Reception 53 Creation Department 54 Processing section 55 Authentication Department 56 Text Generation Unit 57 Specific Processing Department 71 Communications Department 72 Reception unit (an example of an acquisition unit) 73 Creation Department 74 Display control unit 75 Sound input / output control section 78 Connection 91 Communications Department 92 Reception unit (example of acquisition unit) 93 Creation Department 94 Display control unit 95 Sound input / output control section 98 Connection 600 screens 611, 611a, 631a Specific area 610 Display area (an example of a captured image display area or a first captured image display area) 620 Display area (example of 3D image display area) 624 Specific Images 630 display area (an example of a second captured image display area) 507 Display (an example of a display unit) 5001 User Device Management DB 5002 Virtual Room Management DB 5003 3D image management DB 5004 Travel Route Management DB [Prior art documents] [Patent documents]

[0369] [Patent Document 1] Japanese Patent Publication No. 2023-140923

Claims

1. a creating unit that creates a screen including a first captured image display area that displays a first predetermined area image that is a first predetermined area in a first captured image obtained by capturing an image of an object at a first capturing position with a capturing device, and a three-dimensional image display area that displays at least a part of a three-dimensional image aligned with the first captured image and also displays a capturing position image that indicates a position corresponding to a second capturing position at a predetermined capturing date and time; The creation unit an information processing device including on the screen a second captured image display area that displays a specific image indicating the position of the specific area superimposed on the second specified area image, based on a second specified area image that is a second specified area in a second captured image at the specified shooting date and time associated with the second capturing position and that is stored in a memory unit in association with the second capturing position, and a specific area identified from the second captured image.

2. the creation unit creates the screen that displays a shooting position image indicating a position corresponding to the first shooting position in the three-dimensional image display area.

2. The information processing device according to claim 1.

3. the creation unit creates the screen that displays text information in a text display area in association with a specific image that indicates the position of the specific area identified from the second captured image.

2. The information processing device according to claim 1.

4. the creation unit creates the second captured image display area to be included on the screen based on the first capturing position and the second capturing position.

2. The information processing device according to claim 1.

5. the creation unit includes, on the screen, the second photographed image display area that displays a specific image indicating a position of a specific area identified from the second photographed image by superimposing it on the second photographed image corresponding to the first photographed image based on the first photographed image; 2. The information processing device according to claim 1.

6. the creation unit creates the screen that displays the second text information corresponding to the first text information and the second text information corresponding to the second photographed image, based on first text information corresponding to the first photographed image and second text information corresponding to the second photographed image, and that displays the specific image superimposed on the second photographed image corresponding to the second text information.

4. The information processing device according to claim 3.

7. The image processing device further includes a processing unit that performs a process of associating the specific area identified from the second captured image with the first captured image.

2. The information processing device according to claim 1.

8. the creation unit creates the screen that displays text information in a text display area in association with a specific image that indicates the position of the specific area identified from the second captured image; the processing unit performs a process of associating the second captured image with the text information.

8. The information processing device according to claim 7.

9. a processing unit that performs processing to associate the specific area identified from the first captured image with the second captured image.

2. The information processing device according to claim 1.

10. the creation unit creates the screen that displays text information in a text display area in association with a specific image that indicates the position of the specific area identified from the first captured image; the processing unit performs a process of associating the second captured image with the text information.

10. The information processing device according to claim 9.

11. a specific processing unit that identifies the specific area from the first captured image or the second captured image, the specified area is an area specified by a user in the first captured image display area or the second captured image display area; The information processing device according to claim 1 .

12. the specific area is an area identified by image recognition processing using text information, which is text generated from a voice collected together with the captured image or text input. The information processing device according to claim 11.

13. execute a creation process to create a screen including a first captured image display area that displays a first predetermined area image, which is a first predetermined area in a first captured image obtained by capturing an image of an object at a first capturing position with a capturing device, and a three-dimensional image display area that displays at least a part of a three-dimensional image aligned with the first captured image and also displays a capturing position image that indicates a position corresponding to a second capturing position at a predetermined capturing date and time; The creation process includes, on the screen, a second captured image display area that displays a specific image indicating the position of the specific area superimposed on the second specified area image, based on a second specified area image that is a second specified area in a second captured image at the specified shooting date and time associated with the second capturing location and that is stored in a memory unit in association with the second capturing location, and a specific area identified from the second captured image.

14. A program for causing a computer to execute the method according to claim 13.

15. An information processing system having an information processing device and a display terminal that communicates with the information processing device, a creating unit that creates a screen including a first captured image display area that displays a first predetermined area image that is a first predetermined area in a first captured image obtained by capturing an image of an object at a first capturing position with a capturing device, and a three-dimensional image display area that displays at least a portion of a three-dimensional image aligned with the first captured image and also displays a capturing position image that indicates a position corresponding to the second capturing position at a predetermined capturing date and time; a display control unit that causes the screen created by the creation unit to be displayed on a display unit of the display terminal; and The creation unit includes on the screen a second captured image display area that displays a specific image indicating the position of the specific area superimposed on the second specified area image, based on a second specified area image that is a second specified area in a second captured image at the specified shooting date and time associated with the second capturing location and that is stored in a memory unit in association with the second capturing location, and a specific area identified from the second captured image.

Images