Information processing device, screen creation method, program, and information processing system

Abstract

The aim is to improve user convenience by supplementing the images obtained through photography. [Solution] The present disclosure relates to an information processing device having: a processing unit that associates position information indicating a first position in a captured image obtained by photographing an object with position information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object; a creation unit that creates a screen including: a first display area that displays a predetermined region image which is a predetermined region in the captured image; a second display area that displays at least a part of a first three-dimensional image generated based on design information, which is a three-dimensional image to which the first position and the second position in the captured image are associated by the processing unit; and a third display area that displays at least a part of a second three-dimensional image generated based on point cloud information.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a screen creation method, a program, and an information processing system.

Background Art

[0002] In recent years, as an imaging range including portions that cannot be fully confirmed within a normal viewing angle, a wide-view image (hereinafter referred to as a "wide-view image") having a wide viewing angle, such as a 360-degree image (also referred to as an omnidirectional image, a panoramic image, or a full-surround image) in which a full 360-degree circumference is imaged, is known. If all of such a wide-view image is to be displayed on a display terminal, since the wide-view image appears curved and is difficult to view, a predetermined area image indicating a predetermined area in the wide-view image is displayed on the display terminal, and the user views the predetermined area image (see Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, it has been difficult for the user to grasp what and where the predetermined area image that is displayed is capturing.

[0004] In addition, the imaging device captures objects around the imaging device to 360 degrees in a single shot from the imaging position, but it cannot capture the objects from 360-degree directions. Therefore, even for an object within the imaging range, the user cannot capture the state of the back side or the like of the object because the back side or the like of the object cannot be imaged.

[0005] Furthermore, it has been impossible for the user to grasp whether the predetermined area image corresponding to the time information is appropriate only by viewing the predetermined area image within the displayed range.

[0006] The above problems were the same even when the captured image was not an image that curved like a wide-view image. [[ID=3)5]]

[0007] This disclosure is made in light of the circumstances described above, and aims to improve user convenience by supplementing the captured images obtained through photography. [Means for solving the problem]

[0008] The disclosure relating to claim 1 is an information processing device having: a processing unit that associates position information indicating a first position in a captured image obtained by photographing an object with position information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object; a creation unit that creates a screen including: a first display area that displays a predetermined region image which is a predetermined region in the captured image; a second display area that displays at least a part of a first three-dimensional image generated based on design information, which is a three-dimensional image to which the first position and the second position in the captured image are associated by the processing unit; and a third display area that displays at least a part of a second three-dimensional image generated based on point cloud information. [Effects of the Invention]

[0009] According to this disclosure, creating images that complement the captured images obtained through photography has the effect of improving user convenience. [Brief explanation of the drawing]

[0010] [Figure 1] (a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a top view of the imaging device. [Figure 2] This is a diagram illustrating the use of the imaging device. [Figure 3] (a) is a hemispherical image (front) taken with the imaging device, (b) is a hemispherical image (back) taken with the imaging device, and (c) is an image represented using the Mercator projection. [Figure 4] (a) A conceptual diagram showing the state of the sphere being covered by a Mercator image, and (b) A diagram showing a full-sphere image. [Figure 5] This diagram shows the positions of a virtual camera and a predetermined region when a 360-degree spherical image is treated as a three-dimensional sphere. [Figure 6] (a) is a stereoscopic perspective view of Figure 5, (b) is a diagram showing the predetermined region image in the state of (a) on the display, (c) is a diagram showing the predetermined region after changing the viewpoint of the virtual camera IC in (a), and (d) is a diagram showing the predetermined region image in the state of (c) on the display. [Figure 7] This is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. [Figure 8] This is a conceptual diagram showing the relationship between a designated area and a point of interest. [Figure 9] This is a schematic diagram of the communication system according to the embodiment. [Figure 10] This is a hardware configuration diagram of the imaging device. [Figure 11] This is a hardware configuration diagram of the relay device. [Figure 12] This is a hardware configuration diagram of a communication control device and a communication terminal. [Figure 13] This is a diagram illustrating the functional configuration of the communication system according to the embodiment. [Figure 14] This is a conceptual diagram of the user device management table. [Figure 15] This is a conceptual diagram of the virtual room management table. [Figure 16A] This is a conceptual diagram of a three-dimensional image management table (part 1). [Figure 16B] This is a conceptual diagram of the three-dimensional image management table (part 2). [Figure 17] This is a sequence diagram illustrating the communication process of content data in a communication system. [Figure 18] This is a sequence diagram showing the process of initiating video recording and audio recording in a communication system. [Figure 19] This is a sequence diagram showing the process of stopping video recording and audio recording in a communication system. [Figure 20] This is a sequence diagram showing the process of playing back video and audio recordings in a communication system. [Figure 21] This is a diagram showing the recording data selection screen. [Figure 22]It is a flowchart showing the processes executed by the communication control device in the screen display process. [Figure 23] It is a flowchart showing the processes executed by the communication control device in the screen display process. [Figure 24] It is a flowchart showing the processes executed by the communication control device in the screen display process. [Figure 25] It is a diagram showing an initial display example of the screen to be displayed on the communication terminal 9a. [Figure 26] It is a diagram showing the state in which the position and the angle of view of the virtual camera IC2 are changed from the state of the first three-dimensional image shown in FIG. 25. [Figure 27] It is a diagram showing the first three-dimensional images displayed for each different position of the virtual camera IC2. [Figure 28] It is a diagram showing a screen displaying a state in which a predetermined area image is enlarged as the first three-dimensional image is enlarged. [Figure 29] It is a diagram showing a screen displaying a state in which a predetermined area image is reduced as the first three-dimensional image is reduced. [Figure 30] It is a diagram showing a screen on which a corresponding predetermined area, an icon of the virtual camera IC1, and the line of sight of the virtual camera IC1 are displayed with respect to the display area 620A. [Figure 31] It is a diagram showing a screen displaying a state in which the first three-dimensional image and the second three-dimensional image are changed as the time information is changed. [Figure 32] It is a diagram showing a screen displaying a state in which the first three-dimensional image and the second three-dimensional image are changed as the time information is changed. [Figure 33] It is a diagram showing a screen displaying a state in which the first three-dimensional image and the second three-dimensional image are changed as the time information is changed. [Figure 34] It is a diagram showing a screen on which progress identification information is superimposed and displayed. [Figure 35] It is a diagram showing a screen before the first difference identification information and the second difference identification information are superimposed and displayed. [Figure 36] It is a diagram showing a screen after the first difference identification information is superimposed and displayed. [Figure 37] This figure shows the screen after the first difference identification information and the second difference identification information are superimposed. [Figure 38] This figure shows different screens where the three-dimensional model changes even though the change in time information is the same. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings.

[0012] [Overview of 360° Spherical Images] Figures 1 to 8 illustrate the method for generating a 360° spherical image. A 360° spherical image, also known as a 360° panoramic image, is an example of a wide-field video with a broad field of view. Wide-field images also include simple panoramic images with a field of view of approximately 180°.

[0013] First, the external appearance of the imaging device 10 will be described using Figure 1. The imaging device 10 is a digital camera used to obtain the image that will be the basis for the 360-degree spherical image. Figure 1(a) is a left side view of the imaging device, Figure 1(b) is a front view of the imaging device, and Figure 1(c) is a top view of the imaging device.

[0014] As shown in Figure 1(a), the imaging device 10 is small enough for a person to hold in one hand. Also, as shown in Figures 1(a), (b), and (c), the imaging device 10 has an image sensor 103a on the front side and an image sensor 103b on the rear side. Furthermore, as shown in Figure 1(b), an operating section 115, such as a shutter button, is provided on the front side of the imaging device 10.

[0015] Next, the usage of the imaging device 10 will be explained using Figure 2. Figure 2 is an illustrative diagram of the imaging device in use. As shown in Figure 2, the imaging device 10 is connected to a relay device 3 installed on some kind of stand 2, and is used to photograph surrounding subjects, scenery, etc. In this case, two hemispherical images can be obtained by capturing subjects around the user with the image sensors 103a and 103b shown in Figure 1, respectively. Note that if the imaging device 10 does not transmit the 360-degree spherical image obtained through imaging to other communication terminals or systems, the relay device 3 is unnecessary.

[0016] Next, using Figures 3 and 4, we will outline the process from the image captured by the imaging device 10 to the creation of a full-sphere image. Figure 3(a) shows the hemispherical image (front) captured by the imaging device, Figure 3(b) shows the hemispherical image (rear) captured by the imaging device, and Figure 3(c) shows the image represented by the equirectangular projection (hereinafter referred to as the "equistratic projection image"). Alternatively, an image represented by the Mercator projection (hereinafter referred to as the "Mercator image") may also be used. Figure 4(a) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 4(b) shows the full-sphere image. The "equistratic projection image" is a full-sphere image in equirectangular format, as an example of the wide-field image described above.

[0017] As shown in Figure 3(a), the image obtained by the image sensor 103a becomes a curved hemispherical image (front side) due to the wide-angle lens 102a, such as a fisheye lens, as described later. Similarly, as shown in Figure 3(b), the image obtained by the image sensor 103b becomes a curved hemispherical image (rear side) due to the wide-angle lens 102b, such as a fisheye lens, as described later. The imaging device 10 then combines the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) to create an equirectangular projection image EC as shown in Figure 3(c).

[0018] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to overlay the equirectangular projection image EC so that it covers the sphere, as shown in Figure 4(a), and creates a full-sphere image CE as shown in Figure 4(b). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. OpenGL ES is merely one example of software that performs image processing, and the full-sphere image CE may be created using other software. The full-sphere image CE may be a still image or a video. Here, the imaging device 10 is described as generating a full-sphere image, but similar image processing or some of the image processing steps may be performed by the communication control device 5 or the communication terminals 7 and 9.

[0019] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is superimposed to cover the sphere, as shown in Figure 4(a), creating a full-sphere image as shown in Figure 4(b). In this way, the full-sphere image is represented as an image where the Mercator image is facing the center of the sphere.

[0020] As described above, the 360-degree spherical image CE is an image pasted over a sphere, which can cause discomfort to the human eye. Therefore, the communication terminals 7 and 9 can display a predetermined region of the 360-degree spherical image (hereinafter referred to as the "predetermined region image") as a flat image with minimal curvature (distortion), thereby providing a display that does not cause discomfort to the human eye. This will be explained using Figures 5 to 8.

[0021] Figure 5 shows the positions of the virtual camera and the predetermined region when the 360-degree spherical image is treated as a three-dimensional sphere. The virtual camera IC1 corresponds to the position of the virtual viewpoint of the user viewing the 360-degree spherical image CE, which is displayed as a three-dimensional sphere. In Figure 6, (a) is a stereoscopic perspective view of Figure 5, (b) is a diagram showing the predetermined region image in the state of (a) displayed on the screen, (c) is a diagram showing the predetermined region after changing the viewpoint of the virtual camera IC1 in (a), and (d) is a diagram showing the predetermined region image in the state of (c) displayed on the screen.

[0022] If the resulting spherical image CE is considered a solid sphere CS, then, as shown in Figure 5, the virtual camera IC1 is located inside the spherical image CE. A predetermined region T in the spherical image CE is the imaging region of the virtual camera IC1 and is identified by field-of-view information indicating the imaging direction and field of view of the virtual camera IC1 in the three-dimensional virtual space containing the spherical image CE.

[0023] Furthermore, zooming in on a predetermined region T can also be represented by moving the virtual camera IC1 closer to or further away from the 360-degree image CE. The predetermined region image Q is the image of the predetermined region T in the 360-degree image CE. Therefore, the predetermined region T can be determined by the field of view α and the distance f from the virtual camera IC1 to the 360-degree image CE.

[0024] Furthermore, when the virtual viewpoint of the virtual camera IC1 is moved (also called "changed") to the right (left side in the drawing) from the state shown in Figure 6(a) to the state shown in Figure 6(c), a predetermined region T in the 360-degree image CE is moved to a predetermined region T' accordingly, and the predetermined region image Q displayed on the predetermined display is changed to a predetermined region image Q'. As a result, the image shown in Figure 6(b) is changed to the image shown in Figure 6(d) displayed on the predetermined display.

[0025] Next, we will explain the relationship between the field of view information and the image of a predetermined region T using Figures 7 and 8. Figure 7 is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. Figure 8 is a conceptual diagram showing the relationship between a predetermined region and a point of interest (center point).

[0026] Here, let (r,θ,φ) be the arbitrary position coordinates when the center point CP shown in Figure 7 is expressed in spherical polar coordinates. (r,θ,φ) are the radial vector, polar angle, and azimuth angle, respectively. The radial vector r is the distance from the origin of the three-dimensional virtual space containing the full-sphere image to any point (center point CP in Figure 8), and is equal to the distance f shown in Figure 8.

[0027] Furthermore, as shown in Figure 8, if we consider the center of the predetermined region T, which is the imaging area of ​​the virtual camera IC1, as the center point CP in Figure 7, then the trigonometric function shown in (Equation 1) below generally holds true.

[0028] L / f = tan(α / 2) ... (Equation 1)

[0029] Here, f is the distance from the virtual camera IC1 to the center point CP. L is the distance between any vertex of the predetermined region T and the center point CP (2L is the diagonal). α is the field of view. In this case, the field of view information for identifying the predetermined region T can be represented by pan(θ), tilt(φ), and fov(α). Note that zooming in on the predetermined region T can be represented by widening or narrowing the range (arc) of the field of view α.

[0030] [Overview of the communication system] Next, the outline of the communication system 1 according to the embodiment will be explained using Figure 9. Figure 9 is a schematic diagram of the communication system according to the embodiment.

[0031] As shown in Figure 9, the communication system 1 of this embodiment consists of a communication control device 5, a camera 10, a relay device 3, a communication terminal 7, and communication terminals 9a and 9b. The communication terminals 9a and 9b are collectively referred to as "communication terminal 9". The communication control device 5, communication terminal 7, and communication terminal 9 are also examples of information processing devices. Furthermore, communication terminals 7 and 9 may also be referred to as "display terminals" that display images, etc.

[0032] Of these, the imaging device 10 is a digital camera for obtaining wide-field images (such as 360-degree images), as described above. The relay device 3 has the function of a cradle for charging the imaging device 10 and transmitting and receiving data. In addition, the relay device 3 can communicate data with the imaging device 10 via contacts, and can also communicate data with the communication control device 5 via the communication network 100. The communication network 100 includes, for example, the Internet, a LAN (Local Area Network), a (wireless) router, etc.

[0033] Furthermore, the communication control device 5 is, for example, a computer and can communicate data with the relay device 3, communication terminals 7 and 9 via the communication network 100. The communication control device 5 can also be referred to as an "information management device" because it manages field-of-view information and other data.

[0034] Communication terminals 7 and 9 are computers, such as notebook PCs (Personal Computers), and can communicate data with the communication control device 5 via the communication network 100. OpenGL ES is installed on communication terminals 7 and 9, and they create a predetermined region image (see Figure 6) from the 360-degree image received from the communication control device 5. The communication control device 5 may be composed of a single computer or multiple computers.

[0035] Furthermore, the camera 10 and relay device 3 are installed in designated locations by the organizer X at the site Sa, such as a construction site, exhibition hall, educational venue, or medical facility. The communication terminal 7 is operated by the organizer X. The communication terminal 9a is operated by participant A, such as a viewer, who is located remotely at the site Sa. Similarly, the communication terminal 9b is operated by participant B, such as a viewer, who is located remotely at the site Sa. Participants A and B may be in the same location or in different locations.

[0036] The communication control device 5 transmits (distributes) the wide-field image obtained from the imaging device 10 via the relay device 3 to the communication terminals 7 and 9. The communication control device 5 also transmits (distributes) the captured images obtained from each communication terminal 7 to the communication terminals 7 and 9. While the captured image sent from the imaging device 10 via the relay device 3 is a wide-field image, if a single-lens reflex camera is used instead of the imaging device 10, the captured image will be a typical narrow-field image. The captured image may be a video or a still image.

[0037] [Hardware configuration] Next, the hardware configuration of the imaging device 10, relay device 3, communication control device 5, and communication terminals 7 and 9 of this embodiment will be described in detail with reference to Figures 10 to 12.

[0038] <Hardware configuration of the imaging device> Figure 10 is a hardware configuration diagram of the imaging device. As shown in Figure 10, the imaging device 10 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output interface 116, a short-range communication circuit 117, an antenna 117a for the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network interface 121.

[0039] Of these, the imaging unit 101 includes wide-angle lenses 102a and 102b (hereinafter referred to as lenses 102 unless otherwise specified) capable of capturing images with a field of view of 180° or more to form a hemispherical image, and two image sensors 103a and 103b provided in correspondence with each of the lenses 102a and 102b.

[0040] Furthermore, the image sensors 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the lens 102a, 102b, etc., into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers for which various commands or parameters necessary for the operation of the image sensor are set. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example; it may have only one, or three or more.

[0041] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus).

[0042] The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. In addition, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, the control unit 115, the input / output interface 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network interface 121, among others.

[0043] The image processing unit 104 receives image data output from image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create data for an equirectangular projection image (an example of a wide-field image), which will be described later.

[0044] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0045] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 10, there may also be functions to display a preview or video on a display (for example, the display of an external terminal such as a smartphone that communicates with the imaging device 10 via a short-range communication circuit 117). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0046] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0047] The CPU 111 controls the overall operation of the imaging device 10 and performs necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress of processing by the image processing unit 104 and data of completed equirectangular projection images.

[0048] The control unit 115 is a collective term for various operation buttons, power switches, shutter buttons, and a touch panel that combines display and operation functions. The user inputs various imaging modes, imaging conditions, etc., by operating the control unit 115.

[0049] The input / output interface (I / F) 116 is a general term for interface circuits (such as USB interfaces) to external media such as SD cards or personal computers. The I / F 116 can be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded to external media via the I / F 116, or transmitted to an external terminal (device) via the I / F 116 as needed.

[0050] The short-range communication circuit 117 communicates with an external terminal (device) via an antenna 117a provided on the imaging device 10 using short-range wireless communication technologies such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi. The short-range communication circuit 117 can transmit equirectangular projection image data to the external terminal (device).

[0051] The electronic compass 118 calculates the orientation of the imaging device 10 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data.

[0052] The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) associated with the movement of the imaging device 10. Changes in angle are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

[0053] The acceleration sensor 120 is a sensor that detects acceleration in three axes.

[0054] The imaging device 10 can also calculate its own orientation (angle relative to the direction of gravity) using an electronic compass 118, an acceleration sensor 120, etc. Furthermore, by providing the acceleration sensor 120, the accuracy of image correction can be improved in the imaging device 10.

[0055] Network I / F 121 is an interface for data communication using a communication network 100 such as the Internet via a router or the like. Furthermore, the hardware configuration of the imaging device 10 is not limited to that shown herein, and any configuration that can realize the functional configuration of the imaging device 10 is acceptable. In addition, at least a part of the above hardware configuration may reside on the relay device 3 or the communication network 100.

[0056] <Hardware configuration of the relay device> Figure 11 is a hardware configuration diagram of the relay device 3. Note that Figure 11 is a hardware configuration diagram when the relay device 3 is a cradle with wireless communication capabilities.

[0057] As shown in Figure 11, the relay device 3 includes a CPU 310, ROM 302, RAM 303, EEPROM 304, CMOS sensor 305, bus line 310, communication unit 313, antenna 313a, positioning unit 314, and input / output I / F 316.

[0058] Of these, the CPU 301 controls the operation of the entire relay device 3. The ROM 302 stores programs used to drive the CPU 301, such as the IPL (Initial Program Loader). The RAM 303 is used as the work area for the CPU 301.

[0059] The EEPROM (Electrically Erasable and Programmable ROM) 304 reads or writes data according to the control of the CPU 301. The EEPROM 304 stores the operating system (OS) executed by the CPU 301, other programs, and various data.

[0060] The CMOS (Complementary Metal Oxide Semiconductor) sensor 305 is a solid-state image sensor that captures an image of a subject and obtains image data according to the control of the CPU 301.

[0061] The communication unit 313 uses the antenna 313a to communicate with the communication network 100 via wireless communication signals.

[0062] The positioning unit 314 receives positioning signals including the location information (latitude, longitude, and altitude) of the relay device 3 from GNSS (Global Navigation Satellite System) satellites such as GPS (Global Positioning Systems) satellites, or from IMES (Indoor Messaging System) as an indoor GPS.

[0063] The input / output interface (I / F316) is an interface circuit (such as a USB interface) that is electrically connected to the input / output interface (I / F116) of the imaging device 10. The input / output interface (I / F316) can be wireless or wired.

[0064] Bus line 310 is an address bus, data bus, etc., used to electrically connect various components such as the CPU 301.

[0065] <Hardware configuration of communication control devices and communication terminals> Figure 12 shows the hardware configuration of the communication control device 5. Note that the hardware configurations of the communication terminals 7 and 9 are the same as those of the communication control device 5, so their explanation is omitted.

[0066] The communication control device 5, as a computer, includes a CPU 501, ROM 502, RAM 503, SSD 504, external device connection I / F 505, network I / F 506, display 507, operation unit 508, media I / F 509, bus line 510, CMOS sensor 511, speaker 512, and positioning unit 514, as shown in Figure 12.

[0067] Of these, the CPU 501 controls the operation of the entire communication control device 5. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501.

[0068] SSD504 reads or writes various types of data according to the control of CPU501. Note that if the communication terminals 7 and 9 are smartphones or the like, SSD504 may not be required. Alternatively, an HDD (Hard Disk Drive) may be used instead of SSD504.

[0069] The External Device Connection I / F 505 is an interface for connecting various external devices. These external devices include displays, speakers, keyboards, mice, USB memory sticks, and printers.

[0070] Network I / F 506 is an interface for data communication via communication network 100.

[0071] Display 507 is a type of display unit that displays various images, such as liquid crystal displays (LCDs) or organic EL (Electro-Luminescence) displays.

[0072] The operation unit 508 is an input means for selecting and executing various instructions such as various operation buttons, power switches, shutter buttons, and touch panels, selecting the object to be processed, and moving the cursor.

[0073] The media interface 509 controls the reading or writing (storage) of data to or from the recording media 509m, such as flash memory. The recording media 509m includes DVDs and Blu-ray Discs (registered trademarks), etc.

[0074] The CMOS sensor 511 is a type of imaging means that captures an image of a subject according to the control of the CPU 501 and obtains image data. A CCD sensor may be used instead of a CMOS sensor.

[0075] Speaker 512 is a circuit that converts electrical signals into physical vibrations to produce sound such as music and speech.

[0076] The positioning unit 314 receives positioning signals containing location information (latitude, longitude, and altitude) of communication terminals 7 and 9 from GNSS satellites such as GPS satellites, or from IMES as an indoor GPS.

[0077] Bus line 510 is an address bus, data bus, etc., used to electrically connect various components such as the CPU 501.

[0078] [Functional configuration of the embodiment] Next, the functional configuration of the first embodiment will be described using Figures 13 to 16.

[0079] <Functional configuration of the imaging device> As shown in Figure 13, the imaging device 10 includes a reception unit 12, a detection unit 13, an imaging unit 16, a sound collection unit 17, a connection unit 18, and a storage / reading unit 19. Each of these units is a function or means realized by any of the components shown in Figure 10 operating according to instructions from the CPU 111 that follow a program for imaging and storage deployed on SRAM 113 to DRAM 114.

[0080] Furthermore, the imaging device 10 has a storage unit 1000 constructed from ROM 112, SRAM 113, and DRAM 114 as shown in Figure 10.

[0081] (Functional configuration of each part of the imaging device) The reception unit 12 of the imaging device 10 is realized by the processing of the operation unit 115 to the CPU 111, and receives operation input from the user.

[0082] The detection unit 13 is mainly implemented by processing the electronic compass 118, gyro sensor 119, and acceleration sensor 120, etc., from the CPU 111, and obtains attitude information by detecting the attitude of the imaging device 10.

[0083] The imaging unit 16 is mainly realized by processing from the CPU 111 to the imaging unit 101, image processing unit 104, imaging control unit 105, and CPU 111, and captures images of landscapes, etc., and obtains captured images.

[0084] The sound collection unit 17 is mainly realized by processing performed by the CPU 111 on the sound processing unit 109, and it collects sounds from the surrounding area of ​​the imaging device 10.

[0085] The connection section 18 is mainly implemented by processing the input / output interface 116 from the CPU 111 and communicates data with the relay device 3.

[0086] The memory / read unit 19 is mainly implemented by the CPU 111 and stores various data (or information) in the memory unit 1000 and reads various data (or information) from the memory unit 1000.

[0087] <Functional configuration of the relay device> As shown in Figure 13, the relay device 3 has a communication unit 31 and a connection unit 38. Each of these units is a function or means realized by any of the components shown in Figure 11 operating according to instructions from the CPU 301 that follow the program for the relay device 3 deployed from the EEPROM 304 onto the RAM 303.

[0088] (Functional configuration of relay device 3) The communication unit 31 of the relay device 3 is mainly realized by processing from the CPU 301 shown in Figure 11 to the communication unit 313, and performs data communication between the imaging device 10 and the communication control device 5 via the communication network 100.

[0089] The connection section 38 is mainly implemented by processing the input / output interface 316 from the CPU 301 and communicates data with the imaging device 10.

[0090] <Functional Configuration of Communication Control Device> Next, the functional configuration of the communication control device 5 will be described in detail using Figure 13. The communication control device 5 includes a communication unit 51, a receiving unit 52, a creation unit 53, a processing unit 54, an authentication unit 55, and a storage / reading unit 59. Each of these units is a function or means realized by the operation of any of the components shown in Figure 12 by instructions from the CPU 501 according to the program for the communication control device 5 deployed from the SSD 504 onto the RAM 503.

[0091] Furthermore, the communication control device 5 has a storage unit 5000 constructed from the RAM 503 and HD 504 shown in Figure 12. This storage unit 5000 contains a user / device management DB 5001, a virtual room management DB 5002, and a three-dimensional image management DB 5003.

[0092] (User / Device Management Database) Figure 14 is a conceptual diagram of the user / device management table. The user / device management DB5001 consists of the user / device management table shown in Figure 14. In the user / device table, the user ID (or device ID), password, name, user image, and IP address are stored and managed in association with each other.

[0093] Of these, the User ID is an example of user identification information used to identify a user (e.g., organizer X, participant A, B). The Device ID is an example of device identification information used to identify a device such as the imaging device 10. Note that if a head-mounted display or the like is used in addition to the imaging device 10, the head-mounted display or the like will also be treated as a device.

[0094] The name is the name of the user or device. Note that each user name may also be the name of the user's communication terminal.

[0095] User images are pre-registered by each user and include images that are schematic representations of each user's face, or photographs of the user's face.

[0096] An IP address is an example of destination identification information for devices such as communication terminals 7 and 9, and imaging device 10, used by the user.

[0097] (Virtual Room Management Database) Figure 15 is a conceptual diagram of the virtual room management table. The virtual room management DB5002 is composed of the virtual room management table shown in Figure 15. The virtual room management table stores and manages the virtual room ID, virtual room name, device ID, organizer ID, participant ID, content ID, content URL (information on the storage location of image and sound content data), and 3D image ID in association with each other.

[0098] Among these, the virtual room ID is an example of virtual room identification information used to identify a virtual room.

[0099] The virtual room name is the name of the virtual room, and it is assigned by the user or other user.

[0100] The device ID is the same as the device ID in Figure 14, and is the ID of the device that joined the virtual room indicated by the virtual room ID in the same record.

[0101] The Organizer ID is an example of organizer identification information used to specifically identify the Organizer ID among the User IDs in Figure 14, and is the ID of the organizer who participated in the virtual room indicated by the virtual room ID in the same record.

[0102] The participant ID is an example of participant identification information used to specifically identify the participant among the user IDs in Figure 14, and is the ID of the participant who joined the virtual room indicated by the virtual room ID in the same record.

[0103] Content ID is an example of content identification information used to identify image and sound content data. In this case, the image is a wide-field image obtained at the time of capture, and the sound is sound (including speech) obtained at the same time of capture.

[0104] A content URL is an example of content storage location information that indicates where the content (wide-field image, sound information) data is stored. Along with the content data, the content URL also stores the time of capture (video recording) and sound recording (audio recording), as well as the capture location (absolute location on Earth).

[0105] The three-dimensional image ID is an example of three-dimensional image identification information that identifies the first three-dimensional image displayed in the display area 620A and the second three-dimensional image displayed in the display area 620B, as described below. Here, the first three-dimensional image is a three-dimensional image displayed in the display area 620A based on design data, and the second three-dimensional image is a three-dimensional image displayed in the display area 620B based on point cloud data acquired by a 3D scanning device, etc. Note that the first three-dimensional image and the second three-dimensional image may be managed with separate IDs.

[0106] (3D image management DB) (Part 1) Figure 16A is a conceptual diagram of the three-dimensional image management table (part 1). The three-dimensional image management DB5003 is composed of the three-dimensional image management table shown in Figure 16A. In the three-dimensional image management table shown in Figure 16A, the model ID, location information, and point cloud acquisition date are associated and managed for each three-dimensional image ID.

[0107] The model ID is an example of three-dimensional model identification information used to identify a three-dimensional model. The three-dimensional model is generated based on point cloud data acquired by, for example, a TOF (Time-of-Flight) camera.

[0108] Location information is information that indicates the position of a three-dimensional model in a three-dimensional virtual space using three-dimensional coordinates (XYZ). For example, it is indicated by the three-dimensional coordinates of eight points that define the rectangular space occupied by the three-dimensional model. Location information is managed in association with an absolute position on Earth, based on a positioning signal that includes the position information (latitude, longitude, and altitude) of a TOF camera received from a GNSS satellite such as a GPS satellite, or from IMES as an indoor GPS, and the three-dimensional coordinates of the point cloud relative to the TOF camera's reference position.

[0109] The point cloud acquisition date is an example of information indicating the date on which the point cloud (point cloud data) was acquired, for example, by a TOF camera. Figure 16A shows an example where the date of acquisition of point cloud data is shown in year, month, and day format. The point cloud acquisition date may also be information indicating the date and time the point cloud data was acquired.

[0110] (Part 2) Figure 16B is a conceptual diagram of the three-dimensional image management table (part 2). The three-dimensional image management DB5003 is composed of the three-dimensional image management table shown in Figure 16B. The three-dimensional image management table shown in Figure 16B is a table for managing attribute information that indicates the attributes of the parts (three-dimensional models) that make up a structure contained in a three-dimensional virtual space. In the three-dimensional image management table shown in Figure 16B, attribute information such as part number, material information, dimension information, color information, material information, location information, and construction date information are associated and managed for each three-dimensional image ID. Part number, material information, dimension information, color information, material information, location information, and construction date information are obtained from design data, for example.

[0111] Component information is information that identifies components such as walls, floors, ceilings, windows, pipes, and doors, which are made up of parts.

[0112] Dimensional information is information that identifies the dimensions of a part in virtual space, and is represented, for example, by numerical values ​​in the three axes of X, Y, and Z.

[0113] Color information identifies the color of a part, while material information identifies the material of a part.

[0114] Location information identifies the position of a component in virtual space and is represented, for example, by coordinates in the three axes of X, Y, and Z. This makes it possible to identify whether or not multiple components are adjacent to each other. Location information is obtained from design data such as CAD (Computer-Aided Design) data and BIM (Building Information Modeling) data.

[0115] Construction date information indicates the planned date on which a component will be installed in the real world. This makes it possible to identify a structure excluding components that have not yet been installed at a given point in time. Furthermore, if the planned installation date of a component changes in the real world, the construction date information may be updated to reflect the revised date. In the three-dimensional image management table shown in Figure 16B, construction period information, which indicates the period over which a component will be installed, may also be obtained, for example, from design data and managed in association with the construction date information.

[0116] As explained above, the location information in Figures 16A and 16B is managed in correspondence with absolute positions on Earth and indoors. As an example, by associating the origin of the location information in Figures 16A and 16B (X=0, Y=0, Z=0) with absolute positions on Earth (latitude, longitude, altitude), all coordinates in the first and second three-dimensional images, including the three-dimensional model and parts, are associated with absolute positions on Earth.

[0117] Furthermore, the 3D image management DB5003 may manage 3D point clouds, mesh objects, textured mesh objects, etc., instead of 3D models.

[0118] (Functional configuration of each communication control device) Next, we will explain in detail the functional configuration of the communication control device 5 using Figure 13.

[0119] The communication unit 51 of the communication control device 5 is mainly implemented by processing from the CPU 501 shown in Figure 12 to the network interface 505, and performs data communication with other devices (relay device 3, communication terminals 7 and 9) via the communication network 100. The communication unit 51 also acts as an acquisition unit, acquiring instruction information indicating instructions sent from the communication terminals 7 and 9.

[0120] The reception unit 52 is implemented by the processing of the operation unit 508 to the CPU 501 and receives operation input from the user (in this case, the system administrator, etc.).

[0121] The creation unit 53 is mainly implemented by the CPU 501 and uses data stored in the storage unit 5000 to create screens to be sent to each communication terminal 7, 9. For example, the creation unit 53 creates a screen 600 that includes a display area 610 that displays a predetermined area image, which is a predetermined region in a wide-field image (an example of a captured image) captured by the imaging device 10, and display areas 620A and 620B that display at least a portion of a first three-dimensional image and a second three-dimensional image, respectively, which are associated with a first position and a second position in the wide-field image by the processing unit 54 (see Figure 25).

[0122] The processing unit 54 is mainly implemented by the CPU 501 and performs the process of associating position information indicating the position in the captured image obtained by photographing the object with the imaging device 10 (an example of a first position) with position information indicating the position in the first three-dimensional image and the second three-dimensional image, which include the three-dimensional region corresponding to the object (an example of a second position). The processing unit 54 can also be referred to as the "association unit".

[0123] The authentication unit 55 performs authentication to determine whether each user is a legitimate person to use the virtual room.

[0124] The memory / read unit 59 is mainly implemented by the CPU 501, and stores various data (or information) in the memory unit 5000 and reads various data (or information) from the memory unit 5000.

[0125] <Functional Configuration of Communication Terminal 7> Next, the functional configuration of the communication terminal 7 will be described in detail using Figure 13. The communication terminal 7 includes a communication unit 71, a receiving unit 72, a creation unit 73, a display control unit 74, an audio input / output control unit 75, a connection unit 78, and a storage / reading unit 79. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 501 that follow the program for the communication terminal 7 deployed from the SSD 504 onto the RAM 503.

[0126] The communication unit 71 of the communication terminal 7 is mainly realized by processing from the CPU 501 shown in Figure 12 to the network I / F 505, and performs data communication with other devices (communication control device 5) via the communication network 100.

[0127] The reception unit 72 is primarily implemented by the processing of the operation unit 508 to the CPU 501, and receives instructions from the user (in this case, the organizer X) through operation input. The reception unit 72 also functions as an acquisition unit, and can be said to acquire instructions made by the user's operation.

[0128] The creation unit 73 is mainly implemented by the processing of the CPU 501 and uses data stored in the memory unit 7000 to create a screen for display on its own terminal (communication terminal 7). When the creation unit 53 of the communication control device 5 creates the screen, the communication terminal 7 may or may not have a creation unit 73.

[0129] The display control unit 74 is mainly implemented by the processing of the CPU 501 and controls the display of various images on the display 507 of the communication terminal 7 or an external display connected to the external device connection I / F 505.

[0130] The sound input / output control unit 75 is primarily implemented by the CPU 501 of the communication terminal 7 and controls the collection of sound from an external microphone connected to the external device connection I / F 505. If the communication terminal 7 has a built-in microphone, it controls the collection of sound from this microphone. The sound input / output control unit 75 also controls the output of sound to the speaker 512 of the communication terminal 7 or to an external speaker connected to the external device connection I / F 505.

[0131] The creation unit 73 is mainly implemented by the processing of the CPU 501, and adds narration, captions, etc. to the content data recorded and audio recorded by the communication terminal 7 to create content data for educational purposes, etc.

[0132] The memory / read unit 79 is mainly implemented by the CPU 501, and stores various data (or information) in the memory unit 7000 and reads various data (or information) from the memory unit 7000.

[0133] <Functional configuration of communication terminal 9> Next, we will explain in detail the functional configuration of the communication terminal 9 using Figure 13.

[0134] The communication terminal 9 includes a communication unit 91, a reception unit 92, a creation unit 93, a display control unit 94, an audio input / output control unit 95, a connection unit 98, and a storage / reading unit 99. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 501 that follow the program for the communication terminal 9 deployed from the SSD 504 onto the RAM 503.

[0135] Furthermore, the communication terminal 9 has a storage unit 9000 constructed from the RAM 503 and SSD 504 shown in Figure 12.

[0136] The communication unit 91 of the communication terminal 9 is mainly implemented by processing from the CPU 501 to the network interface 505, and performs data communication with other devices (communication control device 5) via the communication network 100.

[0137] The reception unit 92 is primarily implemented by the processing performed by the operation unit 508 on the CPU 501, and receives instructions from the user (in this case, the participant) through operational input. The reception unit 92 also functions as an acquisition unit, and can be said to acquire instructions made by the user's operations.

[0138] The creation unit 93 is mainly implemented by the processing of the CPU 501 and uses data stored in the storage unit 9000 to create a screen for display on its own terminal (communication terminal 9). When the creation unit 53 of the communication control device 5 creates the screen, the communication terminal 9 may or may not have a creation unit 93.

[0139] The display control unit 94 is mainly implemented by the processing of the CPU 501 and controls the display of various images on the display 507 of the communication terminal 9 or an external display connected to the external device connection I / F 505.

[0140] The sound input / output control unit 95 is primarily implemented by the CPU 501 of the communication terminal 9 and controls the collection of sound from an external microphone connected to the external device connection I / F 505. If the communication terminal 9 has a built-in microphone, it controls the collection of sound from this microphone. The sound input / output control unit 95 also controls the output of sound to the speaker 512 of the communication terminal 9 or to an external speaker connected to the external device connection I / F 505.

[0141] The connection unit 98 is mainly implemented by processing from the CPU 501 to the external device connection I / F 505, and performs data communication with external devices connected via wired or wireless connection.

[0142] The memory / read unit 99 is mainly implemented by the CPU 501, and stores various data (or information) in the memory unit 9000 and reads various data (or information) from the memory unit 9000.

[0143] [Processing or operation of the embodiment] Next, the processing or operation of the embodiment will be described using Figures 17 to 38. Note that the following processing takes place after the imaging device 10, communication terminals 7 and 9 have already joined the same virtual room.

[0144] <Communication processing of content data in communication systems> First, the communication processing of content data in communication system 1 will be explained using Figure 17. Figure 17 is a sequence diagram showing the communication processing of wide-field images and each angle of view information in the communication system. In this embodiment, we will explain the case where the shooting device 10, the organizer X's communication terminal 7, participant A's communication terminal 9a, and participant B's communication terminal 9b are in the same virtual room. When the virtual room is constructed, the storage / reading unit 59 adds one record to the virtual room management DB (see Figure 15) and manages the virtual room ID, virtual room name, device ID, organizer ID, and participant ID in association. The content ID, content URL, and three-dimensional image ID will be stored later. Note that the processing S11 to S15 in Figure 17 is repeated, for example, about 30 or 60 times per second.

[0145] S11: In the imaging device 10, the imaging unit 16 captures a full 360-degree image of the area Sa and collects sound to obtain content data (wide-field image, sound information), after which the connection unit 18 sends the content data to the relay device 3. In this case, the connection unit 18 also sends a virtual room ID to identify the virtual room in which the imaging device 10 is participating, and a device ID to identify the imaging device 10. As a result, the connection unit 38 of the relay device 3 obtains the content data, virtual room ID, and device ID.

[0146] S12: In the relay device 3, the communication unit 31 transmits the content data, virtual room ID, and device ID acquired by the connection unit 38 in processing S11 to the communication control device 5 via the communication network 100. As a result, the communication unit 51 in the communication control device 5 receives the content data, virtual room, and device ID.

[0147] Alternatively, the camera 10 may transmit the content data, virtual room ID, and device ID to the communication terminal 7 instead of the relay device 3 (S11d). In this case, the communication terminal 7 transmits the content data, virtual room ID, and device ID to the communication control device 5 (S12d).

[0148] S13: In the communication control device 5, the storage / reading unit 59 searches the virtual room management DB 5002 based on the virtual room ID received in processing S12, and reads the user IDs (organizer ID and participant ID) of users participating in the same virtual room as the camera 10. The storage / reading unit 59 also searches the user / device management DB 5001 based on the retrieved organizer ID and participant ID, and reads the corresponding user image and IP address of the communication terminal 7 for organizer X, as well as the corresponding user images and IP addresses of the communication terminals 9a and 9b for participants A and B. The communication unit 51 then refers to the IP address of the communication terminal 7 and transmits screen data, including the content data received in processing S12, to the communication terminal 7. As a result, the communication unit 71 of the communication terminal 7 receives the screen data. The detailed processing at this time (screen display processing) will be described later.

[0149] S14: The communication unit 51 of the communication control device 5 refers to the IP address of the communication terminal 9a and transmits the content data received in process S12 to the communication terminal 9a. As a result, the communication unit 91 of the communication terminal 9a receives the content data. The detailed processing at this time (screen display processing) will be described later.

[0150] S15: Similarly, the communication unit 51 of the communication control device 5 refers to the IP address of the communication terminal 9b and transmits the content data received in process S12 to the communication terminal 9b. As a result, the communication unit 91 of the communication terminal 9b receives the content data. The detailed processing at this time (screen display processing) will be described later.

[0151] <Process for initiating video and audio recording in a communication system> Next, Figure 18 will be used to explain the process of starting video recording and audio recording in communication system 1. Figure 18 is a sequence diagram showing the process of starting video recording and audio recording in the communication system.

[0152] S31: First, at the organizer X's communication terminal 7, the reception unit 72 receives the command from organizer X to start recording and audio recording.

[0153] S32: Before recording video and audio, the communication unit 71 of the communication terminal 7 sends a sharing instruction for field of view information to the communication control device 5. This sharing instruction includes the virtual room ID of the virtual room in which the communication terminal 7 is participating, and the device ID of the camera 10. As a result, the communication unit 51 of the communication control device 5 receives the sharing instruction for field of view information.

[0154] S33: In the communication control device 5, the storage / reading unit 59 sets the content URL and the field of view information URL of the virtual room management DB (see Figure 15). Then, the communication unit 51 sends a recording start instruction and a field of view information upload request to the communication terminal 7. This instruction includes information indicating the content URL, which is the location where the communication terminal 7 will save the recorded content data. This request also includes information indicating the field of view information URL for saving the field of view information. As a result, the communication unit 71 in the communication terminal 7 receives the recording start instruction and the field of view information upload request.

[0155] S34: The communication unit 51 also sends a request to the communication terminal 9a to upload the field of view information. This request includes information about a URL for saving the field of view information. As a result, the communication terminal 9a receives the request for uploading the field of view information from the communication unit 91.

[0156] S35: Similarly, the communication unit 51 sends a request to the communication terminal 9b to upload the field of view information. This request includes information about a URL for saving the field of view information. As a result, the communication terminal 9b receives the request for uploading the field of view information from the communication unit 91.

[0157] S36: Next, in the communication terminal 7, the storage / reading unit 79 acts as both a recording unit and an audio recording unit, and begins recording the content data received in process S13 as shown in Figure 17. In the case of process S12d as shown in Figure 17, the communication terminal 7 may also begin recording the content data received from the camera 10 in process S11d, rather than the content data received from the communication control device 5 in process S13.

[0158] S37: In the communication terminal 7, for example, while displaying a predetermined region image (see Figure 6(b)), which is an image of a predetermined region (see Figure 6(a)) of the wide-field image received in processing S13, the reception unit 72 receives a request from the organizer X to change the field of view, and the display control unit 74 displays a predetermined region image (see Figure 6(d)), which is the changed predetermined region (see Figure 6(c)) of the same wide-field image. In this case, the reception unit 72 also acts as an acquisition unit, and when it receives a request from the user (in this case, the organizer X) to display a predetermined region in the wide-field image, it acquires field of view information (pan, tilt, fov) to identify the predetermined region to be displayed on the display 507 in the wide-field image. Then, the communication unit 71 transmits the field of view information to the field of view information URL (communication control device 5) received in processing S33 to identify the changed predetermined region. This field of view information includes the user ID of the organizer X of the communication terminal 7, which is the source of the transmission. As a result, the communication control device 5 receives the field of view information from the communication unit 51. The storage / reading unit 59 then stores the user ID, the source IP address, the field of view information, and the timestamp in the field of view information management DB. This timestamp indicates the time when the field of view information was received by process S37.

[0159] S38: The communication terminal 9a and the communication control device 5 also perform the same processing as in processing S37, independently of processing S37. In this case, the user ID transmitted is the user ID of participant A.

[0160] S39: In the communication terminal 9b and the communication control device 5, the same processing as in process S37 is performed independently of processes S37 and S38. In this case, the user ID transmitted is the user ID of participant B.

[0161] Furthermore, processes S37 to S39 may be executed collectively to the communication control device 5 at the end of recording.

[0162] <Process to stop recording video and audio in a communication system> Next, Figure 19 will be used to explain the process of stopping video recording and audio recording in communication system 1. Figure 19 is a sequence diagram showing the process of stopping video recording and audio recording in the communication system.

[0163] S51: First, at the organizer X's communication terminal 7, the reception unit 72 receives a request from organizer X to stop recording and audio recording.

[0164] S52: The memory / reading unit 79 stops recording and audio recording of the content data.

[0165] S53: The communication unit 71 uploads (transmits) the recorded content data to the predetermined content URL (communication control device 5) received in processing S33. This content data includes the time (timestamp) from the start to the end of the recording. As a result, the communication control device 5 receives the content data from the communication unit 51.

[0166] S54: In the communication control device 5, the storage / reading unit 59 stores content data along with a timestamp at a predetermined content URL. Furthermore, the storage / reading unit 59 converts the timestamp managed in the field of view information management DB into the playback elapsed time, matching the total recording time of the content data from which recording was stopped.

[0167] S55: The communication unit 51 sends a recording completion notice to the communication terminal 7. The completion notice includes information indicating a predetermined content URL. As a result, the communication unit 71 of the communication terminal 7 receives the recording completion notice.

[0168] S56: Similarly, the communication unit 51 sends a recording completion notice to the communication terminal 9a. The completion notice includes information indicating a predetermined content URL. As a result, the communication unit 91 of the communication terminal 9a receives the recording completion notice.

[0169] S57: Similarly, the communication unit 51 sends a recording completion notice to the communication terminal 9b. The completion notice includes information indicating a predetermined content URL. As a result, the communication unit 91 of the communication terminal 9b receives the recording completion notice.

[0170] In the case of process S55, it is not necessary to include the specified content URL in the completion notification.

[0171] <Processing of playback of recorded video and audio in communication systems> Next, Figures 20 to 26 will be used to explain the process of playing back recorded video and audio in the communication system. Figure 20 is a sequence diagram showing the process of playing back recorded video and audio in the communication system. Figure 21 is a diagram showing the video data selection screen. Here, participant A plays back the recorded video and audio content data using the communication terminal 9a.

[0172] S71: First, when the reception unit 92 of the communication terminal 9a receives a login request from user A by inputting a user ID and password, the communication unit 91 sends a login request to the communication control device 5. This request includes user A's user ID and password. As a result, the communication control device 5's communication unit 51 receives the login request, and the authentication unit 55 performs authentication by referring to the user / device management DB (see Figure 14). From here on, we will proceed with the assumption that user A has been determined to be a legitimate accessor through login authentication.

[0173] S72: In the communication control device 5, the creation unit 53 creates a recording data selection screen 940 as shown in Figure 21. In this case, the storage / reading unit 59 uses the user ID received in processing S71 as a search key to search the virtual room management DB (see Figure 15) and reads all corresponding virtual room IDs, virtual room names, and content URLs. The creation unit 53 then creates thumbnails 941, 942, and 943 using the images from each content data (with timestamps) stored in the content URL. As a result, the creation unit 53 adds the virtual room name ("Construction Site α", etc.) and the recording time ("2022 / 10 / 31 15:00", etc.) indicating a predetermined time for the timestamp (for example, the recording start time) to each thumbnail.

[0174] S73: The communication unit 51 transmits the selection screen data created in process S72 to the communication terminal 9a. This selection screen data includes a content ID for each thumbnail to identify the wide-field image from which each thumbnail was created. As a result, the communication unit 91 of the communication terminal 9a receives the selection screen data.

[0175] S74: In the communication terminal 9a, the display control unit 94 displays a recording data selection screen on the display 507 of the communication terminal 9a, as shown in Figure 21. Then, the reception unit 92 receives the designation (selection) of a predetermined thumbnail from participant A. Here, we will continue the explanation assuming that thumbnail 941 has been designated (selected).

[0176] S75: The communication unit 71 sends a download request to the communication control device 5 for the content data from which the selected thumbnail 941 was created. This request includes the content ID associated with the thumbnail 941. As a result, the communication unit 51 of the communication control device 5 receives the content data download request.

[0177] S76: In the communication control device 5, the storage / reading unit 59 searches the virtual room management DB (see Figure 15) using the content ID received in processing S75 as a search key, and reads the content data from the corresponding content URL. The content data also includes the location information taken by the imaging device 10, which will be described later. The storage / reading unit 59 also searches the three-dimensional image management DB 5003 using the three-dimensional image ID associated with the content ID received in processing S75 as a search key, and reads the corresponding model ID, location information, and point cloud acquisition date parameters (see Figure 16A), or the corresponding part number, location information, construction date information, etc. (see Figure 16B).

[0178] The communication unit 51 then transmits the requested content data along with the data for the first three-dimensional image and the second three-dimensional image to the communication terminal 9a. As a result, the communication unit 91 of the communication terminal 9a receives the content data, the data for the first three-dimensional image, and the data for the second three-dimensional image.

[0179] S77: The communication terminal 9a performs playback processing. Specifically, the display control unit 94 of the communication terminal 9a displays a screen including the recorded image on the display 507 of the communication terminal 9a, and the sound input / output control unit 95 outputs sound.

[0180] <Details of screen display processing> Next, using Figures 22 to 38, we will explain process S14 among processes S13 to S15. Here, participant A operates the communication terminal 9a, the reception unit 92 receives the operation, and the communication unit 91 transmits operation information indicating the operation content to the communication control device 5. As a result, the communication control device 5 has a communication unit 51, which acts as an acquisition unit, acquire the operation information, and the creation unit 53 creates a screen 600 based on the operation content indicated by the operation information. Then, the communication unit 51 transmits the screen 600 data to the communication terminal 9a, the communication unit 91 of the communication terminal 9a receives the screen 600 data, and the display control unit 94 displays the screen 600 on the display 507 or the like of the communication terminal 9a. In this case, the communication control device 5 is an example of an information processing device.

[0181] Alternatively, the communication unit 91 of the communication terminal 9a may receive the data that is the material for creating the screen 600 from the communication control device 5, and the receiving unit 92, which acts as an acquisition unit, may acquire operation information indicating the content of the operation performed by participant A. In this case, the creation unit 93 of the communication terminal 9a creates the screen 600. In this case, the communication terminal 9a is an example of an information processing device. Communication terminals 7 and 9b can also perform the same processing and roles as the communication terminal 9a.

[0182] Next, we will describe the process by which the communication control device 5 creates a screen 600 to be displayed on the display of the communication terminal 9a 507. Note that processes S13 and S15 are the same as process S14, with only the terminal being displayed being different, so we will omit the explanation of processes S13 and S15. Figures 22 to 24 are flowcharts showing the processes executed by the communication control device in the screen display process.

[0183] First, the creation unit 53 of the communication control device 5 creates the screen 600 that the communication terminal 9a will display, as shown in Figure 25. Figure 25 is a diagram showing an example of the initial display of the screen to be displayed on the communication terminal 9a. The screen 600 includes a display area 610 (an example of a first display area), a display area 620A (an example of a second display area), and a display area 620B (an example of a third display area). In Figure 25, the display areas 610, 620A, and 620B are displayed simultaneously in the same size within the screen 600, but they may be displayed simultaneously in different sizes, or only one of them may be displayed by switching based on the selection of participant A. Alternatively, the display areas 610, 620A, and 620B of the screen 600 may be distributed and displayed on one of several displays.

[0184] The display area 610 displays a predetermined area image, which is a predetermined region in a wide-field image (an example of a captured image) obtained by photographing an object such as a desk, pillar, or window (see Figures 6(a) and (b)).

[0185] In this case, if the captured image is not a curved image like a wide-field image, the display area 610 displays a predetermined area image in which the same area as the capture area of ​​the captured image is the predetermined area.

[0186] Display area 620A includes a three-dimensional model representing the shape of the object in three dimensions based on design data, and displays part or all of the first three-dimensional image to which coordinates (an example of first coordinates) and coordinates (an example of second coordinates) of the wide-field image are associated. Display area 620B includes a three-dimensional model representing the shape of the object in three dimensions based on point cloud data, etc., and displays part or all of the second three-dimensional image to which coordinates (an example of first coordinates) and coordinates (an example of second coordinates) of the wide-field image are associated.

[0187] As explained in Figure 15, the shooting position, which is the position of the shooting device 10, is mapped to an absolute position on Earth, and as explained in Figure 16, all coordinates in the first and second three-dimensional images, including each three-dimensional model and part managed by the three-dimensional image management DB5003, are also mapped to an absolute position on Earth.

[0188] As a result, the processing unit 54 performs a positioning process, for example, in processing S12 (or S12d), it associates the coordinates indicating the shooting position as the first position in the wide-field image of the content data received by the communication control device 5 with the coordinates indicating the second position in the first three-dimensional image, which has the same absolute position as these coordinates. Here, the coordinates are an example of position information.

[0189] As an alternative to the alignment process, the processing unit 54 may, without using absolute positions on Earth or indoors, or to complement absolute positions, associate coordinates in a wide-field image with coordinates indicating a second position in the first three-dimensional image by image processing, such as matching feature points in a wide-field image with those in a first three-dimensional image.

[0190] Then, as described above, the creation unit 53 creates a screen 600 which includes a display area 610 that displays a predetermined region image, which is a predetermined region in a wide-field image, and a second display area and a third display area that display at least a portion of a first three-dimensional image and a second three-dimensional image, respectively, which are associated by the processing unit 54 with a first position (first coordinate) and a second position (second coordinate) in the wide-field image.

[0191] In this case, if the three-dimensional image management DB5003 shown in Figure 16 manages three-dimensional point clouds, mesh objects, or textured mesh objects instead of three-dimensional models, the display area 620B displays at least a portion of a second three-dimensional image, including three-dimensional point clouds, mesh objects, or textured mesh objects, as a three-dimensional area corresponding to the object included in the wide-field image.

[0192] Furthermore, screen 600 includes a time information change field 630 that accepts operations to change the time information of the first three-dimensional image displayed in display area 620A. The time information change field 630 changes the time information of the first three-dimensional image displayed in display area 620A by accepting operations from the user, for example, by moving the display component 632 displayed on the time bar 631 left or right. In the time bar 631 shown in Figure 25, the leftmost bar shows the oldest (most past) time information, the center shows the current time information, and the rightmost bar shows the newest (most future) time information.

[0193] The user can display the first three-dimensional image at the "present" time in the display area 620A by aligning the display component 632 to the "present" position on the time bar 631. The user can display the first three-dimensional image at the earlier point in time, "1.10", in the display area 620A by aligning the display component 632 to the "1.10" position on the time bar 631. The user can display the first three-dimensional image in the future in the display area 620A by moving the display component 632 to the right of the "present" position on the time bar 631. The dates "12.20" and "1.10" displayed to the left of "present" on the time bar 631 may be displayed corresponding to the "point cloud acquisition date" in Figure 16A and the "implementation date information" in Figure 16B.

[0194] Furthermore, screen 600 includes a progress confirmation button 640 for receiving progress confirmation operations from the user, and a first difference confirmation button 650A and a second difference confirmation button 650B for receiving difference confirmation operations from the user. The progress confirmation button 640, the first difference confirmation button 650A, and the second difference confirmation button 650B may always be displayed on screen 600, or they may be displayed when called up by the user as needed. Details of the progress confirmation operation and difference confirmation operation will be described later.

[0195] Additionally, screen 600 displays a close button 609, which is pressed when closing screen 600.

[0196] Furthermore, the display content of screen 600 will be explained in detail below. The virtual camera IC1 is used to identify a predetermined region related to a predetermined region image to be displayed in the display area 610 (see Figure 8), while the virtual camera IC2 shown below is used to identify the first three-dimensional image and the second three-dimensional image to be displayed in display area 620A and display area 620B, respectively.

[0197] S111: First, the creation unit 53 of the communication control device 5 identifies a predetermined area image to be displayed in the display area 610, as shown in Figure 25, based on the pre-set virtual field of view of the virtual camera IC1, from among the wide-field image of the content data received by the communication unit 51.

[0198] S112: The creation unit 53 aligns the position of the virtual camera IC2 with the shooting position associated with the wide-field image, and aligns the virtual field of view of the virtual camera IC2 (an example of a second field of view) with the virtual field of view of the virtual camera IC1 (an example of a first field of view) that has been set in advance, thereby creating the first three-dimensional image and the second three-dimensional image to be displayed in the display area 620A and the display area 620B, respectively.

[0199] S113: The creation unit 53 determines whether the field of view of the virtual camera IC1 in the display area 610 has been changed (see Figure 6(c)). For example, when user A performs an operation to change a predetermined area image within the display area 610 as shown in Figure 25, the reception unit 92 receives the changed field of view, and the communication unit 91 transmits field of view information indicating the changed field of view to the communication control device 5. As a result, when the communication unit 51 receives the field of view information indicating the changed field of view, the creation unit 53 determines that the field of view of the virtual camera IC1 has been changed. If the field of view of the virtual camera IC1 has been changed (S113; YES), the process returns to S111.

[0200] S114: On the other hand, if the field of view of the virtual camera IC1 is not changed (S113; NO), the creation unit 53 determines whether the position (viewpoint) of the virtual camera IC2 has been changed. For example, when the first three-dimensional image in the display area 620A shown in Figure 25 is changed by user A, the reception unit 92 receives the changed position of the virtual camera IC2, and the communication unit 91 transmits position information indicating the changed position to the communication control device 5. As a result, when the communication unit 51 receives position information indicating the changed position, the creation unit 53 determines that the position of the virtual camera IC2 has been changed.

[0201] Here, we will explain how to change the position of the virtual camera IC2 using Figures 26 and 27. Figure 26 shows the state after changing the position and field of view of the virtual camera IC2 from the state of the first three-dimensional image shown in Figure 25. Figure 27 shows the first three-dimensional image displayed for each different position of the virtual camera IC2.

[0202] As shown in Figure 26, when participant A operates cursor c1 on communication terminal 9a, creation unit 53 creates screen 600 as shown in Figure 26, and as a result, display control unit 94 displays screen 600 as shown in Figure 26. Figure 26 shows the state of Figure 27(a). That is, in Figure 27(a), when the virtual camera IC2 is facing south, the first three-dimensional image shown in the lower part of Figure 27(a) is displayed in display area 620A. Also, as shown in Figures 27(b), (c), and (d), when the position of the virtual camera IC2 moves to face east, north, and west, respectively, the first three-dimensional images shown in the lower part of Figures 27(b), (c), and (d) are displayed in display area 620A. Furthermore, according to the changed position and image of the virtual camera IC2, a second three-dimensional image is displayed in display area 620B. In this way, participant A can change the first and second three-dimensional images displayed in display areas 620A and 620B, respectively, by manipulating the display area 620A using cursor c1. This allows participant A to view three-dimensional models that were not displayed in Figure 27(a) and therefore could not be viewed (for example, multiple stacked boxes). In addition, the back side of three-dimensional models that were not displayed in Figure 27(a) and therefore could not be viewed by participant A (for example, a column) becomes viewable.

[0203] S115: Returning to Figure 22, if the position of the virtual camera IC2 is not changed (S114; NO), the creation unit 53 determines whether the field of view of the virtual camera IC2 has been changed, as shown in Figure 23. For example, when participant A changes the first three-dimensional image in the display area 620A shown in Figure 25, the reception unit 92 receives the changed field of view, and the communication unit 91 transmits field of view information indicating the changed field of view to the communication control device 5. As a result, when the communication unit 51 receives field of view information indicating the changed field of view, the creation unit 53 determines that the field of view of the virtual camera IC2 has been changed.

[0204] S116: If the field of view of the virtual camera IC2 is changed (S115; YES), the creation unit 53 enlarges the first three-dimensional image in display area 620A and the second three-dimensional image in display area 620B as shown in Figure 28, or reduces the first three-dimensional image and the second three-dimensional image as shown in Figure 29, according to the change in the field of view of the virtual camera IC2.

[0205] S117: Simultaneously with or immediately after processing S116, the creation unit 53 changes the field of view of the virtual camera IC1 in accordance with (and in conjunction with) the change in the field of view of the virtual camera IC2, thereby changing the predetermined area image to be displayed in the display area 610.

[0206] Figure 28 shows a screen displaying the state in which the predetermined region image has been enlarged in accordance with the enlargement of the first three-dimensional image. Figure 29 shows a screen displaying the state in which the predetermined region image has been reduced in accordance with the reduction of the first three-dimensional image. As an example of changing the predetermined region image, the creation unit 53 enlarges the predetermined region image displayed in the display area 610 as shown in Figure 28, or reduces the predetermined region image displayed in the display area 610 as shown in Figure 29.

[0207] S118: On the other hand, in the process S114 shown in Figure 22, if the position (viewpoint) of the virtual camera IC2 is changed (YES), the creation unit 53 changes the first three-dimensional image and the second three-dimensional image according to the change in the position of the virtual camera IC2. Here, since the first three-dimensional image displayed in the display area 620A and the second three-dimensional image displayed in the display area 620B are images in a virtual space, the position of the virtual camera IC2 can be freely changed (S114; YES), but the shooting position of the real-world shooting device 10 cannot necessarily be freely changed.

[0208] In this embodiment, the shooting position of the shooting device 10 is fixed or manually movable, so the shooting position of the shooting device 10 cannot be automatically changed in response to changes in the position of the virtual camera IC2. However, in another form, the shooting position of the shooting device 10 may be automatically changed in response to changes in the position of the virtual camera IC2 by providing a mechanism for automatically changing the shooting position of the shooting device 10.

[0209] S119: After processing S118, as shown in Figure 23, the creation unit 53 determines whether at least a portion of the predetermined region relating to the predetermined region image being displayed is included in the display area 610 within the first three-dimensional image.

[0210] S120: If, within the first three-dimensional image, the display area 610 contains at least a portion of the predetermined area relating to the predetermined area image being displayed (S119; YES), the creation unit 53 superimposes a corresponding predetermined area 621 corresponding to the field of view of the virtual camera IC1 within the display area 620A, as shown in Figure 30. Figure 30 is a diagram showing a screen displaying the corresponding predetermined area, the icon of the virtual camera IC1, and the line of sight of the virtual camera IC1 on the display area 620A. This corresponding predetermined area 621 is an area corresponding to the predetermined area relating to the predetermined area image displayed in the display area 610. In Figure 30, the corresponding predetermined area 621 is represented by a dashed line, which is an example of how the corresponding predetermined area 621 is displayed. This display method includes dashed and solid lines of a predetermined color, and the thickness of the dashed and solid lines may differ from that of other parts.

[0211] Furthermore, the creation unit 53 superimposes the icon 622 of the virtual camera IC1 within the display area 620A at a position corresponding to the shooting position of the shooting device 10, and superimposes the icon 622 so that it faces the center point CP1 of the corresponding predetermined area 621. Note that the icon 622 is an example of a schematic diagram (image) of the virtual camera IC1. In addition to the icon, this schematic diagram may also include words such as "camera" or diagrams containing these words. The icon 622 may also simply be a circle or the like indicating the shooting position of the shooting device 10, and may not be placed precisely at a position corresponding to the shooting position of the shooting device 10, but rather near the area corresponding to the field of view of the virtual camera IC1. Here, as a modification of the screen shown in Figure 30, in the initial display, the screen 600 does not include the display area 610, but only displays area 620A and 620B, and the reception unit 91 may receive user input on the icon 622 to include the display area 610 in the screen 600. In this case, display areas 620A and 620B do not necessarily have to be included in screen 600, and may be included in a separate screen.

[0212] Furthermore, the creation unit 53 superimposes a line diagram 623 within the display area 620A that shows the line of sight from icon 622 to center point CP1. This line diagram 623 may be a solid line or a dashed line, and may be displayed with a different thickness or color from the other lines.

[0213] In Figure 30, the operation of receiving a time information change operation via the time information change field 630 within the display area 620A and superimposing the corresponding predetermined area 621, icon 622, and diagram 623 is explained. However, the system is not limited to this, and the operation of receiving a time information change operation via the time information change field 630 within the display area 620B and superimposing the corresponding predetermined area 621, icon 622, and diagram 623 may also be implemented.

[0214] S121: If the field of view of the virtual camera IC2 has not been changed in process S115 (NO), if, after process S117, in process S119, at least a portion of the predetermined region relating to the predetermined region image displayed in the display area 610 is not included in the first three-dimensional image (NO), or after process S120, the creation unit 53 determines whether the communication unit 51 has received an instruction to change the time information of the first three-dimensional image displayed in the display area 620A.

[0215] S122: If the communication unit 51 receives an instruction to change the time information of the first three-dimensional image displayed in the display area 620A (S121; YES), the creation unit 53 refers to the three-dimensional image management DB 5003 shown in Figure 16A to determine whether or not point cloud data exists. The point cloud acquisition date in Figure 16A is information indicating the date on which the point cloud data was acquired. Therefore, the creation unit 53 determines that point cloud data exists if the date entered for point cloud acquisition is earlier than the time information to be changed.

[0216] S123: If the communication unit 51 determines in processing S122 that point cloud data exists (S122; YES), as shown in Figure 24, the creation unit 53 displays a first three-dimensional image based on design data in the display area 620A and a second three-dimensional image based on point cloud data, etc., in the display area 620B, corresponding to the modified time information changed in processing S121.

[0217] For example, by aligning the display component 632 to the position "1.10" on the time bar 631, as shown in Figure 31, the first and second three-dimensional images from a previous point in time, "1.10," are displayed in display area 620A and display area 620B, respectively. Figure 31 shows a screen displaying the state in which the first and second three-dimensional images have changed due to a change in time information. In Figure 31, the size of the three-dimensional model m1A included in the first three-dimensional image in display area 620A, and the size of the three-dimensional model m1B included in the second three-dimensional image in display area 620B, have changed relative to the sizes of the three-dimensional model m1A included in the first three-dimensional image in display area 620A, and the size of the three-dimensional model m1B included in the second three-dimensional image in display area 620B, respectively, as shown in Figure 26.

[0218] Furthermore, by aligning the display component 632 to the position "12.20" on the time bar 631, as shown in Figure 32, the first and second three-dimensional images at the current time and the time before "1.10" ("12.20") are displayed in display area 620A and display area 620B, respectively. Figure 32 shows a screen displaying the state in which the first and second three-dimensional images have changed due to the change in time information. In Figure 32, the three-dimensional model m1A included in the first three-dimensional image in display area 620A and the three-dimensional model m1B included in the second three-dimensional image in display area 620B have changed to not being placed.

[0219] In this way, the creation unit 53 can create display areas 620A and 620B that display at least a portion of the first three-dimensional image and the second three-dimensional image, respectively, corresponding to the time information of the time the captured image with a predetermined area image displayed in the display area 610 was taken (present) and the time before the time it was taken (past).

[0220] S124: If the communication unit 51 does not determine that point cloud data exists in processing S122 (S122; NO), as shown in Figure 24, the creation unit 53 displays a first three-dimensional image based on design data in the display area 620A, corresponding to the modified time information changed in processing S121, and does not display a second three-dimensional image based on point cloud data in the display area 620B.

[0221] For example, by aligning the display component 632 to the "1M later (one month later)" position on the time bar 631, as shown in Figure 33, the first three-dimensional image of one month later is displayed in the display area 620A. Figure 33 is a diagram showing the screen displaying the state in which the first and second three-dimensional images have changed due to the change in time information. In Figure 33, the three-dimensional model m2, which was not present in the display area 620A in Figure 26, has been added to the first three-dimensional image in the display area 620A, and the second three-dimensional image is hidden in the display area 620B.

[0222] The creation unit 53 refers to the construction date information in Figure 16B and displays only the components for which the construction date has arrived one month later, while not displaying or partially displaying the components for which the construction date has not yet arrived. This allows the display area 620A to display the first three-dimensional image as it appears one month from the present.

[0223] In this way, the creation unit 53 can create a display area 620A that displays at least a portion of the first three-dimensional image, corresponding to time information of a time later (future) than the time the captured image with the predetermined area image displayed in the display area 610 was taken.

[0224] S125: After processing S123 or S124, the creation unit 53 determines whether the communication unit 51 has received an instruction to check progress. For example, the creation unit 53 determines that it has received an instruction to check progress when it receives an operation from the user to press the progress check button 640.

[0225] S126: If it is determined that an instruction to check the progress has been received in process S125 (S125; YES), the creation unit 53 overlays the difference between the first three-dimensional image before the change in time information and the current image as progress identification information 641 onto the first three-dimensional image after the change in time information, as shown in Figure 24. Figure 34 shows the screen with the progress identification information overlaid. In Figure 34, progress identification information 641, such as shading, is overlaid onto the three-dimensional model m2, which was not present in the display area 620A of Figure 26, making it easier to check the progress from the present (before the change in time information) to one month later (after the change in time information).

[0226] In this way, the creation unit 53 can create a display area 620A by superimposing progress identification information 641, which identifies differences from the first three-dimensional image corresponding to the first time information (e.g., the present), onto the first three-dimensional image corresponding to the second time information (e.g., one month later). Note that the first time information and the second time information may be the past and the present, or both may be the past.

[0227] S127: If it is determined in process S125 that no instruction for progress confirmation has been received (S125; NO), or after process S126, the creation unit 53 determines whether the communication unit 51 has received an instruction for the first difference confirmation. For example, the creation unit 53 determines that it has received an instruction for the first difference confirmation when it receives an operation from the user to press the first difference confirmation button 650A.

[0228] S128: If it is determined that an instruction to check the progress has been received in process S127 (S127; YES), as shown in Figure 24, the creation unit 53 overlays the difference between the first three-dimensional image in display area 620A and the second three-dimensional image in display area 620B as the first difference identification information 651A onto the second three-dimensional image in display area 620B. Figure 35 shows the screen before the first and second difference identification information are overlaid. Figure 36 shows the screen after the first difference identification information is overlaid. Figure 35 shows an example where there is a difference between the three-dimensional model m1A included in the first three-dimensional image in display area 620A, the three-dimensional model m1B included in the second three-dimensional image in display area 620B, and the image M1 of the object included in the predetermined area image of display area 610.

[0229] Figure 36 shows that the first difference identification information 651A, such as diagonal lines, is superimposed on the area in display area 620B that is different from the three-dimensional model m1A included in the first three-dimensional image in display area 620A, making it easier to confirm the difference between the first three-dimensional image in display area 620A and the second three-dimensional image in display area 620B.

[0230] S129: If it is determined in process S127 that no instruction for progress confirmation has been received (S127; NO), or after process S128, the creation unit 53 determines whether the communication unit 51 has received an instruction for a second difference confirmation. For example, the creation unit 53 determines that it has received an instruction for a second difference confirmation when it receives an operation from the user to press the second difference confirmation button 650B.

[0231] S130: If it is determined that an instruction for a second difference confirmation has been received in processing S129 (S129; YES), as shown in Figure 24, the creation unit 53 overlays the difference between the predetermined area image of the display area 610 and the first three-dimensional image of the display area 620A as the second difference identification information 651B onto the predetermined area image of the display area 610. Figure 37 shows the screen after the first difference identification information and the second difference identification information have been overlaid.

[0232] Figure 37 shows that second difference identification information 651B, such as diagonal lines, is superimposed on the parts of display area 610 that differ from the three-dimensional model m1A included in the first three-dimensional image of display area 620A, making it easier to confirm the difference between the predetermined area image of display area 610 and the first three-dimensional image of display area 620A.

[0233] For areas in display area 610 that differ from the three-dimensional model m1A included in the first three-dimensional image of display area 620A, the first three-dimensional image may be converted into a two-dimensional image with the same field of view as the predetermined area image, and the difference may be recognized by image recognition AI. Alternatively, for areas in display area 610 that differ from the three-dimensional model m1A included in the first three-dimensional image of display area 620A, the first three-dimensional image and the predetermined area image may be aligned pixel by pixel, and the difference may be recognized while the three-dimensional data remains intact. The same applies to areas in display area 620B that differ from the three-dimensional model m1A included in the first three-dimensional image of display area 620A.

[0234] Note that the screen in Figure 36 is an example, and the difference between the first three-dimensional image in display area 620A and the second three-dimensional image in display area 620B may be superimposed on the first three-dimensional image in display area 620A as first difference identification information. Also, the screen in Figure 37 is an example, and the difference between the predetermined area image in display area 610 and the first three-dimensional image in display area 620A may be superimposed on the first three-dimensional image in display area 620A as second difference identification information.

[0235] Thus, the creation unit 53 can create a third display area by superimposing first difference identification information, which identifies the differences between the first three-dimensional image and the second three-dimensional image, onto the second three-dimensional image, or create a second display area by superimposing it onto the first three-dimensional image. Furthermore, the creation unit 53 can create a second display area by superimposing second difference identification information, which identifies the differences between the first three-dimensional image and the captured image, onto the first three-dimensional image, or create a first display area by superimposing it onto the captured image.

[0236] If it is determined in process S129 that an instruction for checking the second difference has not been received (S129; NO), if an instruction to change the time information of the first three-dimensional image displayed in the display area 620A has not been received (S121; NO), or after process S130, the screen display process shown in Figure 24 is terminated.

[0237] Figure 38 shows a screen where the three-dimensional model changes differently even though the change in time information is the same. Figure 38(B) shows that the construction date information in the three-dimensional image management table shown in Figure 16B changes from "2024.5.1" to "2024.5.10".

[0238] If the construction date information is "2024.5.1" and one month from now is "2024.5.1", then as shown in Figure 33, the 3D model m2 with "Part No." "5" will be displayed as being included in the first 3D image of display area 620A. On the other hand, if the construction date information is "2024.5.10" and one month from now is "2024.5.1", then as shown in Figure 38(A), the 3D model m2 with "Part No." "5" will not be displayed in the first 3D image of display area 620A.

[0239] Furthermore, if the three-dimensional image management DB5003 shown in Figure 16B manages construction period information indicating the period during which a part is installed, then in Figure 38(A), a portion of the three-dimensional model m2 with "Part No." "5" may be displayed in the first three-dimensional image of the display area 620A.

[0240] Specifically, if the construction period information for the three-dimensional model m2 with "Part No." "5" is "20 days" and the construction date information is "2024.5.10", then as of "2024.5.1", one month from now, it can be calculated that the three-dimensional model m2 is approximately 50% complete based on daily calculations. Therefore, half of the three-dimensional model m2 may be displayed in the first three-dimensional image of display area 620A.

[0241] Thus, when the construction date information in the three-dimensional image management table shown in Figure 16B is changed, the creation unit 53 can create a display area 620A that changes at least one of the changes in the arrangement and size of the three-dimensional model corresponding to the time information, even if the change in time information is the same (for example, one month later).

[0242] [Main effects of the embodiment] As described above, according to this embodiment, the processing unit 54 associates position information indicating a first position in the captured image obtained by photographing the object with position information indicating a second position in the three-dimensional image including the three-dimensional region corresponding to the object, and the creation unit 53 (or creation units 73, 93) creates a screen 600 that includes a display area 610 that displays a predetermined region image which is a predetermined region in the captured image obtained by photographing the object, a display area 620A that displays at least a part of the first three-dimensional image which is a three-dimensional image to which the first position and the second position in the captured image are associated by the processing unit 54 and is generated based on design data, and a display area 620B that displays at least a part of the second three-dimensional image which is generated based on point cloud data.

[0243] This allows users to understand what and where a designated area image was captured by viewing screen 600. Furthermore, by viewing screen 600, users can determine if the designated area image is appropriate by comparing it with the first and second three-dimensional images. Even if the back of an object cannot be photographed, users can understand its condition by viewing screen 600. Therefore, creating images that complement the captured images improves user convenience.

[0244] [supplement] Although embodiments have been described above, the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0245] (1) Each function of each embodiment described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, and devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each function described above.

[0246] (2) A (non-temporary) recording medium such as a DVD-ROM on which any of the above programs are stored may be provided domestically or internationally as a program product.

[0247] (3) There may be multiple CPUs 111, 301, and 501, each serving as a processor.

[0248] The embodiments of the present invention are as follows. <1> A processing unit that associates position information indicating a first position in a captured image obtained by photographing an object with position information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object, A creation unit that creates a screen including: a first display area that displays a predetermined region image which is a predetermined region in the captured image; a second display area that displays at least a part of the first three-dimensional image which is a three-dimensional image in which the first position and the second position in the captured image are associated by the processing unit and is generated based on design information; and a third display area that displays at least a part of the second three-dimensional image which is generated based on point cloud information. It is an information processing device. <2> >The creation unit creates a second display area by superimposing first difference identification information, which indicates the differences between the first three-dimensional image and the second three-dimensional image, onto the first three-dimensional image, or creates a third display area by superimposing the first difference identification information onto the second three-dimensional image. <1> This is the information processing device described above. <3> The creation unit creates a second display area by superimposing second difference identification information, which indicates the differences between the first three-dimensional image and the captured image, onto the first three-dimensional image, or creates a first display area by superimposing the second difference identification information onto the captured image. <1> or <2> This is the information processing device described above. <4> The creation unit creates the second display area or the third display area so as to superimpose the captured image or a position image indicating the shooting position onto the first three-dimensional image or the second three-dimensional image at a position corresponding to the shooting position of the captured image. <1> ~ <3> It is an information processing device described in any one of the items. <5> The creation unit creates the screen which includes the second display area and the third display area but does not include the first display area. When an operation on the position image is received, the screen including the first display area is created <4> This is the information processing device described. <6> The creation unit creates the second display area such that, in accordance with time information of a time before or after the time the captured image was taken, at least one of the arrangement and size of the three-dimensional model included in the first three-dimensional image changes. <1> ~ <5> It is an information processing device described in any one of the items. <7> The creation unit creates the second display area including the time information. <6> This is the information processing device described. <8> A method for creating a screen that is executed by an information processing device, A mapping process that associates positional information indicating a first position in a captured image obtained by photographing an object with positional information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object, A creation process for creating a screen that includes: a first display area for displaying a predetermined region image which is a predetermined region in the captured image; a second display area for displaying at least a portion of the first three-dimensional image which is a three-dimensional image in which the first position and the second position in the captured image are associated by the association process and which is generated based on design information; and a third display area for displaying at least a portion of the second three-dimensional image which is generated based on point cloud information. This is a screen creation method that includes [a specific feature / feature]. <9> On the computer, A mapping process that associates positional information indicating a first position in a captured image obtained by photographing an object with positional information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object, A creation process for creating a screen that includes: a first display area for displaying a predetermined region image which is a predetermined region in the captured image; a second display area for displaying at least a portion of the first three-dimensional image which is a three-dimensional image in which the first position and the second position in the captured image are associated by the association process and which is generated based on design information; and a third display area for displaying at least a portion of the second three-dimensional image which is generated based on point cloud information. This is a program to execute [the command / action]. <10> An information processing system including an information processing device and a display terminal, A processing unit that associates position information indicating a first position in a captured image obtained by photographing an object with position information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object, A creation unit that creates a screen including: a first display area that displays a predetermined region image which is a predetermined region in the captured image; a second display area that displays at least a part of the first three-dimensional image which is a three-dimensional image in which the first position and the second position in the captured image are associated by the processing unit and is generated based on design information; and a third display area that displays at least a part of the second three-dimensional image which is generated based on point cloud information. It is an information processing system that has [this feature]. [Explanation of Symbols]

[0249] 1 Communication system (an example of an information processing system) 3 Relay device 5 Communication control device (an example of an information processing device) 7 Communication terminal (an example of a display terminal, an example of an information processing device) 9, 9a, 9b Communication terminal (an example of a display terminal, an example of an information processing device) 10 Imaging device 51 Communication unit (an example of an acquisition unit) 52 Reception unit 53 Creation unit 71 Communication unit 72 Reception unit (an example of an acquisition unit) 73 Creation unit 74 Display control unit 75 Audio input / output control unit 78 Connection unit 91 Communication unit 92 Reception unit (an example of an acquisition unit) 93 Creation unit 94 Display control unit 95 Audio input / output control unit 98 Connection unit 507 Display (an example of a display unit) 600 Screen 610 Display area (an example of a first display area) 620A Display area (an example of a second display area) 620B Display area (an example of a third display area) 640 Progress confirmation button 641 Progress identification information 650A First difference confirmation button 650B Second difference confirmation button 651A First difference identification information 651B Second difference identification information 5001 User device management DB 5002 Virtual room management DB 5003 3D image management DB

Prior art documents

Patent documents

[0250] [Patent Document 1] Japanese Patent Publication No. 2023-140923

Claims

1. A processing unit that associates position information indicating a first position in a captured image obtained by photographing an object with position information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object, A creation unit that creates a screen including: a first display area for displaying a predetermined region image which is a predetermined region in the captured image; a second display area for displaying at least a portion of the first three-dimensional image which is a three-dimensional image in which the first position and the second position in the captured image are associated by the processing unit and is generated based on design information; and a third display area for displaying at least a portion of the second three-dimensional image which is generated based on point cloud information. An information processing device having

2. The information processing apparatus according to claim 1, wherein the creation unit creates a second display area by superimposing first difference identification information indicating a difference between the first three-dimensional image and the second three-dimensional image onto the first three-dimensional image, or creates a third display area by superimposing the first difference identification information onto the second three-dimensional image.

3. The information processing apparatus according to claim 1, wherein the creation unit superimposes second difference identification information indicating differences between the first three-dimensional image and the captured image onto the first three-dimensional image to create the second display area, or superimposes the second difference identification information onto the captured image to create the first display area.

4. The information processing apparatus according to claim 1, wherein the creation unit creates the second display area or the third display area so as to superimpose the captured image or a position image indicating the shooting position onto the first three-dimensional image or the second three-dimensional image at a position corresponding to the shooting position of the captured image.

5. The creation unit creates the screen which includes the second display area and the third display area but does not include the first display area. The information processing apparatus according to claim 4, which creates the screen including the first display area when an operation on the position image is received.

6. The information processing apparatus according to claim 1, wherein the creation unit creates the second display area such that at least one of the arrangement and size of the three-dimensional model included in the first three-dimensional image changes in response to time information of a time before or after the time the captured image was taken.

7. The information processing apparatus according to claim 6, wherein the creation unit creates the second display area including the time information.

8. A method for creating a screen that is executed by an information processing device, A mapping process that associates positional information indicating a first position in a captured image obtained by photographing an object with positional information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object, A creation process for creating a screen that includes: a first display area for displaying a predetermined region image which is a predetermined region in the captured image; a second display area for displaying at least a portion of the first three-dimensional image which is a three-dimensional image in which the first position and the second position in the captured image are associated by the association process and which is generated based on design information; and a third display area for displaying at least a portion of the second three-dimensional image which is generated based on point cloud information. A screen creation method having the following characteristics.

9. On the computer, A mapping process that associates positional information indicating a first position in a captured image obtained by photographing an object with positional information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object, A creation process for creating a screen that includes: a first display area for displaying a predetermined region image which is a predetermined region in the captured image; a second display area for displaying at least a portion of the first three-dimensional image which is a three-dimensional image in which the first position and the second position in the captured image are associated by the association process and which is generated based on design information; and a third display area for displaying at least a portion of the second three-dimensional image which is generated based on point cloud information. A program to execute.

10. An information processing system including an information processing device and a display terminal, A processing unit that associates position information indicating a first position in a captured image obtained by photographing an object with position information indicating a second position in a three-dimensional image including a three-dimensional region corresponding to the object, A creation unit that creates a screen including: a first display area for displaying a predetermined region image which is a predetermined region in the captured image; a second display area for displaying at least a portion of the first three-dimensional image which is a three-dimensional image in which the first position and the second position in the captured image are associated by the processing unit and is generated based on design information; and a third display area for displaying at least a portion of the second three-dimensional image which is generated based on point cloud information. An information processing system having the following features.

Images