Information processing device, screen creation method, program, and information processing system

Abstract

The aim is to improve user convenience by supplementing the images obtained through photography. [Solution] The present disclosure is an information processing device that can communicate with a shooting device, and includes a creation unit that creates a screen (601) which includes a shooting image display area (610) that displays a predetermined region image which is a predetermined region in a shooting image obtained by shooting an object with the shooting device, and a three-dimensional image display area (620) that displays at least a part of a three-dimensional image aligned with the shooting image, and a processing unit that performs processing to associate each text information from a plurality of text information generated from audio data including sound collected at different times by a sound collection unit in the shooting device or in a display terminal that can communicate with the shooting device with each predetermined region image from a plurality of predetermined region images obtained at different times, with respect to the shooting position of the shooting image.

Description

Technical Field

[0004] ,

[0001] The present disclosure relates to an information processing apparatus, a screen creation method, a program, and an information processing system.

Background Art

[0002] In recent years, as an imaging range including locations that cannot be fully confirmed within a normal viewing angle, for example, a wide-view image (hereinafter referred to as a "wide-view image") having a wide viewing angle such as a 360-degree image (also referred to as an omnidirectional image, panoramic image, or full-surround image) in which the entire 360-degree circumference is imaged is known. When attempting to display all of such a wide-view image on a display terminal, since the wide-view image appears curved and is difficult to view, a predetermined area image indicating a predetermined area in the wide-view image is displayed on the display terminal, and the user browses the predetermined area image (see Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, it was difficult for the user to grasp what and where the predetermined area image was taken from just by browsing the predetermined area image within the displayed range.

[0004] In addition, the imaging device captures objects around the imaging device in a single shot from the imaging position, but it cannot capture objects from 360 degrees around the object. Therefore, even for an object within the imaging range, the back side or the like of the object cannot be imaged, and the user cannot grasp the state of the back side or the like of the object.

[0005] The above problems were the same even when the captured image was not an image that curved like a wide-view image.

[0006] The present disclosure has been made in view of the above circumstances, and an object thereof is to improve the convenience for the user by complementing the captured image obtained by imaging.

Means for Solving the Problems

[0007] The disclosure relating to claim 1 is an information processing device that can communicate with a shooting device, comprising: a creation unit that creates a screen including a shooting image display area that displays a predetermined region image which is a predetermined region in a shooting image obtained by shooting an object with the shooting device, and a three-dimensional image display area that displays at least a part of a three-dimensional image aligned with the shooting image; and a processing unit that performs processing to associate each text information of a plurality of text information generated from audio data including sound collected at different times by a sound collection unit of the information processing device or a display terminal that can communicate with the information processing device with each predetermined region image of a plurality of predetermined region images obtained at different times, with respect to the shooting position of the shooting image. [Effects of the Invention]

[0008] As explained above, this disclosure has the effect of improving user convenience by creating images that complement the captured images obtained through photography. [Brief explanation of the drawing]

[0009] [Figure 1] (a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a top view of the imaging device. [Figure 2] This is a diagram illustrating the use of the imaging device. [Figure 3] (a) is a hemispherical image (front) taken by the imaging device, (b) is a hemispherical image (back) taken by the imaging device, and (c) is an image represented by equirectangular projection. [Figure 4] (a) is a conceptual diagram showing the state of covering the sphere with an equirectangular projection image, and (b) is a diagram showing a full-sphere image. [Figure 5] This diagram shows the positions of a virtual camera and a predetermined region when a 360-degree spherical image is treated as a three-dimensional sphere. [Figure 6](a) is a stereoscopic perspective view of Figure 5, (b) is a diagram showing the predetermined region image in the state of (a) on the display, (c) is a diagram showing the predetermined region after changing the viewpoint of the virtual camera IC in (a), and (d) is a diagram showing the predetermined region image in the state of (c) on the display. [Figure 7] This is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. [Figure 8] This is a conceptual diagram showing the relationship between a designated area and a point of interest. [Figure 9] This is a schematic diagram of the communication system according to the embodiment. [Figure 10] This is a hardware configuration diagram of the imaging device. [Figure 11] This is a hardware configuration diagram of the relay device. [Figure 12] This is a hardware configuration diagram of a communication control device and a communication terminal. [Figure 13] This is a diagram illustrating the functional configuration of the communication system according to the embodiment. [Figure 14] This is a conceptual diagram of the user device management table. [Figure 15] This is a conceptual diagram of the virtual room management table. [Figure 16] This is a conceptual diagram of a three-dimensional image management table (part 1). [Figure 17] This is a conceptual diagram of the three-dimensional image management table (part 2). [Figure 18A] This is a conceptual diagram of a movement history management table created in the past. [Figure 18B] This is a conceptual diagram of a previously created movement history management table. [Figure 18C] This is a conceptual diagram of the movement history management table currently under development. [Figure 19] This is a sequence diagram illustrating the communication process of content data in a communication system. [Figure 20] This is a sequence diagram showing the process of initiating video recording and audio recording in a communication system. [Figure 21] This is a sequence diagram showing the process of stopping video recording and audio recording in a communication system. [Figure 22] It is a sequence diagram showing the processes of recording and playing back recordings in a communication system. [Figure 23] It is a diagram showing a recording data selection screen. [Figure 24] It is a flowchart showing the processes executed by the communication control device in screen display processing. [Figure 25] It is a flowchart showing the processes executed by the communication control device in screen display processing. [Figure 26] It is a diagram showing an initial display example of the screen to be displayed on the communication terminal 9a. [Figure 27] It is a diagram showing a screen on which a diagram indicating a corresponding predetermined area, an icon of the virtual camera IC1, and a line indicating the line of sight of the virtual camera IC1 is displayed for the display area 620. [Figure 28] It is a diagram showing a state in which a change has occurred in the object to be photographed at the same shooting position and angle of view in the communication terminal 9a. [Figure 29] It is a diagram showing another state in which an icon of the virtual camera IC1, a corresponding display area, and a diagram indicating the line of sight are superimposed in the communication terminal 9a. [Figure 30] It is a diagram showing a state in which the movement path of the icon of each virtual camera IC1 indicating the position and angle of view of the photographing device at the time of registration of the photographing date and time, and a diagram indicating the latest corresponding display area and line of sight are superimposed. [Figure 31] It is a diagram showing a screen when playing back a plurality of recorded predetermined area images in the communication terminal 9a. [Figure 32] It is a flowchart showing the processes executed by the communication control device in screen display processing after registration of the photographing date and time. [Figure 33] It is a flowchart showing the processes executed by the communication control device in screen display processing after registration of the photographing date and time. [Figure 34] It is a diagram showing a screen when displaying the current predetermined area image after registration of the photographing date and time in the communication terminal 9a. [Figure 35] It is a diagram showing a screen when additionally displaying a past predetermined area image after registration of the photographing date and time in the communication terminal 9a. [Figure 36] This figure shows a modified screen of the communication terminal 9a when past images of a predetermined area are displayed after the date and time of shooting have been registered. [Figure 37] This diagram shows the screen displayed when the audio, text, and image minutes of a meeting are shown on the communication terminal 9a. [Figure 38] This figure shows the screen displayed on the communication terminal 9a when past images of a predetermined area and three-dimensional images are added along with the meeting minutes. [Figure 39] This figure shows a modified version of the functional configuration diagram explained in Figure 13. [Figure 40] This figure shows the screen displayed on the communication terminal 9a when an image is generated and added to the meeting minutes based on a predetermined area image, a three-dimensional image, and audio comments. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings.

[0011] [Overview of 360° spherical images] Figures 1 to 8 illustrate the method for generating a 360° spherical image. A 360° spherical image, also known as a 360° panoramic image, is an example of a wide-field video with a broad field of view. Wide-field images also include simple panoramic images with a field of view of approximately 180°.

[0012] First, the external appearance of the imaging device 10 will be described using Figure 1. The imaging device 10 is a digital camera used to obtain the image that will be the basis for the 360-degree spherical image. Figure 1(a) is a left side view of the imaging device, Figure 1(b) is a front view of the imaging device, and Figure 1(c) is a top view of the imaging device.

[0013] As shown in Figure 1(a), the imaging device 10 is small enough for a person to hold in one hand. Also, as shown in Figures 1(a), (b), and (c), the imaging device 10 has an image sensor 103a on the front side and an image sensor 103b on the rear side. Furthermore, as shown in Figure 1(b), an operating section 115, such as a shutter button, is provided on the front side of the imaging device 10.

[0014] Next, the usage of the imaging device 10 will be explained using Figure 2. Figure 2 is an illustrative diagram of the imaging device in use. As shown in Figure 2, the imaging device 10 is connected to a relay device 3 installed on some kind of stand 2, and is used to photograph surrounding subjects, scenery, etc. In this case, two hemispherical images can be obtained by capturing subjects around the user with the image sensors 103a and 103b shown in Figure 1, respectively. Note that if the imaging device 10 does not transmit the 360-degree spherical image obtained through imaging to other communication terminals or systems, the relay device 3 is unnecessary.

[0015] Next, using Figures 3 and 4, we will outline the process from the image captured by the imaging device 10 to the creation of a full-sphere image. Figure 3(a) shows the hemispherical image (front) captured by the imaging device, Figure 3(b) shows the hemispherical image (rear) captured by the imaging device, and Figure 3(c) shows the image represented by the equirectangular projection (hereinafter referred to as the "equistratic projection image"). Alternatively, an image represented by the Mercator projection (hereinafter referred to as the "Mercator image") may also be used. Figure 4(a) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 4(b) shows the full-sphere image. The "equistratic projection image" is a full-sphere image in equirectangular format, as an example of the wide-field image described above.

[0016] As shown in Figure 3(a), the image obtained by the image sensor 103a becomes a curved hemispherical image (front side) due to the wide-angle lens 102a, such as a fisheye lens, as described later. Similarly, as shown in Figure 3(b), the image obtained by the image sensor 103b becomes a curved hemispherical image (rear side) due to the wide-angle lens 102b, such as a fisheye lens, as described later. The imaging device 10 then combines the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) to create an equirectangular projection image EC as shown in Figure 3(c).

[0017] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to overlay the equirectangular projection image EC so that it covers the sphere, as shown in Figure 4(a), and creates a full-sphere image CE as shown in Figure 4(b). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. OpenGL ES is merely one example of software that performs image processing, and the full-sphere image CE may be created using other software. The full-sphere image CE may be a still image or a video. Here, the imaging device 10 is described as generating a full-sphere image, but other devices (communication control device 5 or communication terminals 7, 9, etc.) may perform similar image processing or some of the image processing steps.

[0018] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image EC is superimposed to cover the sphere, as shown in Figure 4(a), creating a full-sphere image CE as shown in Figure 4(b). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data.

[0019] As described above, the 360-degree spherical image CE is an image pasted over a sphere, which can cause discomfort to the human eye. Therefore, the communication terminals 7 and 9 can display a predetermined region (hereinafter referred to as the "predetermined region image"), which is a part of the 360-degree spherical image, as a flat image with minimal curvature (distortion), thereby providing a display that does not cause discomfort to the human eye. This will be explained using Figures 5 to 8.

[0020] Figure 5 shows the positions of the virtual camera and the predetermined region when the 360-degree spherical image is treated as a three-dimensional sphere. The virtual camera IC1 corresponds to the position of the virtual viewpoint of the user viewing the 360-degree spherical image CE, which is displayed as a three-dimensional sphere. In Figure 6, (a) is a stereoscopic perspective view of Figure 5, (b) is a diagram showing the predetermined region image in the state of (a) displayed on the screen, (c) is a diagram showing the predetermined region after changing the viewpoint of the virtual camera IC1 in (a), and (d) is a diagram showing the predetermined region image in the state of (c) displayed on the screen.

[0021] If the resulting spherical image CE is considered a solid sphere CS, then, as shown in Figure 5, the virtual camera IC1 is located inside the spherical image CE. A predetermined region T in the spherical image CE is the imaging region of the virtual camera IC1 and is identified by field-of-view information (also called "region information") that indicates the imaging direction and field of view of the virtual camera IC1 in the three-dimensional virtual space containing the spherical image CE.

[0022] Furthermore, zooming in on a predetermined region T can also be represented by moving the virtual camera IC1 closer to or further away from the 360-degree image CE. The predetermined region image Q is the image of the predetermined region T in the 360-degree image CE. Therefore, the predetermined region T can be determined by the field of view a and the distance f from the virtual camera IC1 to the 360-degree image CE.

[0023] Furthermore, when the virtual viewpoint of the virtual camera IC1 is moved (also called "changed") to the right (left side in the drawing) from the state shown in Figure 6(a) to the state shown in Figure 6(c), the predetermined region T in the 360-degree image CE is moved to the predetermined region T' accordingly, and the predetermined region image Q displayed on the predetermined display is changed to the predetermined region image Q'. As a result, the image shown in Figure 6(b) is changed to the image shown in Figure 6(d) displayed on the predetermined display.

[0024] Next, we will explain the relationship between the field of view information and the image of a predetermined region T using Figures 7 and 8. Figure 7 is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. Figure 8 is a conceptual diagram showing the relationship between a predetermined region and a point of interest (center point).

[0025] Here, let (r,θ,φ) be the arbitrary position coordinates when the center point CP shown in Figure 7 is expressed in spherical polar coordinates. (r,θ,φ) are the radial vector, polar angle, and azimuth angle, respectively. The radial vector r is the distance from the origin of the three-dimensional virtual space containing the all-sky image CE to any point (center point CP in Figure 8), and is equal to the distance f shown in Figure 8.

[0026] Furthermore, as shown in Figure 8, if we consider the center of the predetermined region T, which is the imaging area of ​​the virtual camera IC1, as the center point CP in Figure 7, then the trigonometric function shown in (Equation 1) below generally holds true. (L / f) = tan(a / 2)···(Equation 1) Here, f is the distance from the virtual camera IC1 to the center point CP. L is the distance between any vertex of the predetermined region T and the center point CP (2L is the diagonal). a is the field of view. In this case, the field of view information for identifying the predetermined region T can be represented by pan(θ), tilt(φ), and fov(a). Note that zooming in on the predetermined region T can be represented by widening or narrowing the range (arc) of the field of view a.

[0027] [Outline of the communication system] Next, the outline of the communication system 1 according to the embodiment will be explained using Figure 9. Figure 9 is a schematic diagram of the communication system according to the embodiment.

[0028] As shown in Figure 9, the communication system 1 of this embodiment consists of a camera 10, a relay device 3, a communication terminal 7, and communication terminals 9a and 9b. The communication terminals 9a and 9b are collectively referred to as "communication terminal 9". The communication control device 5, communication terminal 7, and communication terminal 9 are also examples of information processing devices. Furthermore, communication terminals 7 and 9 may also be referred to as "display terminals" that display images, etc.

[0029] Of these, the imaging device 10 is a digital camera for obtaining wide-field images (such as 360-degree images), as described above. The relay device 3 has the function of a cradle for charging the imaging device 10 and transmitting and receiving data. In addition, the relay device 3 can communicate data with the imaging device 10 via contacts, and can also communicate data with the communication control device 5 via the communication network 100. The communication network 100 includes, for example, the Internet, a LAN (Local Area Network), a (wireless) router, etc.

[0030] Furthermore, the communication control device 5 is, for example, a computer and can communicate data with the relay device 3 and communication terminals 7 and 9 via the communication network 100. The communication control device 5 can also be referred to as an "information management device" because it manages field-of-view information, etc. The communication control device 5 may be composed of a single computer or multiple computers.

[0031] The communication terminals 7 and 9 are computers such as smartphones and notebook PCs (Personal Computers), and can communicate data with the communication control device 5 via the communication network 100. OpenGL ES is installed on the communication terminals 7 and 9, and they create a predetermined region image (see Figure 6) from the 360-degree image received from the communication control device 5.

[0032] Furthermore, the camera 10 and relay device 3 are installed in designated locations by the organizer X at the site (location Sa), such as a construction site, exhibition hall, educational venue, or medical facility. The communication terminal 7 is operated by the organizer X. The communication terminal 9a is operated by participant A, such as a viewer, who is located remotely from the site Sa. Similarly, the communication terminal 9b is operated by participant B, such as a viewer, who is located remotely from the site Sa. Participants A and B may be in the same location or in different locations.

[0033] The communication control device 5 transmits (distributes) wide-field image and audio data obtained from the imaging device 10 via the relay device 3 to the communication terminals 7 and 9. The communication control device 5 also transmits (distributes) captured image and audio data obtained from each of the communication terminals 7 and 9 to the communication terminals 7 and 9. While the captured image sent from the imaging device 10 via the relay device 3 is a wide-field image, if a single-lens reflex camera or the like is used instead of the imaging device 10, the captured image will be a general narrow-field image. The captured image may be a video or a still image.

[0034] [Hardware configuration] Next, the hardware configuration of the imaging device 10, relay device 3, and communication terminals 7 and 9 of this embodiment will be described in detail with reference to Figures 10 to 12.

[0035] <Hardware configuration of the imaging device> Figure 10 is a hardware configuration diagram of the imaging device. As shown in Figure 10, the imaging device 10 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output interface 116, a short-range communication circuit 117, an antenna 117a for the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network interface 121.

[0036] Of these, the imaging unit 101 includes wide-angle lenses 102a and 102b (hereinafter referred to as lenses 102 unless otherwise specified) capable of capturing images with a field of view of 180° or more to form a hemispherical image, and two image sensors 103a and 103b provided in correspondence with each of the lenses 102a and 102b.

[0037] Furthermore, the image sensors 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the lens 102a, 102b, etc., into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers for which various commands or parameters necessary for the operation of the image sensor are set. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example; it may have only one, or three or more.

[0038] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus).

[0039] The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. In addition, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, the control unit 115, the input / output interface 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network interface 121, among others.

[0040] The image processing unit 104 receives image data output from image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create data for an equirectangular projection image (an example of a wide-field image), which will be described later.

[0041] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 as the master device and the image sensors 103a and 103b as slave devices. The necessary commands and other information are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0042] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 10, there may also be functions to display a preview or video on a display (for example, the display of an external terminal such as a smartphone that communicates with the imaging device 10 via a short-range communication circuit 117). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0043] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0044] The CPU 111 controls the overall operation of the imaging device 10 and performs necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress of processing by the image processing unit 104 and data of completed equirectangular projection images.

[0045] The control unit 115 is a collective term for various operation buttons, power switches, shutter buttons, and a touch panel that combines display and operation functions. The user inputs various imaging modes, imaging conditions, etc., by operating the control unit 115.

[0046] The input / output interface (I / F) 116 is a general term for interface circuits (such as USB interfaces) to external media such as SD cards or personal computers. Communication via the I / F 116 can be wireless or wired. Data of equirectangular projection images stored in the DRAM 114 can be recorded to external media via the I / F 116, or transmitted to external terminals (devices) via the I / F 116 as needed.

[0047] The short-range communication circuit 117 communicates with an external terminal (device) via an antenna 117a provided on the imaging device 10 using short-range wireless communication technologies such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi. The short-range communication circuit 117 can transmit equirectangular projection image data to the external terminal (device).

[0048] The electronic compass 118 calculates the orientation of the imaging device 10 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data.

[0049] The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) associated with the movement of the imaging device 10. Changes in angle are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

[0050] The acceleration sensor 120 is a sensor that detects acceleration in three axes.

[0051] The imaging device 10 can also calculate its own orientation (angle relative to the direction of gravity) using an electronic compass 118, an acceleration sensor 120, etc. Furthermore, by providing the acceleration sensor 120, the accuracy of image correction can be improved in the imaging device 10.

[0052] Network I / F 121 is an interface for data communication using a communication network 100 such as the Internet via a router or the like. Furthermore, the hardware configuration of the imaging device 10 is not limited to that shown herein, and any configuration that can realize the functional configuration of the imaging device 10 is acceptable. In addition, at least a part of the above hardware configuration may reside on the relay device 3 or the communication network 100.

[0053] <Hardware configuration of the relay device> Figure 11 is a hardware configuration diagram of the relay device 3. Note that Figure 11 is a hardware configuration diagram when the relay device 3 is a cradle with wireless communication capabilities.

[0054] As shown in Figure 11, the relay device 3 includes a CPU 301, ROM 302, RAM 303, EEPROM 304, CMOS sensor 305, bus line 310, communication unit 313, antenna 313a, positioning unit 314, and input / output I / F 316.

[0055] Of these, the CPU 301 controls the operation of the entire relay device 3. The ROM 302 stores programs used to drive the CPU 301, such as the IPL (Initial Program Loader). The RAM 303 is used as the work area for the CPU 301.

[0056] The EEPROM (Electrically Erasable and Programmable ROM) 304 reads or writes data according to the control of the CPU 301. The EEPROM 304 stores the operating system (OS) executed by the CPU 301, other programs, and various data.

[0057] The CMOS (Complementary Metal Oxide Semiconductor) sensor 305 is a solid-state image sensor that captures an image of a subject and obtains image data according to the control of the CPU 301.

[0058] The communication unit 313 uses the antenna 313a to communicate with the communication network 100 via wireless communication signals.

[0059] The positioning unit 314 receives positioning signals including the location information (latitude, longitude, and altitude) of the relay device 3 from GNSS (Global Navigation Satellite System) satellites such as GPS (Global Positioning Systems) satellites, or from IMES (Indoor Messaging System) as an indoor GPS.

[0060] The input / output interface (I / F316) is an interface circuit (such as a USB interface) that is electrically connected to the input / output interface (I / F116) of the imaging device 10. Communication via the input / output interface (I / F316) can be wireless or wired.

[0061] Bus line 310 is an address bus, data bus, etc., used to electrically connect various components such as the CPU 301.

[0062] <Hardware configuration of communication control devices and communication terminals> Figure 12 is a hardware configuration diagram of the communication control device and communication terminals. Note that the hardware configuration of communication terminals 7 and 9 is the same as that of the communication control device 5, so their explanation is omitted.

[0063] The communication control device 5, as a computer, includes a CPU 501, ROM 502, RAM 503, SSD 504, external device connection I / F 505, network I / F 506, display 507, operation unit 508, media I / F 509, bus line 510, CMOS sensor 511, speaker 512, microphone 513, and positioning unit 514, as shown in Figure 12.

[0064] Of these, the CPU 501 controls the operation of the entire communication control device 5. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501.

[0065] SSD504 reads or writes various data according to the control of CPU501. Note that if the communication terminals 7 and 9 are smartphones, SSD504 may not be required. Alternatively, an HDD (Hard Disk Drive) may be used instead of SSD504.

[0066] The External Device Connection I / F 505 is an interface for connecting various external devices. These external devices include displays, speakers, keyboards, mice, USB memory sticks, and printers.

[0067] Network I / F 506 is an interface for data communication via communication network 100.

[0068] Display 507 is a type of display unit that displays various images, such as liquid crystal displays (LCDs) or organic EL (Electro-Luminescence) displays.

[0069] The operation unit 508 is an input means for selecting and executing various instructions such as various operation buttons, power switches, shutter buttons, and touch panels, selecting the object to be processed, and moving the cursor.

[0070] The media interface 509 controls the reading or writing (storage) of data to or from the recording media 509m, such as flash memory. The recording media 509m includes DVDs and Blu-ray Discs (registered trademarks), etc.

[0071] The CMOS sensor 511 is a type of imaging means that captures an image of a subject according to the control of the CPU 501 and obtains image data. A CCD sensor may be used instead of a CMOS sensor.

[0072] Speaker 512 is a circuit that converts electrical signals into physical vibrations to produce sound such as music and speech. Microphone 513 is a circuit that converts the collected sound into sound (signal) data.

[0073] The positioning unit 314 receives positioning signals containing location information (latitude, longitude, and altitude) of communication terminals 7 and 9 from GNSS satellites such as GPS satellites, or from IMES as an indoor GPS.

[0074] Bus line 510 is an address bus, data bus, etc., used to electrically connect various components such as the CPU 501.

[0075] [Functional configuration of the embodiment] Next, the functional configuration of the first embodiment will be described using Figures 13 to 16.

[0076] <Functional configuration of the imaging device> As shown in Figure 13, the imaging device 10 includes a reception unit 12, a detection unit 13, an imaging unit 16, a sound collection unit 17, a connection unit 18, and a storage / reading unit 19. Each of these units is a function or means realized by any of the components shown in Figure 10 operating according to instructions from the CPU 111 that follow a program for imaging and storage deployed on SRAM 113 to DRAM 114.

[0077] Furthermore, the imaging device 10 has a storage unit 1000 constructed from at least one of the ROM 112, SRAM 113, and DRAM 114 shown in Figure 9.

[0078] (Functional configuration of each part of the imaging device) The reception unit 12 of the imaging device 10 is realized by the processing of the operation unit 115 to the CPU 111, and receives operation input from the user.

[0079] The detection unit 13 is mainly implemented by processing the electronic compass 118, gyro sensor 119, and acceleration sensor 120, etc., from the CPU 111, and obtains attitude information by detecting the attitude of the imaging device 10.

[0080] The imaging unit 16 is mainly realized by processing from the CPU 111 to the imaging unit 101, image processing unit 104, imaging control unit 105, and CPU 111, and captures images of landscapes, etc., and obtains captured images.

[0081] The sound collection unit 17 is mainly realized by processing performed by the CPU 111 on the sound processing unit 109, and it collects sounds from the surrounding area of ​​the imaging device 10.

[0082] The connection section 18 is mainly implemented by processing the input / output interface 116 from the CPU 111 and communicates data with the relay device 3.

[0083] The memory / read unit 19 is mainly implemented by the CPU 111 and stores various data (or information) in the memory unit 1000 and reads various data (or information) from the memory unit 1000.

[0084] <Functional configuration of the relay device> As shown in Figure 13, the relay device 3 has a communication unit 31 and a connection unit 38. Each of these units is a function or means realized by any of the components shown in Figure 11 operating according to instructions from the CPU 301 that follow the program for the relay device 3 deployed from the EEPROM 304 onto the RAM 303.

[0085] (Functional configuration of relay device 3) The communication unit 31 of the relay device 3 is mainly realized by processing from the CPU 301 shown in Figure 11 to the communication unit 313, and performs data communication between the imaging device 10 and the communication control device 5 via the communication network 100.

[0086] The connection section 38 is mainly implemented by processing the input / output interface 316 from the CPU 301 and communicates data with the imaging device 10.

[0087] <Functional Configuration of Communication Control Device> Next, the functional configurations of the communication control device 5 will be described in detail using Figure 13. The communication control device 5 includes a communication unit 51, a receiving unit 52, a creation unit 53, a processing unit 54, an authentication unit 55, a text generation unit 56, and a storage / reading unit 59. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 501 that follow the program for the communication control device 5 deployed from the SSD 504 onto the RAM 503.

[0088] Furthermore, the communication control device 5 has a storage unit 5000 constructed from at least one of the RAM 503 and HD 504 shown in Figure 12. This storage unit 5000 contains a user / device management DB 5001, a virtual room management DB 5002, and a three-dimensional image management DB 5003.

[0089] (User / Device Management Database) Figure 14 is a conceptual diagram of the user device management table. The user device management DB5001 consists of the user device management table shown in Figure 14. In the user device table, the device ID (or user ID), password, name, device image (or user image), and IP address are stored and managed in association with each other.

[0090] Of these, the User ID is an example of user identification information used to identify a user (e.g., organizer X, participant A, B). The Device ID is an example of device identification information used to identify a device such as the imaging device 10. If a head-mounted display or the like is used in addition to the imaging device 10, the head-mounted display or the like will also be treated as a device.

[0091] The name is the name of the user or device. Note that each user name may also be the name of the user's communication terminal.

[0092] The device image is a schematic representation of a device such as the camera 10, and is registered in advance by a designated user. The user image is a schematic representation of each user's face, a photograph of the user's face, etc., and is registered in advance by each user.

[0093] An IP address is an example of destination identification information for devices such as communication terminals 7 and 9, and imaging device 10, used by the user.

[0094] (Virtual Room Management Database) Figure 15 is a conceptual diagram of the virtual room management table. The virtual room management DB5002 is composed of the virtual room management table shown in Figure 15. The virtual room management table stores and manages the following information in association: recording date, virtual room ID, virtual room name, device ID, organizer ID, participant ID, content ID, content URL (information on the storage location of image and sound content data), field of view information URL (information on the storage location of field of view information), and 3D image ID.

[0095] Of these, the recording date indicates the date on which the process S36 described below was performed.

[0096] A virtual room ID is an example of virtual room identification information used to identify a virtual room.

[0097] The virtual room name is the name of the virtual room, and it is assigned by the user or other user.

[0098] The device ID is the same as the device ID in Figure 14, and is the ID of the device that joined the virtual room indicated by the virtual room ID in the same record.

[0099] The Organizer ID is an example of organizer identification information used to specifically identify the Organizer ID among the User IDs in Figure 14, and is the ID of the organizer who participated in the virtual room indicated by the virtual room ID in the same record.

[0100] The participant ID is an example of participant identification information used to specifically identify the participant among the user IDs in Figure 14, and is the ID of the participant who joined the virtual room indicated by the virtual room ID in the same record.

[0101] Content ID is an example of content identification information used to identify image and sound content data. In this case, the image is a wide-field image obtained at the time of capture, and the sound is sound (including speech) obtained at the same time of capture.

[0102] The content URL is an example of content storage location information that indicates where the content (wide-field image, sound information) data is stored (see processing S54). Along with the content data, the content URL also stores a timestamp indicating the start and end dates (times) of the shooting (video recording) and sound collection (audio recording), as well as the shooting location (absolute location on Earth).

[0103] The field of view information URL is an example of field of view information storage location information that indicates the location where field of view information showing the field of view in a predetermined area image displayed by the user (for example, organizer X, participants A and B) during recording is stored (see processing S37 to S39).

[0104] The three-dimensional image ID is an example of three-dimensional image identification information that identifies the three-dimensional image to be displayed in the display area 620 described later.

[0105] (3D image management DB) (Part 1) Figure 16 is a conceptual diagram of the three-dimensional image management table (part 1). The three-dimensional image management DB5003 is composed of the three-dimensional image management table shown in Figure 16. In the three-dimensional image management table shown in Figure 16, the model ID and location information are associated and managed for each three-dimensional image ID.

[0106] The model ID is an example of three-dimensional model identification information used to identify a three-dimensional model. A three-dimensional model is generated based on point cloud data acquired by, for example, a TOF (Time-of-Flight) camera.

[0107] Position information is information that indicates the position of a three-dimensional model in a three-dimensional virtual space using three-dimensional coordinates (XYZ). For example, it is indicated by the three-dimensional coordinates of eight points that define the rectangular space occupied by the three-dimensional model.

[0108] (Part 2) Figure 17 is a conceptual diagram of the three-dimensional image management table (part 2). The three-dimensional image management DB5003 is composed of the three-dimensional image management table shown in Figure 17. The three-dimensional image management table shown in Figure 17 is a table for managing attribute information that indicates the attributes of the parts (three-dimensional models) that make up a structure contained in a three-dimensional virtual space. In the three-dimensional image management table shown in Figure 17, attribute information such as part number, material information, dimension information, color information, material information, location information, and construction date information are associated and managed for each three-dimensional image ID.

[0109] Component information is information that identifies components such as walls, floors, ceilings, windows, pipes, and doors, which are made up of parts.

[0110] Dimensional information is information that identifies the dimensions of a part in virtual space, and is represented, for example, by numerical values ​​in the three axes of X, Y, and Z.

[0111] Color information identifies the color of a part, while material information identifies the material of a part.

[0112] Position information identifies the location of a part in virtual space and is represented, for example, by coordinates in the three axes of X, Y, and Z. This makes it possible to identify whether or not multiple parts are adjacent to each other.

[0113] Construction date information indicates the planned date on which a component is to be installed in the real world. This makes it possible to identify a structure excluding components that have not yet been installed at a given point in time.

[0114] As explained above, the positional information in Figures 16 and 17 is managed in correspondence with absolute positions on Earth. For example, by associating the origin of the positional information in Figures 16 and 17 (X=0, Y=0, Z=0) with absolute positions on Earth (latitude, longitude, altitude), all coordinates in three-dimensional images, including three-dimensional models and parts, are associated with absolute positions on Earth.

[0115] Furthermore, the 3D image management DB5003 may manage 3D point clouds, mesh objects, textured mesh objects, etc., instead of 3D models.

[0116] (Movement history management database) Figure 18A is a conceptual diagram of a previously created movement history management table. Figure 18B is a conceptual diagram of a previously created movement history management table. Figure 18C is a conceptual diagram of a movement history management table currently being created. The movement history management DB 5004 is composed of the movement history management tables shown in Figures 18A, B, and C. The movement history management DB 5004 is used in the fixed browsing content mode.

[0117] The fixed content mode is a mode that makes the content displayed on screens 600 and 601 the same (common) on all communication terminals 7, 9a, and 9b that are in the same virtual room. In this case, the content displayed on one of the communication terminals 7, 9a, and 9b is fixed to the content currently displayed on that terminal (for example, communication terminal 9a), and the same content is displayed on the other communication terminals (for example, communication terminals 7 and 9b).

[0118] In each movement history management table, the date and time of "shooting and sound collection," shooting location, field of view information, and audio text are associated and stored for each content ID. If sound collection is not performed, the shooting date and time are shown. The position of the shooting device 10 is determined by the positioning unit 314 of the relay device 3 to which the shooting device 10 is attached. Alternatively, the shooting device 10 may be equipped with a positioning unit similar to the positioning unit 314, and the position of the shooting device 10 may be determined by this positioning unit. Here, since the movement history management tables in Figures 18A, B, and C have similar data structures, the movement history management table in Figure 18A will be explained, and the explanations of the movement history management tables in Figures 18B and C will be omitted.

[0119] The content ID shown in Figure 18A is equivalent to the content ID shown in Figure 15.

[0120] The date and time of shooting and sound collection indicates the date and time when the shooting device 10 captured images and sound was collected.

[0121] The shooting position indicates the position (absolute position on Earth) at which the shooting device 10 took the image. If sound was collected, the shooting position is also the sound collection position.

[0122] The field of view information is information used to identify a predetermined region that represents a predetermined region image displayed on one of the communication terminals 7, 9a, or 9b (in this case, communication terminal 9a). For example, if a fixed viewing content mode (participant A's fixed viewing content mode) is set so that screens 600 and 601 displayed and viewed by participant A on communication terminal 9a are displayed with the same content on organizer X's communication terminal 7 and participant B's communication terminal 9b, the field of view information (including user ID) received by the communication control device 5 in the process S38 described later is managed not only in the field of view information URL managed in the virtual room management DB 5002 (see Figure 15) but also in the movement history management DB 5004 (see Figure 18C) according to the date and time of shooting and sound collection. In this case, field of view information is not transmitted by communication terminals other than the fixed target (for example, communication terminals 7 and 9b) (for example, in the processes S37 and S39 described later).

[0123] In the case of a fixed viewing content mode for the content displayed on the communication terminal 9a, the communication control device 5 not only stores the field of view information received in the processes S37 to S39 described below in the field of view information URL managed in the virtual room management DB 5002 (see Figure 15), but also manages the field of view information (field of view information received in process S38) for identifying predetermined area images included in the screens 600 and 601 currently displayed on the communication terminal 9a to be fixed in the movement history management DB 5004.

[0124] Furthermore, in the case of a fixed viewing content mode for the display content of communication terminal 9a, in order to display the same predetermined area image with the same display content as communication terminal 9a on communication terminals 7 and 9b, the communication control device 5 transmits the field of view information (including the user ID of participant A) received in process S38 to communication terminal 7 in process S13 (described later) and to communication terminal 9b in process S15. As a result, communication terminal 7 (display control unit 74) and communication terminal 9a (display control unit 94) can display the same content as the predetermined area image displayed on communication terminal 9a.

[0125] In addition, the content-fixed viewing mode may be set to display the same content on the screen being viewed on the organizer X's communication terminal 7, and the same content may be displayed on other communication terminals 9a and 9b, or it may be set to display the same content on the screen being viewed on the organizer X's communication terminal 9b, and the same content may be displayed on other communication terminals 7 and 9a.

[0126] The voice text registered in the "Voice Text" field is an example of text information converted from voice data recorded at the same recording date and time in the same record. The voice text generated from voice data collected by the recording device 10 (e.g., recording device α) is managed chronologically with respect to the device ID, and the voice text generated from voice data collected by the communication terminals 7, 9 (e.g., communication terminal 9a) is managed chronologically with respect to the user ID of the user of the communication terminal 7, 9 (e.g., communication terminal 9a). Note that the "Voice Text" field only needs to manage voice text based on voice data collected by at least one of the recording device 10 and the communication terminals 7, 9.

[0127] When the audio text is generated by the text generation unit 56 using the current (streaming) audio, pressing the "Duplicate Audio" button 683 (see Figure 35) will register (duplicate) the past audio text currently displayed in the display area 635 in the "Audio Text" field of Figure 18C by the processing unit 54. This past audio text is the audio text in the virtual room management DB 5002 (see Figure 15) that is identified in the movement history management table (see Figure 18B) by the content ID (e.g., c091) of another record (from a past recording date) that has the same three-dimensional image ID (e.g., v001) associated with the content ID (e.g., c101) of the movement history management table (see Figure 18C) currently being created.

[0128] For example, when the "Duplicate Voice" button 683 shown in Figure 35 is pressed, the voice text "Where is the new box?" from the movement history management table (see Figure 18B), which is the source of the voice text "Where is the new box?" currently displayed in the display area 635, is duplicated in the movement history management table (see Figure 18C) to the "Voice Text" field corresponding to the predetermined area image displayed in the display area 610 shown in Figure 35. In Figure 18C, parentheses are used to indicate that it was duplicated later, so it is shown as "(Where is the new box.)". Here, in Figure 18C, the voice text "Where is the new box?" is registered (duplicated) within the range where the shooting position and field of view information do not change (see S4). Note that the display area 635 is an example of a text display area.

[0129] Furthermore, when duplicating audio text if audio text is already registered in the "Audio Text" field, you can either overwrite the already registered audio text or add to it when duplicating. When adding, the already registered audio text and the newly duplicated audio text will be registered separately to prevent them from being mixed into a single sentence.

[0130] Conversely, by pressing the "Audio Duplicate" button 681 described later, the current audio text generated by the current audio output along with the currently displayed predetermined area image in the display area 610 is registered (duplicated) in the "Audio Text" field of the Movement History Management Table (see Figure 18C) by the processing unit 54. This current audio text is the audio text in the Movement History Management Table (see Figure 18C) identified in the Virtual Room Management DB 5002 (see Figure 15) by the content ID (e.g., c101) of another record (currently being recorded) that has the same three-dimensional image ID (e.g., v001) associated with the content ID (e.g., c091) of a Movement History Management Table (see Figure 18B) created in the past.

[0131] For example, when the "Duplicate Voice" button 681 shown in Figure 35 is pressed, the voice text "I'll reinforce the bottom of the box." from the movement history management table (see Figure 18C), which is the source of the voice text "I'll reinforce the bottom of the box." being recorded, is duplicated in the movement history management table (see Figure 18B) to the "Voice Text" field corresponding to the predetermined area image displayed in the display area 630 shown in Figure 35. Note that in Figure 18B, parentheses are used to indicate that it was duplicated later, so it is shown as "(I'll reinforce the bottom of the box.)". Here, in Figure 18B, the voice text "I'll reinforce the bottom of the box." is registered (duplicated) within the range where the shooting position and field of view information do not change (see S3).

[0132] Furthermore, if voice text is already registered in the "Voice Text" field, when duplicating voice text, it will either be overwritten or added to as a duplicate, as described above.

[0133] Furthermore, each movement history management table is provided with a registration field for registering the display position of the virtual camera IC1 icon for each shooting and sound collection date and time. For example, in the movement history management table (see Figure 18B), when a flag is registered in the registration field, the creation unit 53 can display the virtual camera IC1 icon 622a, etc., at the shooting position in the display area 620 of the previously registered three-dimensional image, as shown in Figure 34. Similarly, for example, in the movement history management table (see Figure 18C), when a flag is registered in the registration field, the creation unit 53 can display the virtual camera IC1 icon at the shooting position in the display area 620 of the previously registered three-dimensional image, where the present is considered the past, as shown in Figure 34.

[0134] (Functional configuration of each communication control device) Next, we will explain in detail the functional configuration of the communication control device 5 using Figure 13.

[0135] The communication unit 51 of the communication control device 5 is mainly implemented by processing from the CPU 501 shown in Figure 11 to the network interface 505, and performs data communication with other devices (relay device 3, communication terminals 7, 9) via the communication network 100. The communication unit 51 also acts as an acquisition unit, acquiring instruction information indicating instructions sent from the communication terminals 7, 9.

[0136] The reception unit 52 is implemented by processing from the CPU 501 to the operation unit 508 and receives operation input from the user (in this case, the system administrator, etc.).

[0137] The creation unit 53 is mainly implemented by the processing of the CPU 501 and uses data stored in the storage unit 5000 to create screens to be sent to each communication terminal 7,9. For example, the creation unit 53 creates a screen 600 that includes a display area 610 that displays a predetermined region image, which is a predetermined region in a wide-field image (an example of a captured image) captured by the imaging device 10, and a display area 620 that displays at least a part of a three-dimensional image including a second position (coordinate) corresponding to a first position (coordinate) in the wide-field image, by the processing unit 54 (see Figure 26).

[0138] The processing unit 54 is mainly implemented by the CPU 501 and performs the process of associating position information indicating the position in the captured image obtained by photographing the object with the imaging device 10 (an example of a first position) with position information indicating the position in the three-dimensional image including the three-dimensional region corresponding to the object (an example of a second position). The processing unit 54 can also be referred to as the "association unit".

[0139] The authentication unit 55 is primarily implemented by the CPU 501 and performs authentication to determine whether each user is a legitimate user for using the virtual room.

[0140] The text generation unit 56 is mainly implemented by the processing of the CPU 501 and generates speech text data from the sound (voice) data of the content data using speech recognition technology.

[0141] The memory / read unit 59 is mainly implemented by the CPU 501, and stores various data (or information) in the memory unit 5000 and reads various data (or information) from the memory unit 5000.

[0142] <Functional Configuration of Communication Terminal 7> Next, the functional configuration of the communication terminal 7 will be described in detail using Figure 13. The communication terminal 7 includes a communication unit 71, a receiving unit 72, a creation unit 73, a display control unit 74, an audio input / output control unit 75, a connection unit 78, and a storage / reading unit 79. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 501 that follow the program for the communication terminal 7 deployed from the SSD 504 onto the RAM 503.

[0143] The communication unit 71 of the communication terminal 7 is mainly implemented by processing from the CPU 501 to the network interface 505, and performs data communication with other devices (communication control device 5) via the communication network 100.

[0144] The reception unit 72 is primarily implemented by processing from the CPU 501 to the operation unit 508, and receives instructions from the user (in this case, the organizer X) through operation input. The reception unit 72 also functions as an acquisition unit, and can be said to acquire instructions made by the user's operation.

[0145] The creation unit 73 is mainly implemented by the processing of the CPU 501 and uses data stored in the memory unit 7000 to create a screen for display on its own terminal (in this case, the communication terminal 7). When the creation unit 53 of the communication control device 5 creates the screen, the communication terminal 7 may or may not have a creation unit 73.

[0146] The display control unit 74 is mainly implemented by the processing of the CPU 501 and controls the display of various images on the display 507 of the communication terminal 7 or an external display connected to the external device connection I / F 505.

[0147] The sound input / output control unit 75 is primarily implemented by the CPU 501 of the communication terminal 7 and controls the output of sound to the speaker 512 of the communication terminal 7 or an external speaker connected to the external device connection I / F 505. The sound input / output control unit 75 also controls the collection of sound from the microphone 513 and an external microphone connected to the external device connection I / F 505. The sound input / output control unit 75 also performs the same functions as the sound collection unit 17 of the imaging device 10, and is therefore also an example of a sound collection unit.

[0148] The connection section 78 is mainly realized by processing from the CPU 501 to the external device connection I / F 505, and when the communication terminal 7 acts as a substitute for the relay device 3, it communicates data with the connection section 18 of the imaging device 10.

[0149] The memory / read unit 79 is mainly implemented by the CPU 501, and stores various data (or information) in the memory unit 7000 and reads various data (or information) from the memory unit 7000.

[0150] <Functional Configuration of Communication Terminal 9> Next, we will explain in detail the functional configuration of the communication terminal 9 using Figure 13.

[0151] The communication terminal 9 includes a communication unit 91, a reception unit 92, a creation unit 93, a display control unit 94, an audio input / output control unit 95, a connection unit 98, and a storage / reading unit 99. Each of these units is a function or means realized by any of the components shown in Figure 12 operating according to instructions from the CPU 501 that follow the program for the communication terminal 9 deployed from the SSD 504 onto the RAM 503.

[0152] Furthermore, the communication terminal 9 has a storage unit 9000 constructed from the RAM 503 and SSD 504 shown in Figure 12.

[0153] The communication unit 91 of the communication terminal 9 is mainly implemented by processing from the CPU 501 to the network interface 505, and performs data communication with other devices (communication control device 5) via the communication network 100.

[0154] The reception unit 92 is primarily implemented by processing from the CPU 501 to the operation unit 508, and receives instructions from the user (in this case, the participant) through operational input. The reception unit 92 also functions as an acquisition unit, and can be said to acquire instructions made by the user's operations.

[0155] The creation unit 93 is mainly implemented by the processing of the CPU 501 and uses data stored in the memory unit 9000 to create a screen for display on its own terminal (communication terminal 9). When the creation unit 53 of the communication control device 5 creates the screen, the communication terminal 9 may or may not have a creation unit 73.

[0156] The display control unit 94 is mainly implemented by the processing of the CPU 501 and controls the display of various images on the display 507 of the communication terminal 9 or an external display connected to the external device connection I / F 505.

[0157] The sound input / output control unit 95 is mainly implemented by the CPU 501 of the communication terminal 9 and controls the output of sound to the speaker 512 of the communication terminal 9 or an external speaker connected to the external device connection I / F 505. The sound input / output control unit 75 controls the collection of sound from the microphone 513 and an external microphone connected to the external device connection I / F 505. The sound input / output control unit 95 also performs the same function as the sound collection unit 17 of the imaging device 10, and is therefore also an example of a sound collection unit.

[0158] The connection unit 98 is primarily realized by processing the external device connection I / F 505 of the communication terminal 7 from the CPU 501, and performs data communication with external devices connected by wire or wireless.

[0159] The memory / read unit 99 is mainly implemented by the CPU 501, and stores various data (or information) in the memory unit 9000 and reads various data (or information) from the memory unit 9000.

[0160] [Processing or operation of the embodiment] Next, the processing or operation of the embodiment will be described using Figures 19 to 35. Note that the following processing takes place after the imaging device 10 and communication terminals 7 and 9 have already joined the same virtual room.

[0161] <Communication processing of content data in communication systems> First, we will explain the content data communication process in communication system 1 using Figure 19. Figure 19 is a sequence diagram showing the content data communication process in the communication system. In this embodiment, we will explain the case where the shooting device 10, the organizer X's communication terminal 7, participant A's communication terminal 9a, and participant B's communication terminal 9b are in the same virtual room. When the virtual room is established, the storage / reading unit 79 adds one record to the virtual room management DB (see Figure 15) and manages the virtual room ID, virtual room name, device ID, organizer ID, and participant ID in association. The contents of the content ID, content URL, field of view information URL, and three-dimensional image ID are stored later. Note that the processes S11 to S15 in Figure 19 are repeated, for example, about 30 or 60 times per second.

[0162] S11: In the shooting device 10, the imaging unit 16 takes a 360-degree spherical image of the site Sa and collects sound to obtain content (wide-field image, sound information) data, after which the connection unit 18 sends the content data to the relay device 3. In this case, the sound collected is mainly the voice of the organizer X of the site Sa. Furthermore, the connection unit 18 also sends a virtual room ID to identify the virtual room in which the camera 10 is participating, and a device ID to identify the camera 10. In addition, the camera 10 transmits information about the shooting position, which is the position (absolute position) of the camera 10, to the relay device 3 at predetermined intervals (for example, every second). As a result, the connection unit 38 of the relay device 3 acquires the content data, virtual room ID, device ID, and shooting position information. The shooting position is the position (absolute position on Earth) obtained from the positioning unit 314.

[0163] S12: In the relay device 3, the communication unit 31 transmits the content data, virtual room ID, device ID, and shooting location information acquired by the connection unit 38 in processing S11 to the communication control device 5 via the communication network 100. As a result, the communication unit 51 in the communication control device 5 receives the content data, virtual room ID, device ID, and shooting location information.

[0164] Then, the storage and reading unit 59 of the communication control device 5 stores the content data received in process S12 in the virtual room management DB 5002 (see Figure 15) for the content URL corresponding to the virtual room ID received in process S12.

[0165] Alternatively, the imaging device 10 may transmit content data, virtual room ID, device ID, and imaging location information to the communication terminal 7 instead of the relay device 3 (S11d). In this case, the communication terminal 7 transmits content data, virtual room ID, device ID, and imaging location information to the communication control device 5 (S12d).

[0166] S13: In the communication control device 5, the storage / reading unit 59 searches the virtual room management DB 5002 based on the virtual room ID received in processing S12, and reads the user IDs (organizer ID and participant ID) of users participating in the same virtual room as the camera 10. The storage / reading unit 59 also searches the user / device management DB 5001 based on the read organizer ID and participant ID, and reads the corresponding user image and IP address of the communication terminal 7 for organizer X, as well as the corresponding user images and communication terminals 9a and 9b and their respective IP addresses for participants A and B. The communication unit 51 then refers to the IP address of the communication terminal 7 and performs bidirectional communication with the communication terminal 7. In this case, the communication terminal 7 transmits content data (image data captured by its own terminal and audio data obtained by sound collection), the virtual room ID, and the user ID of user X of the communication terminal 7 to the communication control device 5 at any time (for example, every 1 / 30 or 1 / 60 second). The contents of this bidirectional communication will be described later.

[0167] S14: The communication unit 51 of the communication control device 5 refers to the IP address of the communication terminal 9a and performs bidirectional communication with the communication terminal 9a. In this case, the communication terminal 9a transmits content data (image data captured by its own terminal and audio data collected), a virtual room ID, and the user ID of communication terminal 9a user A to the communication control device 5 at intervals (for example, every 1 / 30 or 1 / 60 of a second). The contents of this bidirectional communication will be described later.

[0168] S15: Similarly, the communication unit 51 of the communication control device 5 refers to the IP address of the communication terminal 9b and performs bidirectional communication with the communication terminal 9b. In this case, the communication terminal 9b transmits content data (image data captured by its own terminal and audio data collected), a virtual room ID, and the user ID of communication terminal 9a user B to the communication control device 5 at any time (for example, every 1 / 30 or 1 / 60 of a second). The contents of this bidirectional communication will be described later.

[0169] <Process for initiating video and audio recording in a communication system> Next, Figure 20 will be used to explain the process of starting video recording and audio recording in communication system 1. Figure 20 is a sequence diagram showing the process of starting video recording and audio recording in the communication system.

[0170] S31: First, at the organizer X's communication terminal 7, the reception unit 72 receives the command from organizer X to start recording and audio recording.

[0171] S32: Before recording video and audio, the communication unit 71 of the communication terminal 7 sends a sharing instruction for field of view information to the communication control device 5. This sharing instruction includes the virtual room ID of the virtual room in which the communication terminal 7 is participating, and the device ID of the camera 10. As a result, the communication unit 51 of the communication control device 5 receives the sharing instruction for field of view information.

[0172] S33: In the communication control device 5, the storage / reading unit 59 sets the content URL and the field of view information URL of the virtual room management DB (see Figure 15). Then, the communication unit 51 sends a recording start instruction and a field of view information upload request to the communication terminal 7. This instruction includes information indicating the content URL, which is the location where the communication terminal 7 will save the recorded content data. This request also includes information indicating the field of view information URL for saving the field of view information. As a result, the communication unit 71 in the communication terminal 7 receives the recording start instruction and the field of view information upload request.

[0173] S34: The communication unit 51 also sends a request to the communication terminal 9a to upload the field of view information. This request includes information about the field of view information URL for saving the field of view information. As a result, the communication terminal 9a receives the request for uploading the field of view information from the communication unit 91.

[0174] S35: Similarly, the communication unit 51 sends a request to the communication terminal 9b to upload the field of view information. This request includes information about the field of view information URL for saving the field of view information. As a result, the communication terminal 9b receives the request to upload the field of view information from the communication unit 91.

[0175] S36: Next, in the communication terminal 7, the storage / reading unit 79 acts as both a recording unit and an audio recording unit, and begins recording the content data received in process S13 as shown in Figure 19. In the case of process S12d as shown in Figure 19, the communication terminal 7 may also begin recording the content data received from the camera 10 in process S11d, rather than the content data received from the communication control device 5 in process S13.

[0176] S37: In the communication terminal 7, for example, while displaying a predetermined region image (see Figure 6(b)), which is a predetermined region (see Figure 6(a)) of the wide-field image received in processing S13, the reception unit 72 receives a request from the organizer X to change the field of view, and the display control unit 74 displays a predetermined region image (see Figure 6(d)), which is the same wide-field image after the change (see Figure 6(c)). In this case, the reception unit 72 also acts as an acquisition unit, and when it receives a request from the user (in this case, the organizer X) to display a predetermined region in the wide-field image, it acquires field of view information (pan, tilt, fov) to identify the predetermined region to be displayed on the display 507 in the wide-field image.

[0177] Then, the communication unit 71 transmits field-of-view information to the field-of-view information URL (communication control device 5) received in processing S33, in order to identify the predetermined area after the change. This field-of-view information includes the user ID of the organizer X of the communication terminal 7, which is the source of the transmission. As a result, the communication control device 5 receives the field-of-view information from the communication unit 51.

[0178] The memory / reading unit 79 then stores the user ID and field of view information of the organizer X, as well as the date and time of their reception, associated with the field of view information URL managed in the virtual room management DB (see Figure 15). In this case, the date and time of reception is managed as the date and time when the reception unit 72 of the communication terminal 7 received the field of view change from the organizer X.

[0179] S38: The communication terminal 9a and the communication control device 5 also perform the same processing as in processing S37, independently of processing S37. In this case, the user ID transmitted is the user ID of participant A, and the reception date and time in this case is managed as the date and time when the reception unit 92 of the communication terminal 9a received the change in field of view from participant A.

[0180] S39: In the communication terminal 9b and the communication control device 5, the same processing as in process S37 is performed independently of processes S37 and S38. In this case, the user ID transmitted is the user ID of participant B, and the reception date and time in this case is managed as the date and time when the reception unit 92 of the communication terminal 9b received the change in field of view from participant B.

[0181] As a result, when the predetermined area image is changed on the communication terminal 7, as shown in Figures 6(b) to 6(d), the communication control device 5 can manage the predetermined area that the organizer X is focusing on by viewing the predetermined area image shown in Figure 6(d). Similarly, the communication control device 5 can manage the predetermined areas that each participant A and B are focusing on by viewing them on the communication terminals 9a and 9b. Note that processes S37 to S39 may be executed collectively on the communication control device 5 at the end of recording.

[0182] On the other hand, in the fixed viewing content mode, during processing S37 to S39, only the designated communication terminal to be fixed (for example, communication terminal 9a) transmits the field of view information (including the user ID of participant A) to the communication control device 5.

[0183] <Process to stop recording video and audio in a communication system> Next, Figure 21 will be used to explain the process of stopping video recording and audio recording in communication system 1. Figure 21 is a sequence diagram showing the process of stopping video recording and audio recording in the communication system.

[0184] S51: First, at the organizer X's communication terminal 7, the reception unit 72 receives a request from organizer X to stop recording and audio recording.

[0185] S52: The memory / reading unit 79 stops recording and audio recording of the content data.

[0186] S53: The communication unit 71 uploads (transmits) the recorded content data to the predetermined content URL (communication control device 5) received in processing S33. This content data includes the timestamps for each stage from the start to the end of the recording. As a result, the communication control device 5 receives the content data from the communication unit 51. The timestamps are the same as the recording and sound collection dates and times shown in Figure 18.

[0187] S54: In the communication control device 5, the storage / reading unit 59 stores content data along with a timestamp at a predetermined content URL. Furthermore, the storage / reading unit 59 converts this timestamp (date and time of shooting and sound collection) into the elapsed playback time, matching it to the total recording time of the content data from which recording was stopped, and manages the playback elapsed time in association with the timestamp.

[0188] S55: The communication unit 51 sends a recording completion notice to the communication terminal 7. The completion notice includes information indicating a predetermined content URL. As a result, the communication unit 71 of the communication terminal 7 receives the recording completion notice.

[0189] S56: Similarly, the communication unit 51 sends a recording completion notice to the communication terminal 9a. The completion notice includes information indicating a predetermined content URL. As a result, the communication unit 91 of the communication terminal 9a receives the recording completion notice.

[0190] S57: Similarly, the communication unit 51 sends a recording completion notice to the communication terminal 9b. The completion notice includes information indicating a predetermined content URL. As a result, the communication unit 91 of the communication terminal 9b receives the recording completion notice.

[0191] In the case of process S55, it is not necessary to include the specified content URL in the completion notification.

[0192] <Processing of playback of recorded video and audio in communication systems> Next, Figures 22 to 28 will be used to explain the process of playing back recorded video and audio in the communication system. Figure 22 is a sequence diagram showing the process of playing back recorded video and audio in the communication system. Figure 23 is a diagram showing the video data selection screen. Here, we will explain the case where participant A plays back the recorded video and audio content data using communication terminal 9a, but the organizer X may play back the content data using communication terminal 7, or participant B may play back the content data using communication terminal 9b.

[0193] S71: First, when the reception unit 92 of the communication terminal 9a receives a login request from participant A by inputting a user ID (participant A's participant ID) and password, the communication unit 91 sends a login request to the communication control device 5. This request includes participant A's user ID and password. As a result, the communication control device 5's communication unit 51 receives the login request, and the authentication unit 55 performs authentication by referring to the user / device management DB (see Figure 14). From here on, the explanation will proceed assuming that participant A has been determined to be a legitimate accessor through login authentication.

[0194] S72: In the communication control device 5, the creation unit 53 creates a recording data selection screen 940 as shown in Figure 21. In this case, the storage / reading unit 59 uses the user ID received in processing S71 (in this case, the participant ID of participant A) as a search key to search the virtual room management DB (see Figure 15) and reads all corresponding virtual room IDs, virtual room names, and content URLs. The creation unit 53 then uses the images from each content data (with timestamps) stored in the content URL to create thumbnails 941, 942, and 943. As a result, the creation unit 53 adds the virtual room name ("Construction Site α", etc.) and the recording time ("2022 / 10 / 31 15:00", etc.) indicating a predetermined time for the timestamp (for example, the recording start time) to each thumbnail.

[0195] S73: The communication unit 51 transmits the selection screen data created in process S72 to the communication terminal 9a. This selection screen data includes a content ID for each thumbnail to identify the wide-field image from which each thumbnail was created. As a result, the communication unit 91 of the communication terminal 9a receives the selection screen data.

[0196] S74: In the communication terminal 9a, the display control unit 94 displays a recording data selection screen on the display 507 of the communication terminal 9a, as shown in Figure 23. Then, the reception unit 92 receives the designation (selection) of a predetermined thumbnail from participant A. Here, we will continue the explanation assuming that thumbnail 941 has been designated (selected).

[0197] S75: The communication unit 71 sends a download request to the communication control device 5 for the content data from which the selected thumbnail 941 was created. This request includes the content ID associated with the thumbnail 941. As a result, the communication unit 51 of the communication control device 5 receives the content data download request.

[0198] S76: In the communication control device 5, the storage / reading unit 59 searches the virtual room management DB (see Figure 15) using the content ID received in processing S75 as a search key, and reads the content data from the corresponding content URL. The content data also includes the location information at the time of shooting by the shooting device 10. The storage / reading unit 59 also searches the three-dimensional image management DB (see Figure 16 or Figure 17) in the virtual room management DB using the three-dimensional image ID associated with the content ID received in processing S75 as a search key, and reads the corresponding model ID and location information parameters (see Figure 16A), or the corresponding part number and location information parameters (see Figure 16B).

[0199] The communication unit 51 then transmits the requested content data along with the three-dimensional image data to the communication terminal 9a. As a result, the communication unit 91 of the communication terminal 9a receives the content data and the three-dimensional image data.

[0200] S77: The communication terminal 9a performs playback processing. Specifically, the display control unit 94 of the communication terminal 9a displays a screen including the image recorded in processing S36 on the display 507 of the communication terminal 9a, and the sound input / output control unit 95 outputs sound.

[0201] <Details of screen display process (registration)> Next, using Figures 24 to 31, we will explain the processing in the case of a fixed viewing content mode, where the display is fixed to a predetermined communication terminal (in this case, communication terminal 9a). Specifically, we will explain process S14 of processes S13 to S15. Here, when participant A operates the communication terminal 9a, the reception unit 92 receives the operation, and the communication unit 91 transmits operation information indicating the operation content to the communication control device 5. As a result, the communication control device 5 has a communication unit 51, which acts as an acquisition unit, acquire the operation information, and the creation unit 53 creates the screen 600 based on the operation content indicated by the operation information. Then, the communication unit 51 transmits the screen 600 data to the communication terminal 9a, the communication unit 91 of the communication terminal 9a receives the screen 600 data, and the display control unit 94 displays the screen 600 on the display 507 of the communication terminal 9a. In this case, the communication control device 5 is an example of an information processing device.

[0202] Alternatively, the communication unit 91 of the communication terminal 9a may receive the data that is the material for creating the screen 600 from the communication control device 5, and the receiving unit 92, which acts as an acquisition unit, may acquire operation information indicating the content of the operation performed by participant A. In this case, the creation unit 93 of the communication terminal 9a creates the screen 600. In this case, the communication terminal 9a is an example of an information processing device. Communication terminals 7 and 9b can also perform the same processing and roles as the communication terminal 9a.

[0203] Next, we will describe the process by which the communication control device 5 creates a screen 600 to be displayed on the display 507 of the communication terminal 9a. Note that processes S13 and S15 are the same as processes S14, with only the terminal being displayed being different, so we will omit the explanation of processes S13 and S15. Figures 24 and 25 are flowcharts showing the processes executed by the communication control device in the screen display process.

[0204] First, the creation unit 53 of the communication control device 5 creates the screen 600 to be displayed on the communication terminal 9a, as shown in Figure 26. Figure 26 is a diagram showing an example of the initial display of the screen to be displayed on the communication terminal 9a. The screen 600 includes a display area 610 (a captured image display area, an example of a first captured image display area) and a display area 620 (an example of a three-dimensional image area). In Figure 26, the display area 610 and the display area 620 are displayed simultaneously on the screen 600 at the same size, but they may be displayed simultaneously at different sizes, or either one may be displayed by switching based on the selection of participant A. Alternatively, the display area 610 may be displayed on one of the two displays of the screen 600, and the display area 620 may be displayed on the other.

[0205] The display area 610 displays a predetermined area image, which is a predetermined region in a wide-field image (an example of a captured image) obtained by photographing objects such as desks, pillars, and windows at the site Sa using the shooting device 10 (see Figures 6(a) and (b)).

[0206] In this case, if the captured image is not a curved image like a wide-field image, the display area 610 displays a predetermined area image in which the same area as the capture area of ​​the captured image is the predetermined area.

[0207] The display area 620 displays a three-dimensional image that includes a three-dimensional model representing the shape of the object in three dimensions, as well as coordinates (an example of second coordinates) that correspond to coordinates (an example of first coordinates) that represent the position in the wide-field image. In this case, the display area 620 displays part or all of the three-dimensional image.

[0208] As explained in Figure 11, the shooting position, which is the position of the shooting device 10, is mapped to an absolute position on Earth, and as explained in Figures 16 and 17, all coordinates in the three-dimensional images, including each three-dimensional model and part managed by the three-dimensional image management DB5003, are also mapped to an absolute position on Earth.

[0209] As a result, the processing unit 54 performs a process of aligning the wide-field image and the three-dimensional image, for example, in processing S12 (or S12d), it associates the coordinates indicating the shooting position as the first position in the wide-field image of the content data received by the communication control device 5 (an example of the first coordinates) with the coordinates indicating the second position in the three-dimensional image that has the same absolute position as these coordinates (an example of the second coordinates). Here, the coordinates are an example of position information.

[0210] As another form of alignment processing, the processing unit 54 may, without using absolute positions on Earth (including indoors), or to complement absolute positions, associate coordinates indicating a first position in a wide-field image with coordinates indicating a second position in a three-dimensional image by image processing, such as matching feature points like edges or textures in a wide-field image and a three-dimensional image.

[0211] Then, as described above, the creation unit 53 creates a screen 600 that includes a display area 610 for displaying a predetermined region image, which is a predetermined region in a wide-field image, and a display area 620 for displaying at least a part of a three-dimensional image in which a second position (second coordinate) is associated with a first position (first coordinate) in the wide-field image by the processing unit 54.

[0212] In this case, if the table of the 3D image management DB5003 manages 3D point clouds, mesh objects, or textured mesh objects instead of 3D models, the display area 620 displays at least a portion of the 3D image, including the 3D point cloud, mesh objects, or textured mesh objects, as a 3D area corresponding to the object included in the wide-field image.

[0213] Furthermore, a "Register" button 680 is displayed on screen 600. The "Register" button 680 is pressed when displaying an icon for a virtual camera IC1 whose shooting area is the angle of view (information) that identifies a predetermined area of ​​the predetermined area image displayed in display area 610, at a position within the display area 620 corresponding to the shooting position of the shooting device 10.

[0214] Additionally, screen 600 displays a close button 609, which is pressed when closing screen 600.

[0215] Furthermore, the content displayed on screen 600 will be explained in detail below. Virtual camera IC1 is used to identify a predetermined region related to a predetermined region image to be displayed in display area 610 (see Figure 8), while virtual camera IC2, shown below, is used to identify a three-dimensional image to be displayed in display area 620.

[0216] S111: First, the creation unit 53 of the communication control device 5 identifies a predetermined area image to be displayed in the display area 610, as shown in Figure 26, based on the field of view information (in this case, the virtual field of view of the virtual camera IC1 which is set in advance) from the wide-field image of the content data received by the communication unit 51.

[0217] S112: The processing unit 54 aligns the position of the virtual camera IC2 with the shooting position associated with the wide-field image obtained by shooting with the shooting device 10, and also aligns the virtual field of view of the virtual camera IC2 (an example of a second field of view) with the virtual field of view of the virtual camera IC1 (an example of a first field of view) that is set in advance. As a result, the creation unit 53 creates a three-dimensional image to be displayed in the display area 620.

[0218] S113: The communication unit 51 determines whether it has received a request from the communication unit 91 to register the shooting date and time, based on the fact that the "Register" button 680 has been pressed on the communication terminal 9a using the cursor c1 or the like.

[0219] S114: If a registration request is received (S113; YES), the processing unit 54 registers the requested shooting date and time in the movement history management DB 5004 (see Figure 18). In this case, the processing unit 54 registers (stores) a flag corresponding to the registration request in the registration field shown in Figure 18. In this case, the date and time when the registration request was received is registered as the shooting and sound collection date and time.

[0220] As explained in Figures 18B,B, the movement history management DB5004 stores and manages the date and time of shooting and sound collection, shooting location, field of view information, and audio text associated with each content ID.

[0221] Furthermore, as explained in Figure 15, the virtual room management DB5002 stores and manages content IDs and content URLs (information on the storage location of wide-field image and sound content data) in association with each other.

[0222] This associates the registered shooting and sound collection dates and times with the shooting location, field of view information, wide-field image, and audio (speech) text.

[0223] Here, since the field of view information is information for identifying a predetermined area that represents a predetermined area image displayed on the communication terminal 9, the shooting location, the predetermined area image, and the sound (voice) text are associated with the registered shooting and sound collection date and time.

[0224] S115: As shown in Figure 27, the creation unit 53 superimposes an icon 622a of the virtual camera IC1, based on the shooting position and field of view associated with the shooting and sound collection date and time registered in process S114, onto the position corresponding to the shooting position within the display area 620. Figure 27 is a diagram showing a screen displaying the corresponding predetermined area, the icon of the virtual camera IC1, and a line diagram indicating the line of sight of the virtual camera IC1 on the display area 620.

[0225] S116: Furthermore, the creation unit 53 superimposes a corresponding predetermined area 621a corresponding to the field of view of the virtual camera IC1 within the display area 620, as shown in Figure 27. Figure 27 shows the state in which the icon of the virtual camera IC11, the corresponding display area, and the line diagram indicating the line of sight are superimposed on the communication terminal 9a. This corresponding predetermined area 621a is an area corresponding to the predetermined area related to the predetermined area image displayed in the display area 610. In Figure 27, the corresponding predetermined area 621a is represented by a dashed line, which is an example of the display mode of the corresponding predetermined area 621a. This display mode includes dashed and solid lines of a predetermined color, and the thickness of the dashed and solid lines may differ from that of other parts.

[0226] Furthermore, the creation unit 53 superimposes the icon 622a of the virtual camera IC1 within the display area 620 at a position corresponding to the shooting position of the shooting device 10, and superimposes the icon 622a so that it faces the center point CP1a of the corresponding predetermined area 621a. Note that the icon 622a is an example of a schematic diagram (image) of the virtual camera IC1. In addition to the icon, this schematic diagram also includes words such as "camera" or diagrams containing these words.

[0227] Furthermore, the creation unit 53 superimposes a diagram 623a within the display area 620 that shows the line of sight from the icon 622a to the center point CP1a of the corresponding predetermined area 621a. This diagram 623a may be a solid line or a dashed line, and may be displayed with a different thickness or color from the other lines.

[0228] Figure 28 shows the state in the communication terminal 9a where a change has occurred in the object being photographed, while maintaining the same shooting position and field of view. As shown in Figure 28, in the display area 610 where the current photographed image is displayed, item M11, which was initially located on the left and not displayed, is moved and becomes visible within the display area 610.

[0229] Figure 29 shows another state in the communication terminal 9a where the icon for the virtual camera IC1, the corresponding display area, and the line of sight diagram are superimposed. As shown in Figure 29, when the shooting position of the shooting device 10 changes, the contents displayed in the display areas 610 and 620 also change. Here, the corresponding predetermined area 621α, the icon for the virtual camera IC1 622α, and the line of sight diagram 623α of the virtual camera IC1 are shown.

[0230] S117: After processing S116, or if no registration request was received in processing S113 (NO), the creation unit 53 determines whether the position or field of view of the virtual camera IC1 in the display area 610 has been changed (see Figure 6(c)). For example, when participant A performs an operation to change a predetermined area image within the display area 610 shown in Figure 27, the reception unit 92 receives the changed field of view, and the communication unit 91 transmits field of view information indicating the changed field of view to the communication control device 5. As a result, when the communication unit 51 receives the field of view information indicating the changed field of view, the creation unit 53 determines that the field of view of the virtual camera IC1 has been changed. If the position or field of view of the virtual camera IC1 has been changed (S117; YES), the process returns to S111. In this way, as processes S111 to S116 are repeated, each time the "Register" button 680 is pressed at a different time, the communication unit 51 receives multiple registration requests, and the processing unit 54 registers flags corresponding to the multiple registration requests in the movement history management DB 5004. Also, as shown in Figure 30, the creation unit 53 superimposes icons 622c, 622b, 622a, etc. of multiple virtual camera IC1s in the display area 620, as well as movement path images Rcb, Rba indicating the movement path of the imaging device 10.

[0231] Figure 30 shows the movement paths of the icons for each virtual camera IC1, which indicate the position and field of view of the shooting device at the time of registration of the shooting date and time, as well as a superimposed diagram showing the latest corresponding display area and line of sight. In Figure 30, icon 622b is superimposed when the "Register" button 680 was pressed one step before icon 622a, and icon 622c is superimposed when the "Register" button 680 was pressed one step before icon 622b.

[0232] If the shooting positions of multiple virtual camera IC1 are the same, the creation unit 53 may superimpose the icon 622 of one virtual camera IC1 within the display area 620, and superimpose a list of multiple shooting dates and times associated with this icon 622 of one virtual camera IC1.

[0233] S118: If the reception unit 92 accepts that participant A has finished displaying screen 600 (YES), such as by pressing the close button 609, the display of screen 600 will end. On the other hand, if the reception unit 92 does not accept that the display of screen 600 has ended (NO), the process returns to S117, and the processes in Figures 24 and 25 continue until the transmission of captured image data by the communication control device 5 is completed in process S14 shown in Figure 19.

[0234] <Examples of registration variations> In the example above, a predetermined region image from the currently being captured image was displayed, but this is not the only example. For example, as shown in Figure 31, multiple predetermined region images from the same captured image that have already been recorded, but corresponding to different shooting dates and times, may be displayed. Figure 31 is a diagram showing the screen when multiple recorded predetermined region images are played back on the communication terminal 9a.

[0235] In this case, the creation unit 53 creates a screen 600 that includes a display area 610a which displays a first predetermined region image, which is a predetermined region of a first shooting date and time in a captured image aligned with an undisplayed three-dimensional image, and a display area 610b which displays a second predetermined region image, which is a predetermined region of a second shooting date and time in the same captured image. In Figure 31, the predetermined region image displayed in display area 610b is taken at a later time than the predetermined region image displayed in display area 610a. Therefore, as shown in display area 610b, it can be seen that the position of the item M11 has moved.

[0236] Furthermore, in Figure 31, display areas 650a and 650b are displayed below each of the display areas 610a and 610b, respectively.

[0237] Display area 650a displays audio text based on audio recorded within a predetermined time period, including the elapsed playback time of the image currently displayed in display area 610a. For example, in Figure 18B, if the shooting and audio recording date and time (elapsed playback time) is "2023 / 11 / 11 10:00:05", the creation unit 53 uses the registered audio text "We plan to move the new box to the left." recorded during the period when there is no change in the shooting position and field of view information (2023 / 11 / 11 10:00:04 to 2023 / 11 / 11 10:00:06) to create a screen 600 to be displayed in display area 650a.

[0238] Similarly, the display area 650b displays audio text based on audio recorded within a predetermined time period, including the elapsed playback time of the image currently displayed in the display area 610b. For example, in Figure 18C, if the shooting and sound collection date and time (elapsed playback time) is "2023 / 12 / 11 10:00:08", the creation unit 53 uses the registered audio text "I'll reinforce the area under the pillar." which was recorded during the period when there was no change in the shooting position and field of view information (2023 / 12 / 11 10:00:07 to 2023 / 12 / 11 10:00:09) to create the screen 600 to be displayed in the display area 650b.

[0239] Additionally, "Register" buttons 680a and 680b are displayed at the bottom of each display area 610a and 610b. The "Register" buttons 680a and 680b serve the same purpose as the "Register" button 680.

[0240] Furthermore, when the communication unit 51 receives a registration request to superimpose an icon of a virtual camera IC1, whose shooting area is the field of view of the displayed first predetermined area image, at the first shooting position at the first shooting date and time, or an icon of a virtual camera, whose shooting area is the field of view of the displayed second predetermined area image, at the second shooting position at the second shooting date and time, within the three-dimensional image display area that displays at least a part of the three-dimensional image, the processing unit 54 registers a flag corresponding to the registration request in the registration column of the movement history management DB 5004 (see Figures 18A, B, C).

[0241] <Details of screen display processing (after registration)> Next, using Figures 32 to 35, we will explain the processing in the case of a fixed viewing content mode, where the display is fixed to a predetermined communication terminal (in this case, communication terminal 9a). Specifically, among the processes S13 to S15, we will explain the screen display processing after registration, specifically process S14. Note that in Figures 34 and 35, components with the same reference numerals as in Figures 26 to 32 have the same role or function. Here, when participant A operates the communication terminal 9a, the reception unit 92 receives the operation, and the communication unit 91 transmits operation information indicating the operation content to the communication control device 5. As a result, the communication unit 51, which has the role of an acquisition unit, acquires the operation information, and the creation unit 53 creates the screen 601 based on the operation content indicated by the operation information. Then, when the communication unit 51 transmits the screen 601 data to the communication terminal 9a, the communication unit 91 of the communication terminal 9a receives the screen 601 data, and the display control unit 94 displays the screen 601 on the display 507 of the communication terminal 9a. In this case, the communication control device 5 is an example of an information processing device.

[0242] Next, the process by which the communication control device 5 creates a screen 601 to be displayed on the display 507 of the communication terminal 9a will be described. Note that processes S13 and S15 are the same processes as process S14, with only the terminal being displayed being different, so the explanation of processes S13 and S15 will be omitted. Figures 32 and 33 are flowcharts showing the processes executed by the communication control device in the screen display process after the registration of the shooting date and time.

[0243] First, the creation unit 53 of the communication control device 5 creates the screen 601 that the communication terminal 9a will display, as shown in Figure 34. Figure 34 is a diagram showing the screen displayed when the current predetermined area image is displayed on the communication terminal 9a after the date and time of shooting has been registered. The screen 601 includes current time information 603, display area 610 (image display area, an example of the first image display area), display area 620 (an example of the three-dimensional image area), and display area 630 (an example of the second image display area). In Figure 34, the display areas 610, 620, and 630 are displayed simultaneously on the screen 601 at the same size, but they may be displayed simultaneously at different sizes, or one of them may be displayed by switching based on participant A's selection. Alternatively, the display areas 610, 620, and 630 may be displayed on each of the three displays of the screen 601.

[0244] The display area 610 displays a predetermined region image, which is a predetermined area in the current wide-field image (an example of the current captured image) obtained by photographing the object with the imaging device 10 (see Figures 6(a) and (b)).

[0245] The display area 620 includes a three-dimensional model representing the shape of an object (table, column, etc.) in three dimensions, and also displays part or all of the three-dimensional image in which coordinates (an example of second coordinates) are associated with coordinates (an example of first coordinates) in the wide-field image.

[0246] The display area 630 displays a predetermined region image, which is a predetermined region in a past wide-field image (an example of a past captured image) obtained by photographing an object with the imaging device 10 (see Figures 6(a) and (b)).

[0247] In this case, if the captured image is not a curved image like a wide-field image, the display areas 610 and 630 display a predetermined area image in which the same area as the capture area of ​​the captured image is the predetermined area.

[0248] Here as well, the processing unit 54 aligns the display areas 610, 620, and 630 as described above. Then, the creation unit 53 creates a screen 601 that includes a display area 610 that displays a predetermined area image which is a predetermined area in the current wide-field image, a display area 620 that displays at least a part of a three-dimensional image in which a second position (second coordinate) is associated with a first position (first coordinate) in the wide-field image, and a display area 630 that displays a predetermined area image which is a predetermined area in a past wide-field image in which a third position (third coordinate) is associated with a first position (first coordinate).

[0249] Additionally, a "Register" button 680 is displayed on screen 601. The "Register" button 680 performs the same role or function as the "Register" button 680 shown in Figure 26.

[0250] Furthermore, the content displayed on screen 601 will be explained in detail below. Virtual camera IC1 is used to identify predetermined regions related to each predetermined region image to be displayed in display regions 610 and 630 (see Figure 8), while virtual camera IC2 is used to identify the three-dimensional image to be displayed in display region 620.

[0251] S211: First, the creation unit 53 of the communication control device 5 identifies the current predetermined area image to be displayed in the display area 610, as shown in Figure 34, based on the field of view information (in this case, the virtual field of view of the virtual camera IC1 which is set in advance) from the wide-field image of the content data received by the communication unit 51.

[0252] S212: The processing unit 54 aligns the position of the virtual camera IC2 with the shooting position associated with the wide-field image (content data) obtained by shooting with the shooting device 10, and also aligns the virtual field of view of the virtual camera IC2 (an example of a second field of view) with the virtual field of view of the virtual camera IC1 (an example of a first field of view) that is set in advance. As a result, the creation unit 53 creates a three-dimensional image to be displayed in the display area 620.

[0253] Also, as shown in FIG. 34, when the "Register" button 680 is pressed, the same processes as the above-described processes S113 to S116 are performed. As a result, the creation unit 53 can superimpose, within the display area 620, at a position corresponding to the current shooting position, a diagram 623A showing the line of sight of an icon 622A of the virtual camera IC1 (an example of an image showing the first virtual camera) with respect to the center point CP1A of the corresponding predetermined area 621A, which is the corresponding predetermined area.

[0254] S213: Further, as shown in FIG. 34, the creation unit 53 superimposes, within the display area 620, icons 622a to 622c of the virtual camera IC1 (an example of a past shooting position image, an example of an image showing the second virtual camera) based on the shooting position and the shooting angle associated with the registered shooting date and time. When there are a plurality of such icons, the creation unit 53 also superimposes the movement path images Rba and Rcb.

[0255] When the shooting positions of each of the plurality of virtual cameras IC1 are the same, the creation unit 53 superimposes an icon 622 of one virtual camera IC1 within the display area 620, and associates with this icon 622 of one virtual camera IC1, and superimposes a list of a plurality of shooting dates and times.

[0256] S214: The creation unit 53 determines whether there is an icon of the virtual camera IC1 related to a past shooting position within a predetermined range from the current shooting position. When assuming an absolute position, the predetermined range is, for example, 3 m.

[0257] When there is an icon of the virtual camera IC1 related to a past shooting position within a predetermined range from the current shooting position (S214: YES), the creation unit 53 identifies a predetermined icon at the past shooting position that is closest to the current shooting position (the position of the icon 622A of the virtual camera IC1) within the predetermined range among the icons 622a, 622b, and 622c of the virtual camera IC1 at a plurality of positions. Here, the icon 622a is identified.

[0258] S216: If there is no icon of the virtual camera IC1 related to the past shooting position within a predetermined range from the current shooting position for one party (S214: NO), the communication unit 51 determines whether an instruction for selection (or designation) by the participant A of the icon of a predetermined past virtual camera IC1 has been obtained among the icons of the plurality of past virtual camera IC1s. If the communication unit 51 does not obtain the selection (S216; NO), the communication unit 51 repeats the determination of obtaining this selection.

[0259] As described in process S213, when the creation unit 53 superimposes a list of a plurality of shooting dates and times in association with one icon 622 of the virtual camera IC1 within the display area 620, the communication unit 51 determines whether an instruction for selection (or designation) by the participant A of a predetermined past shooting date and time has been obtained among the plurality of past shooting dates and times.

[0260] S217: After process S215, or when the selection is obtained in process S216 (S216; YES), the creation unit 53 changes the display form of the icon of the predetermined past virtual camera IC1 (here, the icon 622a) as shown in FIG. 35. FIG. 35 is a diagram showing a screen when a past predetermined area image is additionally displayed on the communication terminal 9a after registration of the shooting date and time. The change in the display form is, for example, a change in the shape, pattern, color, etc. of the icon.

[0261] Also, the creation unit 53 displays the icon 622a on the upper surface of the icon 622A as shown in FIG. 35. In FIGS. 34 and 35, when the creation unit 53 superimposes the icon 622a at the past shooting position and the icon 622A at the current shooting position exactly at the same position, the size of either one of the icons may be changed or they may be superimposed with a slight shift in position.

[0262] Furthermore, the creation unit 53 adds a display area 631 for past audio to the screen 601 for displaying audio text corresponding to past predetermined area images to be displayed in the display area 630. In this case, the creation unit 53 includes in the display area 631 the audio text that is managed in the movement history management DB 5004 in association with the shooting position and field of view information that identifies past predetermined area images to be displayed in the display area 630. The creation unit 53 also includes the shooting and sound collection date and time (registered "✓" portion) associated with the audio text in the movement history management DB 5004, and reads the corresponding name and device image (or user image) from the user / device management DB 5001 based on the device ID (or user ID) managed in the movement history management DB 5004, and includes them near the shooting and sound collection date and time.

[0263] In Figure 35, the display area 631 shows a display field 635a containing user A's user image, the date and time of capture and sound collection, user A's name and user ID, and a display field 635b containing the voice text. Furthermore, the display area 631 displays, in chronological order, a display field 636a containing the device image of the capture device 10 (here, capture device α), the date and time of capture and sound collection, the name and device ID of the capture device 10, and a display field 636b containing the voice text.

[0264] Furthermore, since the user ID can identify the communication terminal 7, it is also an example of identification information for identifying the sound input / output control unit 75 (sound collection unit) of the communication terminal 7. Also, since the user ID can identify the communication terminal 9, it is also an example of identification information for identifying the sound input / output control unit 95 (sound collection unit) of the communication terminal 9. In addition, since the device ID can identify the imaging device 10, it is also an example of identification information for identifying the sound collection unit 17 of the imaging device 10.

[0265] S218: The creation unit 53 superimposes within the display area 620, in the same manner as in processing S116, a corresponding predetermined area 621a related to a predetermined virtual camera IC1 icon from the past, and a diagram 623a showing the line of sight of the virtual camera IC1 icon 622a to the center point CP1a of this corresponding predetermined area 621a.

[0266] S219: Then, as shown in Figure 35, the creation unit 53 adds and includes a display area 630 (an example of a second captured image display area) to the screen 601, which displays a past predetermined region image (an example of a second predetermined region image) corresponding to the icon 622a of the virtual camera IC1. Furthermore, the creation unit 53 adds and includes a display area 635 to the screen 601, which displays voice text (text information) generated from the audio recorded when the past predetermined region image to be displayed was captured. In this case, as shown in Figure 18B, the creation unit 53 creates the screen 601 to display the voice text in the display area 635 for each time (period) S4 to S6 in which both the shooting position and field of view of the predetermined region image to be displayed in the display area 630 have not changed. That is, the past predetermined region image displayed in the display area 630 is a part of the captured image at a past point in time (for example, a predetermined 1 second), but the past voice text displayed in the display area 635 may be the content spoken over several seconds (see S4 in Figure 18).

[0267] Furthermore, the creation unit 53 includes the aforementioned "Register" button 680 within the screen 601, as well as newly includes "Duplicate Audio" buttons 681 and 683. The "Duplicate Audio" button 681 is for duplicating the current audio text output together with the current predetermined area image displayed in the display area 610, associating it with the past predetermined area image displayed in the display area 630. The "Duplicate Audio" button 683 is for duplicating the past audio text output together with the past predetermined area image displayed in the display area 630, associating it with the current predetermined area image displayed in the display area 610. Note that the creation unit 53 associates the past audio text displayed in the top-level display field 635b of the display area 631 with the current predetermined area image displayed in the display area 610, but it may also associate past audio text displayed in a display field other than the top-level display field (for example, display field 636b) of the display area 631 with the current predetermined area image displayed in the display area 610.

[0268] S220: The processing unit 54 determines, via the acquisition unit (reception unit 92, communication unit 51), whether there is an instruction to duplicate past voice text in association with the current predetermined area image when the "voice duplication" button 683 is pressed.

[0269] S221: If there is an instruction to duplicate (S220; YES), the processing unit 54 associates the past audio text (text information) displayed in the display area 635 with the date and time of capture and sound collection of the current predetermined area image (predetermined area) displayed in the display area 610, and registers (duplicates) it in the "Audio Text" field of the Movement History Management DB 5004 (see Figure 18C). This makes it easier for participant A to understand the difference between the past and present not only by the predetermined area image but also by the audio text, as the audio text duplicated in the Movement History Management DB (Figure 18C) (for example, "Where is the new box?") and the audio text registered in the Movement History Management DB (Figure 18C) (for example, "The new box has been moved to the left") are displayed consecutively when the current predetermined area image is displayed later.

[0270] S222: If there is no instruction to duplicate after processing S221 or in processing S220 (S220; NO), the processing unit 54 determines, via the acquisition unit (reception unit 92, communication unit 51), whether there is an instruction to duplicate the voice text generated from the current voice in association with a past predetermined area image, by pressing the "Voice Duplication" button 681.

[0271] S223: If there is an instruction to duplicate the current audio to the past (S222; YES), the processing unit 54 registers the current audio text (text information) generated by the text generation unit 56 from the current audio data (content data) being output, in association with the date and time of the capture and sound collection of the past predetermined area image (predetermined area) displayed in the display area 630, in the "Audio Text" field of the movement history management DB 5004 (see Figure 18B). This makes it easier for participant A to understand the difference between the time of recording and the future, not only by the predetermined area image but also by the audio text, as the past audio text from the time of recording (for example, "Where are the new boxes?") and the audio text generated from the audio at the time of current shooting and recording (for example, "The area under the pillar looks unstable." and "I'll reinforce the area under the pillar.") are displayed consecutively when the predetermined area image is redisplayed later.

[0272] Note that the order of processes S220 and S221 and processes S222 and S223 may be reversed.

[0273] S224: After processing S223, or if there is no instruction to duplicate in processing S223 (S222; NO), the creation unit 53 will terminate the display of screen 601 if the reception unit 92 has received confirmation that participant A has finished displaying screen 601, such as by pressing the close button 609 (YES). On the other hand, if the termination of the display of screen 601 is not accepted (NO), the process returns to S214, and the processes in Figures 32 and 33 continue until the transmission of captured image data by the communication control device 5 is completed in processing S14 shown in Figure 19.

[0274] (Modified version of Figure 35) Here, a modified version of Figure 35 will be explained using Figure 36. Figure 36 shows a modified screen of the communication terminal 9a when past predetermined area images are added after the date and time of shooting are registered. Note that in Figure 35, a display area 631 for displaying past voice text was displayed, whereas in the modified version of Figure 36, in addition to the display area 631, a display area 632 for displaying voice text related to the past voice text is displayed. In this case, the processing unit 54 searches the virtual room management DB 5002 for voice text generated from voice in a virtual room with a recording date (e.g., 2023.10.11) that is different from the recording date (e.g., 2023.11.11) of the voice related to the past voice text displayed in Figure 36, using each keyword (e.g., "new box") in the past voice text (e.g., "Where is the new box?") as a search key, and reads, for example, "Please decide the location of the new box." as "past past voice text".

[0275] As shown in Figure 36, the creation unit 53 includes a display area 632 for past related audio, and in the display area 632, it creates a screen 601 that includes a display field 637a containing the device image of the shooting device 10 (here, shooting device α), the date and time of shooting and sound collection, the name and device ID of the shooting device 10, and a display field 637b for the audio text.

[0276] Furthermore, regarding related audio from the past, the processing unit 54 may also retrieve the corresponding audio text by searching the movement history management DB 5004 (see Figure 18A) for a different recording (audio recording) date with the same three-dimensional image ID, using the shooting location (or shooting location and field of view information) associated with the audio text currently displayed in the display area 631 of the past audio as a search key, and include it in the display field 632.

[0277] [Display of meeting minutes] By managing each data shown in the above-mentioned FIGS. 14, 15, and 18A, B, C, the creation unit 53 can create a minutes screen 602 as shown in FIGS. 37 to 40. FIG. 37 is a diagram showing a screen when displaying the minutes of voice text and images on the communication terminal 9a.

[0278] The screen 602 shown in FIG. 37 includes a display area 602a for voice text on the left side that serves as the basis of the minutes, a display area 602b for a predetermined area image corresponding to the voice text, and a display area 602c for a three-dimensional image corresponding to the predetermined area image.

[0279] The creation unit 53 includes the voice text managed in the movement history management DB5004 (see FIG. 18C) in chronological order in the display area 602a. The display mode regarding the voice text is the same as the display area 631 in FIG. 35.

[0280] Also, the creation unit 53 includes, in the display area 602b, the display area 601a of a past predetermined area image based on the angle-of-view information associated with the voice text in the movement history management DB5004 (see FIG. 18C). Further, the creation unit 53 includes, in the display area 602c, the display area 620a of a past three-dimensional image related to the display area 610a, similar to FIG. 27. That is, the predetermined area image in the display area 610a and the predetermined area image in the display area 620a are aligned as described above by the processing unit 54.

[0281] Note that the past three-dimensional image in the display area 620a includes an image of the article M21 corresponding to the article M11 in the display area 610a (an identification image for identifying that it is a different part). In this case, the creation unit 53 creates an image of the article M21 based on the position and shape of the article M11 in the past predetermined area image corresponding to the past three-dimensional image, and superimposes it on the past three-dimensional image.

[0282] As a result, viewers of the meeting minutes screen 603 (for example, User A) can view the automatically generated audio text, along with the corresponding designated area image and three-dimensional image, together (simultaneously), allowing them to grasp the contents of the meeting minutes in detail.

[0283] (Variation 1) Next, we will explain modification 1 of Figure 37 using Figure 38. Figure 38 is a diagram showing the screen when past predetermined area images and three-dimensional images are displayed along with the meeting minutes on the communication terminal 9a.

[0284] In this case, the processing unit 54 searches the virtual room management DB 5002 (see Figure 15) based on the virtual room ID (e.g., "101r") of the virtual room on the recording date for which each audio text displayed on the screen 602 shown in Figure 37 was generated, and reads the content ID (e.g., "c091") associated with another virtual room ID (e.g., "091r") with the same three-dimensional image ID (e.g., "v001"). The processing unit 54 then reads a predetermined shooting position from among the shooting positions managed in the movement history management DB 5004 (see Figure 18B) managed by this content ID that is the same as or near the shooting position of the predetermined region image currently displayed in the display area 610a (the difference in each parameter of the shooting position is within a threshold), and reads predetermined field of view information associated with this predetermined shooting position. The creation unit 53 then includes the past predetermined region image in the display area 630a within the display area 602b, as shown in Figure 38, based on the predetermined content data (image data), predetermined shooting position, and predetermined field of view information.

[0285] Furthermore, the processing unit 54 may extract past predetermined region images from the above-mentioned predetermined content data (image data) that have a similarity of a predetermined value or higher by performing a similar image search based on the predetermined region image displayed in the display area 610a.

[0286] Furthermore, the creation unit 53 includes a past three-dimensional image in the display area 640a within the display area 602c that is aligned with a past predetermined area image in the display area 630a. The past predetermined area image in the display area 630a is an example of a related image to the predetermined area image in the display area 610a.

[0287] As a result, viewers of the meeting minutes screen 603 (for example, User A) can simultaneously view the automatically generated audio text (602a), the corresponding predetermined region image (610a) and three-dimensional image (620a), and even past predetermined region images (630a) and past three-dimensional images (640a), thereby gaining a more detailed understanding of the meeting minutes.

[0288] (Modification 2) Next, we will explain a modified example of Figure 37, Part 2, using Figures 39 and 40.

[0289] Figure 39 is a modified version of the functional configuration diagram described in Figure 13. The communication control device 5 further includes an image generation unit 57, and an image generation model 5005 is constructed in the storage unit 5000 of the communication control device 5. The other configurations are the same as in Figure 13.

[0290] The image generation unit 57 is an example of an image generation means, and is implemented by instructions from the CPU 501 shown in Figure 12, and generates image information based on the image generation model 5005.

[0291] Image generation model 5005 is a trained machine learning model (generative AI) that generates images from text data, or from text data and images. Image generation model 5005 is trained using training data that includes, for example, text data and image data. This training data includes, for example, training data (text data, or text data and image data) as input and image data as ground truth data for output. For example, machine learning is performed so that the images generated by image generation model 5005 when given training data as input approach the ground truth images included in the training data.

[0292] Figure 40 shows the screen displayed on the communication terminal 9a when an image is generated and added to the meeting minutes based on a predetermined area image, a three-dimensional image, and audio comments.

[0293] The image generation unit 57, for example, uses the image generation model 5005 to generate a predetermined region image according to the voice text (for example, "I'll reinforce the bottom of the pillar.") and a predetermined region image corresponding to this voice text. In the display area 630b of Figure 40, the image generation unit 57 includes an identification image (in this case, an image of item M12) to identify the difference. Then, the creation unit 53 includes the predetermined region image generated by the image generation unit 57 in the display area 630b within the display area 602b.

[0294] Furthermore, the image generation unit 57 generates a three-dimensional image corresponding to the display area 630b for the display area 640b within the display area 602c. In the display area 640b of Figure 40, the image generation unit 57 includes an identification image (in this case, an image of item M22) to identify the difference. The creation unit 53 then includes the three-dimensional image generated by the image generation unit 57 into the display area 640b within the display area 602c. Note that the predetermined area image of the display area 630b is an example of a related image to the predetermined area image of the display area 610a.

[0295] As a result, viewers of the meeting minutes screen 603 (for example, User A) can view the automatically generated audio text, the corresponding predetermined region image and 3D image, and any further generated predetermined region image and 3D image together (simultaneously), thereby gaining a more detailed understanding of the meeting minutes' contents.

[0296] [Main effects of the embodiment] As described above, according to this embodiment, the processing unit 54 associates a second position in a three-dimensional image including a three-dimensional region corresponding to the object with a first position in the captured image obtained by photographing the object, and the creation unit 53 (or creation units 73, 93) creates a screen 600 that includes a display area 610 that displays a predetermined region image which is a predetermined region in the captured image obtained by photographing the object, and a display area 620 that displays at least a part of the three-dimensional image to which the second position in the captured image has been associated with the first position in the captured image by the processing unit 54.

[0297] This allows users (participant A, etc.) to understand what was photographed and where in the designated area image by viewing screen 600. Furthermore, even if the back of the object cannot be photographed, users can understand the condition of the back of the object by viewing screen 600. Therefore, by creating images that complement the images obtained through photography, it is possible to improve user convenience.

[0298] Furthermore, by associating the date and time of shooting (and date and time of sound collection), shooting location, field of view during display, and audio text using the movement history management DB 5004 shown in Figures 18B and C, if participant A presses the "Register" button 680 while a predetermined area image is being displayed in the display area 610, the processing unit 54 can register the date and time of shooting and sound collection, shooting location, field of view, and audio text during display in the movement history management DB 5004 (registering a flag in the registration field).

[0299] Furthermore, as shown in Figure 27, the creation unit 53 can superimpose an icon 622a of a virtual camera IC1, etc., which has a field of view of the displayed predetermined area image as its shooting area, at a position corresponding to the shooting position within the display area 620.

[0300] Furthermore, as shown in Figure 34, the creation unit 53 can display the virtual camera IC1 icon 622a, etc., at the shooting position in the previously registered display area 620. When participant A, etc. presses the icon 622a, as shown in Figure 35, the creation unit 53 creates a screen 601 in the display area 630 that displays a predetermined area image showing the shooting date and time corresponding to the icon 622a and the field of view corresponding to the icon 622a, as well as the audio text for this shooting date and time. As a result, participant A, etc. can use the position of the virtual camera IC1 icon as a reference to identify the shooting date and time and field of view in past shooting images, and thereby detect the predetermined area image at the desired shooting time from past shooting images.

[0301] 〔supplement〕 Although embodiments have been described above, the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0302] (1) Each function of each embodiment described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" in this specification includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, and devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each function described above.

[0303] (2) A (non-temporary) recording medium such as a DVD-ROM on which any of the above programs are stored may be provided domestically or internationally as a program product.

[0304] (3) There may be multiple CPUs 111, 301, and 501, which are hardware processors.

[0305] (4) In the above embodiment, the voice text generated by the text generation unit 56 is registered in Figure 18, but it is not limited to this. For example, without using the text generation unit 56, the reception unit 92 may receive text entered by the user (participant A, etc.), the data of this text may be transmitted to the communication unit 51 of the communication control device 5 via the communication unit 91, and the processing unit 54 may register the text in the movement history management DB 5004. The record registered in this way is a record of the same shooting and sound collection date and time as the date and time the communication unit 51 received the text data.

[0306] [Additional notes] The contents of this embodiment can be described as follows. Note that the symbols in parentheses are those used in this embodiment, but the present invention is not intended to be limited thereto.

[0307] [Additional Note 1] An information processing device (5 / 7 / 9) that can communicate with the imaging device (10), A creation unit that creates a screen including a captured image display area that displays a predetermined region image, which is a predetermined region in the captured image obtained by photographing an object with the aforementioned photographing device, and a three-dimensional image display area that displays at least a part of a three-dimensional image aligned with the captured image. A processing unit performs a process to associate each text information from among a plurality of text information generated from audio data including sound collected at different times by a sound collection unit (75 / 95) in the shooting device or a display terminal (7 / 9) that can communicate with the shooting device, with each predetermined region image from among a plurality of predetermined region images obtained at different times, with respect to the shooting position of the captured image. An information processing device having [Explanation of Symbols]

[0308] 1. Communication System (An Example of an Information Processing System) 3. Relay device 5. Communication control device (an example of an information processing device) 7. Communication terminals (examples of display terminals and information processing devices) 9,9a,9b Communication terminals (examples of display terminals and information processing devices) 10. Imaging device 51 Communications Department (An example of an acquisition department) 52 Reception Department 53 Creation Section 54 Processing Unit 55 Certification Department 56 Text generation unit 71 Communications Department 72 Reception Department (An example of an acquisition department) 73 Creation Section 74 Display Control Unit 75. Sound Input / Output Control Unit (Example of a Sound Collection Unit) 78 Connection part 91 Communications Department 92 Reception Department (An example of an acquisition department) 94 Display Control Unit 95. Sound Input / Output Control Unit (Example of a Sound Collection Unit) 98 Connection section 600 screens 610 Display area (Example of captured image display area, Example of the first captured image display area) 620 Display area (an example of a three-dimensional image display area) 630 Display area (an example of the second captured image display area) 631 Display area (Example of a text display area) 507 Display (Example of a display unit) 5001 User / Device Management Database (An example of a user / device management unit) 5002 Virtual Room Management Database (An example of a virtual room management unit) 5003 Three-dimensional image management database (an example of a three-dimensional image management unit) 5004 Movement History Management Database (An example of a movement history management unit) [Prior art documents] [Patent Documents]

[0309] [Patent Document 1] Japanese Patent Publication No. 2023-140923

Claims

1. An information processing device capable of communicating with a photographic device, A creation unit that creates a screen including a captured image display area that displays a predetermined region image, which is a predetermined region in the captured image obtained by photographing an object with the aforementioned photographing device, and a three-dimensional image display area that displays at least a part of a three-dimensional image aligned with the captured image. A processing unit performs a process to associate each of the text pieces of text generated from audio data including sound collected at different times by a sound collection unit in the shooting device or a display terminal capable of communicating with the shooting device with each of the predetermined region images obtained at different times, with respect to the shooting position of the aforementioned captured image. An information processing device having

2. The information processing apparatus according to claim 1, wherein the aforementioned audio data includes audio collected by the sound collection unit of the imaging device.

3. The aforementioned audio data includes audio collected by multiple sound collection units, The information processing apparatus according to claim 1 or 2, wherein the processing unit performs a process of associating identification information for identifying each of the plurality of sound collection units with each of the plurality of text information.

4. The information processing apparatus according to claim 1, wherein the creation unit creates a screen for displaying an image corresponding to the shooting position at a position corresponding to the shooting position in the three-dimensional image, and for displaying predetermined text information from among the plurality of predetermined text information corresponding to the shooting position and predetermined region images corresponding to the predetermined text information from among the plurality of predetermined region images.

5. The information processing apparatus according to claim 4, wherein the creation unit creates the screen for displaying the plurality of text information in chronological order.

6. A screen creation method performed by a computer capable of communicating with a camera, The aforementioned computer, A creation process for creating a screen that includes a captured image display area that displays a predetermined region image, which is a predetermined region in the captured image obtained by photographing an object with the aforementioned photographing device, and a three-dimensional image display area that displays at least a part of a three-dimensional image aligned with the captured image. A registration process is performed to associate each text information from among a plurality of text information generated from audio data including audio collected at different times by the sound collection unit of the shooting device or a display terminal capable of communicating with the shooting device with each predetermined region image from among a plurality of predetermined region images obtained at different times, with respect to the shooting position of the aforementioned captured image. How to create a screen to execute this.

7. A program for causing a computer to perform the method described in claim 6.

8. An information processing system having an information processing device that can communicate with an imaging device, A creation unit that creates a screen including a captured image display area that displays a predetermined region image, which is a predetermined region in the captured image obtained by photographing an object with the aforementioned photographing device, and a three-dimensional image display area that displays at least a part of a three-dimensional image aligned with the captured image. A processing unit performs a process to associate each of the text pieces of text generated from audio data including sound collected at different times by a sound collection unit in the shooting device or a display terminal capable of communicating with the shooting device with each of the predetermined region images obtained at different times, with respect to the shooting position of the aforementioned captured image. An information processing system that has

Images