Display terminal, information processing system, communication system, display method, information processing method, communication method, and program

Abstract

The aim is to play back and display recorded wide-field video footage with as little loss of realism as possible when viewing it later. [Solution] The present disclosure is a display terminal for displaying images, characterized by having a reception unit 32 that receives a designation for a predetermined region still image, which is a predetermined region in a wide-view still image having a wide field of view angle, and a display control unit 33 that causes the display unit to play back and display the predetermined region video, which is the predetermined region in a wide-view video that includes the wide-view still image, based on viewpoint information for identifying the predetermined region relating to the designated predetermined region still image.

Description

Technical Field

[0001] The present disclosure relates to a display terminal, an information processing system, a communication system, a display method, an information processing method, a communication method, and a program.

Background Art

[0002] There is a communication system that transmits images and voices in real time from one base to one or more other bases and enables remote communication using images and voices between users located at remote locations. As this image, for example, a wide-angle image (hereinafter referred to as "wide-field image") having a wide viewing angle captured in a wide imaging range including a 360-degree image (also referred to as an omnidirectional image, an all-round image, or a full-surround image) in which a 360-degree entire circumference including a portion that cannot be fully confirmed in a normal viewing angle is imaged is known. A user can browse another predetermined area after the change in the wide-field image by operating a communication terminal to change a virtual viewpoint for a predetermined area in the wide-field image displayed on the display screen of the communication terminal.

[0003] Also, conventionally, it is possible to record a wide-field video (referred to as "wide-field video") obtained by shooting with a 360-degree omnidirectional camera (see Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there are times when, in addition to wide-field video footage capturing the situation at a site or other location, one may want to later review specific scenes from the video in still images rather than the wide-field video footage itself. For example, one might first review a scene in still images and then check the video for more detailed information, or conversely, one might review the detailed situation in video first and then review the scene in still images. In other words, users will switch between wide-field video footage and still images depending on their purpose. In such cases, because wide-field images have a wide field of view and allow for a virtual change of viewpoint, it becomes difficult to grasp the relationship between video and still images.

[0005] This invention has been made in view of the above-mentioned problems, and aims to make it easier to understand the relationship between wide-field video and still images contained within the video. [Means for solving the problem]

[0006] The invention according to claim 1 is a display terminal for displaying an image, comprising: a reception unit that receives a designation for a predetermined region still image, which is a predetermined region in a wide-view still image having a wide field of view; and a display control unit that causes the display unit to play and display a predetermined region video, which is the predetermined region in a wide-view video including the wide-view still image, based on viewpoint information for identifying the predetermined region relating to the designated predetermined region still image. [Effects of the Invention]

[0007] As described above, the present invention has the effect of making it easier to understand the relationship between wide-field video and still images contained within the video. [Brief explanation of the drawing]

[0008] [Figure 1] This figure illustrates an example of remote communication using wide-field imaging. [Figure 2] This is an example of a schematic diagram of a communication system configuration. [Figure 3]This is an example of a hardware configuration diagram for an imaging device. [Figure 4] This is an example of a hardware configuration diagram for a communication terminal and an information processing system. [Figure 5] (a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a top view of the imaging device. [Figure 6] This is a diagram illustrating the use of the imaging device. [Figure 7] (a) is a hemispherical image (front) captured by the imaging device, (b) is a hemispherical image (back) captured by the imaging device, and (c) is an image represented using equirectangular projection. [Figure 8] (a) A conceptual diagram showing the state of covering the sphere with an equirectangular projection image, and (b) A diagram showing a full-sphere image. [Figure 9] This diagram shows the positions of a virtual camera and a predetermined region when a 360-degree image is treated as a three-dimensional sphere. [Figure 10] (a) is a stereoscopic perspective view of Figure 5, and (b) is a diagram showing the state in which an image of a predetermined area is displayed on the display. [Figure 11] This figure shows the relationship between information from a predetermined region and an image of a predetermined region T. [Figure 12] This is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. [Figure 13] This is an example of a functional configuration diagram for a communication system. [Figure 14] This is a conceptual diagram showing the image management information stored in the image management information storage unit. [Figure 15] This is a conceptual diagram showing the virtual room information stored in the virtual room information storage unit and the tenant information stored in the tenant information storage unit. [Figure 16] This is a conceptual diagram showing the playback-related information stored in the playback-related information storage unit. [Figure 17] (a) is a diagram showing an example of the room entry screen. (b) is a diagram showing an example of the image viewing screen displayed by the communication terminal when a user enters a virtual room. [Figure 18]This is an example of a sequence diagram explaining the process by which a user (or communication terminal) enters a virtual room. [Figure 19] This is a diagram showing an example of a device registration screen displayed on a communication terminal. [Figure 20] (a) is an example of a diagram showing an imaging device registration dialog. (b) is an example of a diagram showing a two-dimensional code screen. [Figure 21] This is a diagram showing an example of a VR goggles registration screen displayed when the VR goggles registration button is pressed. [Figure 22] This is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room. [Figure 23] This is a diagram showing an example of a virtual room association screen (part 2). [Figure 24] This is a diagram showing an example of a virtual room association screen (part 3). [Figure 25] This is a diagram showing an example of a wide-angle image transmission start / stop dialog displayed on a communication terminal. [Figure 26] This is an example of a sequence diagram showing the procedure by which a user registers an imaging device in a virtual room. [Figure 27] This is an example of a sequence diagram explaining the flow in which a wide-angle image is shared. [Figure 28] This is a diagram showing an example of an image viewing screen displayed on a communication terminal. [Figure 29] This is a diagram showing an example of an image viewing screen on a communication terminal where a download button is displayed when the imaging button is pressed. [Figure 30] (a) is an example of a diagram showing an image viewing screen displayed before a user presses the download button. (b) is an example of a diagram showing an image viewing screen displayed after a user presses the download button. [Figure 31] This is an example of a diagram showing an image viewing screen on which a thumbnail image is displayed. [Figure 32] This is a diagram showing an example of an image viewing screen when there are three image columns. [Figure 33]This is an example of a sequence diagram illustrating the process by which a user requests imaging from an imaging device via a communication terminal when sharing wide-field images. [Figure 34] This diagram shows a modified version of a sequence diagram illustrating the process by which a user requests imaging from an imaging device via a communication terminal when sharing wide-field images. [Figure 35] This figure shows an example of a virtual room list screen displayed by any user who has connected a communication terminal to external storage. [Figure 36] This figure shows an example of a wide-field image viewing screen. [Figure 37] This is an example of a sequence diagram illustrating the process by which a user views wide-field images stored on external storage. [Figure 38] This is a sequence diagram showing the processing in frame extraction mode. [Figure 39] This is a conceptual diagram of the frame extraction mode screen. [Figure 40] This is a conceptual diagram of the frame extraction mode screen. [Figure 41] This flowchart explains the main processes of a communication terminal when it is in still image playback mode. [Figure 42] This is a sequence diagram showing the detailed processing of the still image playback mode. [Figure 43] This is a conceptual diagram of a screen that plays back a predetermined range of motion from a predetermined range of still images in still image playback mode. [Figure 44] This is a conceptual diagram of the screen that returns from a predetermined area of ​​video to a predetermined area of ​​still image playback in still image playback mode. [Figure 45] This flowchart explains the main processes of a communication terminal when it is in video playback mode. [Figure 46] This is a sequence diagram showing the detailed processing of the video playback mode. [Figure 47] This is a conceptual diagram of a screen that plays back a predetermined area of ​​still images from a predetermined area of ​​video in video playback mode. [Figure 48] This is a conceptual diagram of the screen that returns from displaying a still image in a predetermined area to displaying a video in a predetermined area during video playback mode. [Figure 49]This diagram illustrates an example of remote communication where a communication system is applied to telemedicine. [Figure 50] This figure shows an example of a virtual room mapping screen used to associate imaging equipment with a virtual room in the case of telemedicine. [Figure 51] This is a conceptual diagram showing the relationship between wide-field video, video of a specified area, wide-field still image, still image of a specified area, information of a specified area (viewpoint information), and playback elapsed time. [Modes for carrying out the invention]

[0009] The following describes an example of an embodiment for carrying out the present invention: an information processing system and an image transmission method performed by the information processing system.

[0010] <An example of remote communication> Figure 1 illustrates an example of remote communication using wide-field imaging. In Figure 1, three locations A to C communicate via an information processing system 50. The number of locations is just an example; there could be two locations, four or more locations, or any number of locations.

[0011] Location A is, for example, a construction site. Locations B and C can be anywhere, such as an office, as long as they are locations where wide-field images can be transmitted. Location A is equipped with an imaging device 10 that can generate wide-field images, such as a 360-degree spherical image, or wide-angle wide-field images of the surroundings, such as 180 to 360 degrees in the vertical or horizontal direction, by capturing images of subjects, etc. Hereafter, such wide-angle images will simply be referred to as "wide-field images." Locations A to C are equipped with various communication terminals 30A to 30C for viewing wide-field images. Hereafter, any of the communication terminals 30A to 30C will be referred to as "communication terminal 30."

[0012] At the construction site, various construction works are being carried out by workers at each location. The imaging device captures the entire construction site to generate a wide-field image that shows the whole site. If there is a construction or work that users at each location A to C want to focus on, users a to c at each location A to C can arbitrarily change their virtual viewpoint to check it. In this case, the viewpoint is the center position or range of a predetermined area displayed on a display screen or similar screen within the entire wide-field image.

[0013] The imaging device 10 can be mounted on a tripod 86 or on an arm 85 via a gimbal 87. A relay device (in Figure 1, the communication terminal 30A also serves as the relay device) is installed at the construction site, and the communication terminal 30A transmits the wide-field images received from the imaging device 10 via wired or wireless connection to the information processing system 50. The communication terminal 30A can also serve as a terminal for viewing the wide-field images. A camera 9 is connected to (or built into) the communication terminal 30, and images of a normal field of view (or 360-degree images) captured by the camera 9 can also be transmitted to the information processing system 50. In addition, user a (for example, a worker) can wear smart glasses 88, and images of a normal field of view (or 360-degree images) generated by the smart glasses 88 may be transmitted to the information processing system 50. Smart glasses 88 are information terminals that display information acquired via the internet on a display while maintaining a field of view. Smart glasses 88 may be placed at any location.

[0014] On the other hand, at site B, a PC (Personal Computer) or smartphone is provided as an example of a communication terminal 30B. Furthermore, any device capable of communicating with the information processing system 50 can be a communication terminal 30B; other devices such as tablet terminals, PDAs (Personal Digital Assistants), electronic whiteboards, or projectors are also acceptable. A camera may be built into or connected to the communication terminal 30B.

[0015] Furthermore, at base C, an example of a communication terminal 30C is a PC, smartphone, VR (Virtual Reality) goggles 89, etc. In Figure 1, a camera 8 is built into or connected to the communication terminal 30C. The VR goggles 89 is an information terminal that displays an artificial world on a computer or a 360-degree image according to the direction of movement of the head or body. The VR goggles 89 may also be a smartphone with VR goggles attached, such as Hakosco (a VR scope made of cardboard with plastic lenses that can be easily assembled and a smartphone inserted to enjoy VR). The camera 8 may be for wide-angle or normal field of view. In addition, the communication terminal 30C may be any device that can communicate with the information processing system 50, such as a tablet terminal, PDA, electronic whiteboard, or projector. The VR goggles 89 may be placed at any base.

[0016] In this embodiment, the imaging device 10 and each communication terminal 30 manage communication within a communication group called a "virtual room." The imaging device 10 is associated with the virtual room, and the communication terminal 30 (the user operating the communication terminal 30) enters this virtual room and receives the wide-field image transmitted by the imaging device 10, allowing the user to view the wide-field image. Smart glasses 88 and VR goggles 89 can also be associated with the virtual room. Cameras 8 and 9 enter the virtual room in the same way as the communication terminal 30.

[0017] Users a to c at locations A to C can arbitrarily change the viewpoint of the wide-field image on their respective communication terminals 30. Therefore, since users a to c are viewing the wide-field image in real time, there is a possibility that they are each seeing a different viewpoint, which could make communication difficult. To address this, this embodiment allows the virtual viewpoint information set on the communication terminal 30 of any given location to be shared with the communication terminals 30 of other locations. The general outline of this sharing is explained below. For illustrative purposes, the case where the viewpoint specified by user b at location B is shared with users a and c at locations A and C is shown.

[0018] (1) Communication terminals 30A to 30C share a wide-field image (an example of a first wide-field image) generated by the imaging device 10. When user b requests a wide-field image while viewing from an arbitrary viewpoint on communication terminal 30B, communication terminal 30B (an example of a first communication terminal) transmits viewpoint information and an imaging request to the information processing system 50.

[0019] (2) In response to the imaging request, the information processing system 50 specifies viewpoint information and requests the imaging device 10 to take an image (either a still image or a video).

[0020] (3) The imaging device 10 takes an image in response to an imaging request and stores the wide-field image (an example of a second wide-field image) and viewpoint information in association with a URL (an example of storage location information; in Figure 1, the storage location on the storage 90) notified by the information processing system 50. The wide-field image stored on the storage 90 can be downloaded and displayed by any communication terminal 30.

[0021] (4) The information processing system 50 sends a URL to the communication terminal 30B.

[0022] (5) The information processing system 50 also automatically or at the request of user b transmits the URL to the communication terminals 30A and 30C (an example of a second communication terminal) that are in the same virtual room.

[0023] (6) The communication terminals 30A and 30C access a URL to receive viewpoint information and a wide-field image, and set the viewpoint of the wide-field image identified by the viewpoint information to align with the center of the image field and display it. However, it is not necessary to align the viewpoint perfectly with the center; the viewpoint may be set to be included in the area near the center of the image field and displayed.

[0024] The same applies when sharing the perspective of user a at location A with users at locations B and C, and when sharing the perspective of a user at location C with users at locations A and B.

[0025] As described above, even when a wide-field image is distributed, the communication system 1a of this embodiment allows for the sharing of viewpoint information without requiring instructions to move the viewpoint relative to the wide-field image generated by imaging so that a predetermined area of ​​focus at each location is displayed, thus facilitating communication of user intentions.

[0026] In addition, in (3), the imaging device 10 can transmit the wide-field image itself to the information processing system 50, and in (4), the information processing system 50 can transmit the wide-field image to the communication terminals 30A to 30C.

[0027] Furthermore, while Figure 1 illustrates an example where the imaging device 10 is deployed at a construction site, this embodiment can also be applied to VR education, event distribution, remote customer service, telemedicine, and the like. In VR education, the imaging device 10 is deployed at a site location such as a laboratory, allowing students to view blackboards, equipment, samples, experimental results, etc., from a remote location by arbitrarily changing their viewpoint. In event distribution, the imaging device 10 is deployed at the event venue, allowing event participants, such as spectators, to view the venue online from a remote location by arbitrarily changing their viewpoint. The venue includes images of performers, contestants, presenters, products and exhibits explained at the event, images of materials explained at the event, and images of the venue's condition. The event venue can be indoors or outdoors and includes venues for sports, concerts, plays, etc. In remote customer service, for example, when applied to customer service at a travel agency, the imaging device 10 is deployed at the travel destination, allowing customers to review their itinerary from a remote location by arbitrarily changing their viewpoint. In telemedicine, the imaging device 10 is placed in a medical setting such as an operating room, and doctors, students, medical equipment personnel, etc., can remotely change their viewpoint and view the actions of doctors and nurses performing medical procedures in the medical setting, the placement of equipment, the patient's condition, vital signs, etc.

[0028] The locations where images are captured are not limited to these; any space where users (viewers) at the viewing location have a need to understand the situation at a remote location, such as schools, factories, warehouses, construction sites, server rooms, or stores, is acceptable.

[0029] <About Terminology> A tenant refers to a group of users associated with a contract unit when contracting to receive an image distribution service from a service provider (in this embodiment, an information processing system). This includes contracting companies, organizations, individuals, etc. Therefore, a tenant can also be referred to as a user group. While a user may belong to a tenant as an example, an individual user may also subscribe to the service. In addition to users, imaging devices, virtual rooms, etc., are registered within a tenant (user group).

[0030] A base of operations refers to a place that serves as the foundation for activities. In this embodiment, a conference room is used as an example of a base of operations. A conference room is a room set up primarily for the purpose of holding meetings. Meetings are also called gatherings, meetings, discussions, assemblies, and other similar terms.

[0031] A device refers to an apparatus other than a general-purpose communication terminal 30 such as a PC or smartphone, and is an imaging device or a wide-field image viewing device. In this embodiment, examples include an imaging device 10, smart glasses 88, and VR goggles 89.

[0032] Viewpoint information refers to parameter information that specifies which predetermined region of a wide-field image to display on the display screen. In this embodiment, the "radial radius," "polar angle," and "azimuth angle" corresponding to the center of the wide-field image displayed on the display screen are described as examples of viewpoint information, but it may also be specified by other parameter information such as the coordinates of diagonal vertices.

[0033] A wide-field image refers to an image that has a wide viewing angle that is wider than the display range that can be displayed at once on the display screen (the area where the wide-field image is displayed) in a given display method. A wide-field image has a display range of up to 360 degrees (or 180 degrees) vertically and 360 degrees horizontally, but any image that has a wide viewing angle that is wider than the display range that can be displayed at once on the display is also included as a wide-field image, even if it is less than 360 degrees vertically or horizontally. Images with a display range of 160 degrees or more vertically and horizontally are also included as wide-field images. For example, an image with a display range wider than the range that a human can see at once is also included as a wide-field image. In addition, depending on the display method, even an image that can be displayed at once on the display screen can be included as a wide-field image if it has a wide viewing angle when switched to or changed to a given display method. In this embodiment, an equirectangular spherical image is used as an example of a wide-field image, but omnidirectional images, hemispherical images, 3D panoramic images, 2D panoramic images, and VR images are also included in the wide-field image category. The wide-field image may also be an image in the form of cube mapping, dome master, etc. Furthermore, the spherical image may be in a format other than equirectangular.

[0034] Images captured with a normal field of view are not wide-field images, but in this embodiment, they will be described as non-wide-field images (planar images).

[0035] A communication group is a collection of users from whom wide-field images are shared (distributed). In a normal space, a communication group is described as a virtual room, meaning that when each user enters the same room, each user can share the wide-field image. Here, "virtual" means that it is realized through information processing via a network.

[0036] Users at each location communicate remotely with each other. Remote communication is a meeting held at a remote location. A meeting is when people gather together for consultation, discussion, or other purposes. Meetings can take various forms, such as customer service, conferences, gatherings, meetings, study groups, classes, seminars, and presentations. It does not necessarily have to be two-way communication. Therefore, a virtual room may also be called a virtual conference room.

[0037] <Example of a communication system configuration> Figure 2 is an example of a schematic diagram of the configuration of communication system 1a. Figure 1 is an example of applying communication system 1a from Figure 2 to remote communication with the field. Communication system 1a is a system that transmits and receives wide-field images and normal-angle images captured by the imaging device 10 bidirectionally between multiple locations. By displaying images distributed from one location at other locations, users at other locations can view the images. As an example of a wide-field image, a 360-degree spherical image captured by the imaging device 10 is distributed. Communication system 1a can, for example, allow a wide-field image captured at a predetermined location to be viewed at other locations in remote locations.

[0038] As shown in Figure 2, in the communication system 1a, the imaging device 10, communication terminal 30A, and information processing system 50 located at base A, and communication terminals 30B and 30C located at each of the multiple bases (bases B and C), are connected in a way that enables communication.

[0039] If the imaging device 10 has a communication function that allows it to connect directly to the communication network N, then a communication terminal 30A acting as a relay device (e.g., a router) is not necessary. In this case, the communication terminal 30A connects to the communication network N without the imaging device 10. However, if the communication terminal 30A is located at site A, the communication terminal 30A also functions as a relay device, allowing user a to view wide-field images in the same way as with communication terminals 30B and 30C. Furthermore, imaging devices 10 may be located at other sites besides site A, or multiple imaging devices 10 may be located at site A.

[0040] Each communication terminal 30 and information processing system 50 can communicate via the communication network N. The communication network N is constructed using the Internet, mobile communication networks, LANs (Local Area Networks), etc. The communication network N may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0041] The imaging device 10 is a digital camera capable of capturing images of subjects, landscapes, etc., to obtain two hemispherical images, which then generate a single full-sphere image. The wide-field image obtained by the imaging device 10 may be a video, a still image, or both. The captured image may also include sound along with the image.

[0042] The communication terminal 30 is a computer such as a PC used by users at each location. The communication terminal 30 displays images obtained by imaging at its own location, wide-field images (still images or videos) distributed from other locations, and images with a normal field of view. For example, the communication terminal 30 acquires wide-field images captured by the imaging device 10 via the communication network N. The communication terminal 30 also has image processing software such as OpenGL ES installed, and can display images based on viewpoint information that identifies a part of the wide-field image. Note that OpenGL ES is just one example of image processing software, and other software may be used. Furthermore, even if image processing software is not installed, image processing may be performed by software received from an external source, or image display may be performed by receiving the results of image processing performed by external software. In other words, the communication terminal 30 can display a predetermined area of ​​a wide-field image.

[0043] The communication terminal 30 can arbitrarily change the viewpoint relative to the display range of a wide-field image in response to user operation. The communication terminal 30 can change the field of view (predetermined area) based on viewpoint information corresponding to the changed viewpoint by moving the virtual viewpoint in response to user operation input (including key input, drag, scrolling, etc.) to a touch panel, directional buttons, mouse, keyboard, touchpad, etc. Furthermore, if the communication terminal 30 is a communication terminal worn by a user, such as VR goggles, the communication terminal 30 may detect the change in posture information of the communication terminal 30 in response to changes in the movements of the user wearing it, and change the field of view (predetermined area) based on viewpoint information corresponding to the changed viewpoint by moving the virtual viewpoint in response to the detected posture information.

[0044] The communication terminal 30A distributes wide-field images acquired from the imaging device 10 via a wired cable, such as a USB (Universal Serial Bus) cable, connected to the input / output I / F 116 described later, to communication terminals 30 at other locations via the information processing system 50. The connection between the imaging device 10 and the communication terminal 30A may be a wireless connection using short-range wireless communication, etc., instead of a wired connection using a wired cable. Multiple communication terminals 30A may be located at location A.

[0045] In some cases, a user at location A may wear smart glasses 88, and the smart glasses 88 may connect to the communication network N. In this case, the images captured by the smart glasses 88 are transmitted to the information processing system 50 via the communication network N, and the information processing system 50 can distribute them to the communication terminals 30 at each location.

[0046] Communication terminal 30B is located at site B where user b is located, and communication terminal 30C is located at site C where user c is located. Multiple communication terminals 30B and 30C may be located at sites B and C. Furthermore, communication terminals 30B and 30C may be carried around by user b and user c, respectively.

[0047] Furthermore, the communication terminals 30A to 30C at locations A to C can have cameras 8 and 9 built-in or externally attached as imaging units, and the communication terminals 30A to 30C can distribute images of their own location captured by their own cameras 8 and 9 to other locations. In addition, any devices may be placed at locations A to C.

[0048] The arrangement of each terminal and device (communication terminal 30 and imaging device) and the user shown in Figure 2 is just an example, and other examples may be used. Furthermore, the communication terminal 30 is not limited to a PC, but may be, for example, a tablet terminal, smartphone, PDA, wearable device (including smart glasses and VR goggles), PJ (Projector), electronic whiteboard (a whiteboard with electronic blackboard functionality that enables mutual communication), or autonomous mobile robot. The communication terminal 30 can be any computer on which a dedicated application for a web browser or image distribution service runs.

[0049] Furthermore, if the imaging device 10 has a display, it may be configured to display images distributed from other locations.

[0050] The information processing system 50 has one or more information processing devices. The information processing system 50 manages and controls communication between the imaging devices 10 and communication terminals 30 at each location, and manages the wide-field images that are transmitted and received. The information processing system 50 provides a platform that allows users to utilize the functions necessary to provide an image distribution service that distributes wide-field images. This platform may be made available to service providers, such as individuals or companies that wish to provide image distribution services, by contract. Hereinafter, in order to distinguish them from tenants that receive image distribution services, service providers who provide image distribution services to users using the contracted platform will be referred to as platform subscribers.

[0051] Therefore, the information processing system 50 may, as a platform, expose an API (Application Programming Interface), allowing platform subscribers to use this API to provide various image distribution services. Platform subscribers only need to develop software such as applications that handle screens displayed by the communication terminal 30 and API calls, and do not need to develop functions provided by APIs such as image distribution from scratch.

[0052] The information processing system 50 may be constructed using a single computer, or it may be constructed using multiple computers, each part (function or means) of which is arbitrarily assigned. Furthermore, all or part of the functions of the information processing system 50 may be server computers located in a cloud environment, or server computers located in an on-premises environment.

[0053] Storage 90 is a memory device that stores data such as wide-field images. Storage 90 may be an external storage device separate from the information processing system 50 (it may be storage located on the cloud or storage located on-premises), or it may be storage included in the information processing system 50.

[0054] <Example Hardware Configuration> Next, the hardware configuration of each device or terminal in the image communication system according to this embodiment will be described using Figures 3 and 4. Note that the hardware configuration shown in Figures 3 and 4 may have components added or removed as needed.

[0055] <<Hardware configuration of the imaging device>> First, the hardware configuration of the imaging device 10 will be explained using Figure 3. Figure 3 is a diagram showing an example of the hardware configuration of the imaging device 10. In the following, the imaging device 10 will be described as a 360-degree (omnidirectional) imaging device using two image sensors, but it may have one image sensor or two or more. Furthermore, it is not necessarily required to be a device dedicated to omnidirectional imaging; an omnidirectional imaging unit can be attached to a regular digital camera or smartphone, etc., to provide essentially the same functionality as the imaging device 10.

[0056] As shown in Figure 3, the imaging device 10 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output interface 116, a short-range communication circuit 117, an antenna 117a for the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network interface 121.

[0057] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lens 102 unless otherwise specified) each having a field of view of 180° or more for forming a hemispherical image, and two image sensors 103a and 103b provided corresponding to each lens. The image sensors 103a and 103b have an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the lenses 102a and 102b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers for setting various commands or parameters necessary for the operation of the image sensors. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example; it may have only one, or three or more.

[0058] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, operation unit 115, input / output I / F 116, short-range communication circuit 117, electronic compass 118, gyro sensor 119, acceleration sensor 120, and network I / F 121, etc.

[0059] The image processing unit 104 receives image data output from image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create data for an equirectangular projection image (an example of a wide-field image), which will be described later.

[0060] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 acting as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0061] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 10, there may also be functions to display a preview or video on a display (for example, the display of an external terminal such as a smartphone that communicates with the imaging device 10 via a short-range communication circuit 117). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0062] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0063] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress of processing by the image processing unit 104 and data of completed equirectangular projection images.

[0064] The control unit 115 is a collective term for various operation buttons, a power switch, a shutter button, and a touch panel that combines display and operation functions. The user inputs various imaging modes, imaging conditions, etc., by operating the control unit 115.

[0065] The input / output interface (I / F) 116 is a general term for interface circuits (such as USB I / F) to external media such as SD cards or personal computers. The I / F 116 can be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded to external media via the I / F 116, or transmitted to an external terminal (device) via the I / F 116 as needed.

[0066] The short-range communication circuit 117 communicates with an external terminal (device) via an antenna 117a provided on the imaging device 10 using a short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi. The short-range communication circuit 117 can transmit equirectangular projection image data to the external terminal (device).

[0067] The electronic compass 118 calculates the orientation of the imaging device 10 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) in accordance with Exif, and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data.

[0068] The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) associated with the movement of the imaging device 10. The changes in angle are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

[0069] The acceleration sensor 120 is a sensor that detects acceleration in three axes. The imaging device 10 calculates its own orientation (angle relative to the direction of gravity) based on the acceleration detected by the acceleration sensor 120. By providing the acceleration sensor 120, the accuracy of image correction in the imaging device 10 is improved.

[0070] Network I / F 121 is an interface for data communication using a communication network N such as the Internet via a router or the like. Furthermore, the hardware configuration of the imaging device 10 is not limited to what is shown here, and any configuration that can realize the functional configuration of the imaging device 10 is acceptable. In addition, at least a part of the above hardware configuration may reside on the communication network N.

[0071] <<Hardware configuration of the communication terminal>> Figure 4 shows an example of the hardware configuration of the communication terminal 30 and the information processing system 50. First, the communication terminal 30 will be described. The hardware configuration of the communication terminal 30 is indicated by codes in the 300 series. The communication terminal 30 is built by a computer and, as shown in Figure 4, is equipped with a CPU 301, ROM 302, RAM 303, HDD (Hard Disk) 304, HDD controller 305, display 306, external device connection I / F 308, network I / F 309, bus line 310, keyboard 311, pointing device 312, DVD-RW (Digital Versatile Disk Rewritable) drive 314, media I / F 316, sound input / output I / F 317, microphone 318, speaker 319, short-range communication circuit 320, and camera 321.

[0072] Of these components, the CPU 301 controls the overall operation of the communication terminal 30. The ROM 302 stores programs used to drive the CPU 301, such as the IPL. The RAM 303 is used as the work area for the CPU 301. The HDD 304 stores various data, such as programs and data. The HDD controller 305 controls the reading or writing of various data to the HDD 304 according to the control of the CPU 301.

[0073] The display 306 displays various information such as cursors, menus, windows, characters, or images. The display 306 may be a touch panel display equipped with input means. Note that the display 306 is just one example of a display unit. The display unit includes not only the display provided on the communication terminal 30, but also an external display of the communication terminal 30, an external display of another communication terminal attached to the communication terminal 30, or a screen projected by a projector (including the object to be displayed by projection mapping).

[0074] The external device connection interface 308 is an interface for connecting various external devices. In this case, external devices include, for example, USB memory or printers. The network interface 309 is an interface for data communication using the communication network N. The bus line 310 is an address bus or data bus, etc., for electrically connecting each component, such as the CPU 301 shown in Figure 4. Note that the HDD 304 and HDD controller 305 are examples of storage for storing programs and data, etc., and may be SSDs (Solid State Drives) or SSD controllers, respectively.

[0075] The keyboard 311 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting or executing various instructions, selecting processing targets, or moving a cursor, etc. Note that the input means may not be limited to the keyboard 311 and the pointing device 312, but may also be a touch panel or an audio input device, etc. The DVD-RW drive 314 controls the reading or writing of various data to the DVD-RW 313, which is an example of a removable recording medium. Note that the DVD-RW 313 may be a DVD-R or a Blu-ray® Disc, etc. The media I / F 316 controls the reading or writing (storage) of data to the recording medium 315, such as flash memory. The microphone 318 is a type of built-in sound collection means for inputting sound. The sound input / output I / F 317 is a circuit that processes the input and output of sound signals between the microphone 318 and the speaker 319 according to the control of the CPU 301. The short-range communication circuit 320 is a communication circuit for communicating with an external terminal (device) using short-range wireless communication technology such as NFC, Bluetooth (registered trademark), or Wi-Fi. The camera 321 is a type of built-in imaging means that captures an image of a subject and obtains image data. Note that the microphone 318, speaker 319, and camera 321 may be external devices rather than being built into the communication terminal 30.

[0076] Furthermore, the hardware configuration of the communication terminal 30 is not limited to that shown herein; any configuration that can realize the functional configuration of the communication terminal 30 is acceptable. In addition, at least a part of the above hardware configuration may reside on the network N.

[0077] <<Hardware configuration of the information processing system>> As shown in Figure 4, each hardware component of the information processing system 50 is indicated by a 500-series code in parentheses. The information processing system 50 is built using a computer and has a configuration similar to that of the communication terminal 30, as shown in Figure 4; therefore, a description of each hardware component is omitted.

[0078] Furthermore, the hardware configuration of the information processing system 50 is not limited to those shown herein; any configuration that can realize the functional configuration of the information processing system 50 is acceptable. In addition, at least a part of the above hardware configuration may reside on a network.

[0079] Furthermore, each of the above programs may be distributed as an installable or executable file recorded on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc (registered trademark), SD card, USB memory, etc. The recording media can also be provided domestically or internationally as a program product. For example, the communication terminal 30 realizes the image display method according to the present invention when the program according to the present invention is executed.

[0080] <Regarding wide-field images and viewpoint information> The following section explains how to generate wide-field images (spherical images) using Figures 5 to 12.

[0081] First, the external appearance of the imaging device 10 will be described using Figure 5. The imaging device 10 is a digital camera for obtaining captured images that will form the basis of a 360° spherical image. Figure 5(a) is a left side view of the imaging device, Figure 5(b) is a front view of the imaging device, and Figure 5(c) is a top view of the imaging device. This external view is merely one example of the imaging device 10, and other external appearances are also possible.

[0082] As shown in Figure 5(a), the imaging device 10 is small enough to be held in one hand, but this shape is merely an example, and other shapes are also possible. Also, as shown in Figures 5(a), 5(b), and 5(c), the imaging device 10 has an image sensor 103a on the front side and an image sensor 103b on the back side at the top. These image sensors 103a and 103b are used in conjunction with optical components (for example, lenses 102a and 102b described later) that can capture hemispherical images (angle of view of 180° or more). Also, as shown in Figure 5(b), an operating section 115 such as a shutter button is provided on the side of the imaging device 10 opposite to the front side. As mentioned above, there may be only one image sensor, or there may be three or more.

[0083] Next, the usage of the imaging device 10 will be explained using Figure 6. Figure 6 is an illustrative diagram of the imaging device in use. As shown in Figure 6, the imaging device 10 is used, for example, to image a subject around the imaging device 10. In this case, two hemispherical images can be obtained by imaging the subject around the imaging device 10 using the image sensors 103a and 103b shown in Figure 5, respectively.

[0084] Next, using Figures 7 and 8, we will outline the process from the image captured by the imaging device 10 to the creation of a full-sphere image. Figure 7(a) shows the hemispherical image (front side) captured by the imaging device, Figure 7(b) shows the hemispherical image (rear side) captured by the imaging device, and Figure 7(c) shows the image represented by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Figure 8(a) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 8(b) shows the full-sphere image. The "equirectangular projection image" is an equirectangular full-sphere image as an example of the wide-field image described above.

[0085] As shown in Figure 7(a), the image obtained by the image sensor 103a is a curved hemispherical image (front side) by the lens 102a described later. Similarly, as shown in Figure 7(b), the image obtained by the image sensor 103b is a curved hemispherical image (rear side) by the lens 102b described later. The imaging device 10 then combines the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) to create an equirectangular projection image EC as shown in Figure 7(c).

[0086] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to overlay an equirectangular projection image EC so that it covers the sphere, as shown in Figure 8(a), and creates a full-sphere image (full-sphere panoramic image) CE as shown in Figure 8(b). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. OpenGL ES is merely one example of software that performs image processing, and the full-sphere image CE may be created using other software. The full-sphere image CE may be a still image or a video. Here, the imaging device 10 has been described as generating a full-sphere image, but the information processing system 50 or the communication terminal 30 may perform similar image processing or some of the image processing steps.

[0087] As described above, the 360-degree spherical image CE is an image pasted to cover a sphere, which can cause discomfort to the human eye. Therefore, by displaying a predetermined area T (hereinafter referred to as the "predetermined area image") of the 360-degree spherical image CE as a flat image with less curvature in the imaging device 10 or the communication terminal 30, a display that does not cause discomfort to the human eye can be achieved. This will be explained with reference to Figures 9 to 10.

[0088] Figure 9 shows the positions of the virtual camera and a predetermined region when the 360-degree spherical image is treated as a three-dimensional sphere. The virtual camera IC corresponds to the position of the virtual viewpoint of the user viewing the 360-degree spherical image CE, which is displayed as a three-dimensional sphere. In Figure 10, (a) is a stereoscopic perspective view of Figure 9, (b) is a diagram showing the predetermined region image in the state of (a) displayed on the screen, (c) is a diagram showing the predetermined region after changing the viewpoint of the virtual camera IC in (a), and (d) is a diagram showing the predetermined region image in the state of (c) displayed on the screen.

[0089] If the resulting spherical image CE is considered a solid sphere CS, then, as shown in Figure 9, the virtual camera IC is located inside the spherical image CE. A predetermined region T in the spherical image CE is the imaging region of the virtual camera IC and is identified by predetermined region information indicating the imaging direction and field of view of the virtual camera IC in the three-dimensional virtual space containing the spherical image CE. Furthermore, zooming in on the predetermined region T can also be represented by moving the virtual camera IC closer to or further away from the spherical image CE. The predetermined region image Q is an image of the predetermined region T in the spherical image CE. Therefore, the predetermined region T can be identified by the field of view α and the distance f from the virtual camera IC to the spherical image CE (see Figure 11).

[0090] Then, the predetermined region image Q shown in Figure 10(a) is displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in Figure 10(b). The image shown in Figure 10(b) is a predetermined region image represented by the predetermined region information that has been initially set (default). The following explanation will use the imaging direction (ea, aa) and field of view (α) of the virtual camera IC. Note that the predetermined region T may be represented by the position coordinates (X, Y, Z) of the imaging area of ​​the virtual camera IC, which is the predetermined region T, rather than by the field of view α and distance f.

[0091] Furthermore, when the virtual viewpoint of the virtual camera IC is moved (also called "changed") to the right (left side in the drawing) from the state shown in Figure 10(a) to the state shown in Figure 10(c), a predetermined region T in the 360-degree image CE is moved to a predetermined region T' accordingly, and the predetermined region image Q displayed on the predetermined display is changed to a predetermined region image Q'. As a result, the image shown in Figure 10(b) is changed to the image shown in Figure 10(d) on the display.

[0092] In this embodiment, wide-field images relating to still images may be referred to as "wide-field still images," and predetermined-area images relating to still images may be referred to as "predetermined-area still images." Similarly, wide-field images relating to videos may be referred to as "wide-field videos," and predetermined-area images relating to videos may be referred to as "predetermined-area videos." Furthermore, "wide-field images" include both still images and videos. "Predetermined-area images" include both still images and videos. Moreover, unless otherwise specified, "shooting" includes both saving still images and recording videos.

[0093] Here, using Figure 51, the relationship between wide-field video, predetermined-area video, wide-field still image, predetermined-area still image, predetermined-area information (viewpoint information), and playback elapsed time will be explained. Figure 51 is a conceptual diagram showing the relationship between wide-field video, predetermined-area video, wide-field still image, predetermined-area still image, predetermined-area information (viewpoint information), and playback elapsed time. Note that "wide-field image" is a concept that includes both wide-field video and wide-field still image.

[0094] As shown in Figure 51, a wide-field video is composed of multiple frames (wide-field still images). That is, a wide-field video includes wide-field still images, and a wide-field still image includes a wide-field video. Furthermore, a predetermined region in a wide-field video is a predetermined region video, and a predetermined region in a wide-field still image is a predetermined region still image. A predetermined region video is identified based on each predetermined region information (viewpoint information). A predetermined region still image is identified based on the predetermined region information (viewpoint information) at the time of extraction. In Figure 51, it is shown that the wide-field still image is a frame extracted at the playback elapsed time te in the wide-field video.

[0095] Next, we will explain the relationship between the predetermined region information and the image of the predetermined region T using Figure 11. Figure 11 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 11, "ea" is the elevation angle, "aa" is the azimuth angle, and "α" is the field of view (Angle). That is, the orientation of the virtual camera IC is changed so that the point of fixation of the virtual camera IC, indicated by the imaging direction (ea,aa), becomes the center point CP(x,y) of the predetermined region T, which is the imaging area of ​​the virtual camera IC. As shown in Figure 11, the center point CP(x,y) when the diagonal field of view of the predetermined region T, represented by the field of view α of the virtual camera IC, is α becomes the parameter ((x,y)) of the predetermined region information. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. f is the distance from the virtual camera IC to the center point CP(x,y). L is the distance between any vertex of a given region T and the center point CP(x,y) (2L is the diagonal). In Figure 11, the following trigonometric function generally holds:

[0096]

number

[0097] Figure 12 shows the relationship described in Figure 11 as a point in a three-dimensional Euclidean space using spherical coordinates. Here, the position coordinates of the center point CP shown in Figure 11, expressed in spherical polar coordinates, are (r,θ,φ). (r,θ,φ) are the radial, polar angle, and azimuth angle, respectively. The radial r is the distance from the origin of the three-dimensional virtual space containing the full-sphere image to the center point CP, and is therefore equal to the distance f shown in Figure 11. Figure 12 is a diagram illustrating these relationships. Hereafter, the position coordinates (r,θ,φ) of the virtual camera IC will be used as an example of viewpoint information for explanation. The viewpoint information only needs to be parameter information that can identify a predetermined region T (predetermined region image Q) that is displayed as an image of the imaging area of ​​the virtual camera IC on the predetermined display shown in Figure 10, as described above, and includes the coordinates of the diagonal vertices of the predetermined region T. Alternatively, the information indicating the field of view α of the virtual camera IC and the information indicating the center point CP(x,y) described in Figure 11 may also be considered as viewpoint information. Furthermore, the information indicating the field of view α and azimuth angle aa of the virtual camera IC described in Figure 11 may also be considered viewpoint information. In addition, viewpoint information may include not only position coordinate information in spherical coordinates, but also position coordinate information in orthogonal coordinates and coordinate difference values ​​from the initially set (default) predetermined area information. Furthermore, viewpoint information may also include information other than coordinate information, such as angles and distances, as shown in Figure 11. In addition, although the center point of the predetermined area T is used as the reference in Figures 11 and 12, the predetermined area T may also be identified by parameter information based on any of the vertices of the predetermined area T. Note that in the above explanation of viewpoint information, the case where the wide-field image is a 360-degree spherical image was used as an example, but in the case of other wide-field images, the information that identifies the predetermined area T in that wide-field image will be the viewpoint information. Furthermore, viewpoint information may include parameter information such as the height and width of the predetermined area T, and parameter information such as the magnification ratio due to the zoom of the virtual camera IC. Furthermore, if the position of each pixel in the equirectangular projection image EC, as shown in Figure 7(c), is associated with the coordinates of the surface of the sphere (for example, coordinates with two axes: latitude and longitude), then parameter information such as the direction and field of view of the virtual camera IC may be used as viewpoint information, or information such as latitude and longitude may be included in the viewpoint information. Thus, viewpoint information is not necessarily limited to information that indicates a point.

[0098] <About the features> Next, the functional configuration of the communication system 1a according to this embodiment will be described using Figure 13. Figure 13 is a diagram showing an example of the functional configuration of the communication system 1a according to this embodiment. In Figure 13, the terminals, devices, and servers shown in Figure 1 that are related to the processing or operation described later are shown.

[0099] <<Functional Configuration of Imaging Device>> First, the functional configuration of the imaging device 10 will be described with reference to Figure 13. The imaging device 10 includes a communication unit 11, a reception unit 12, an imaging processing unit 13, an analysis unit 14, a registration request unit 15, a connection unit 16, a storage processing unit 17, an image transmission control unit 18, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 111 that follow a program deployed on the SRAM 113 or DRAM 114. The imaging device 10 also has a storage unit 1000 constructed from a ROM 112, etc., as shown in Figure 3.

[0100] The communication unit 11 is primarily implemented by the processing of the short-range communication circuit 117 by the CPU 101, and has the function of connecting to the communication network N using wireless communication means such as Wi-Fi to send and receive various data or information with other devices. In this embodiment, the connection unit 16 mainly transmits the wide-field image acquired by the imaging processing unit 13 to the information processing system 50, but it is also possible for the communication unit 11 to transmit the wide-field image to the information processing system 50.

[0101] The reception unit 12 is primarily implemented by the processing performed by the CPU 101 on the operation unit 115, and its function is to receive user input for the imaging device 10. The reception unit 12 receives user input such as power on / off, shutter button on / off (start or stop transmission of wide-field images), and input from the touch panel or buttons.

[0102] The imaging processing unit 13 is primarily implemented by the CPU 101's processing of the image processing unit 104, and captures images of subjects, landscapes, etc., and acquires (generates) the captured images. The captured images acquired by the imaging processing unit 13 may be videos or still images (or both), and may include sound along with the image. The imaging processing unit 13 also captures, for example, a two-dimensional code (see Figure 20) displayed on the display 306 of the communication terminal 30. Furthermore, the imaging processing unit 13 may generate a wide-field image by performing the image processing described in Figures 7 and 8 on the captured image.

[0103] The analysis unit 14 is mainly implemented by the CPU 101 and analyzes the two-dimensional code captured and acquired by the imaging processing unit 13 to extract the information contained in the two-dimensional code (URL for registering the imaging device with the tenant, temporary ID and password).

[0104] The registration request unit 15 is mainly implemented by the CPU 101 and uses the information contained in the two-dimensional code read by the analysis unit 14 to send a request to the information processing system 50 via the communication unit 11 to register the imaging device 10 as a tenant of the information processing system 50.

[0105] The connection section 16 is primarily implemented by the processing of the input / output interface 116 by the CPU 101, and has the function of receiving power supply from the communication terminal 30A and performing data communication.

[0106] The storage processing unit 17 is mainly implemented by the CPU 101 and performs the process of saving wide-field images captured in response to imaging requests from any location to a URL (e.g., storage 90) notified by the information processing system 50.

[0107] The image transmission control unit 18 is primarily implemented by the CPU 101 and controls the transmission of wide-field images to the information processing system 50. For example, the image transmission control unit 18 transmits captured images acquired by the imaging processing unit 13 to the information processing system 50 periodically or in response to user input if they are still images, or at a predetermined FPS (Frames Per Second) if they are videos. The image transmission control unit 18 also switches between the communication unit 11 and the connection unit 16.

[0108] The storage / reading unit 19 is mainly implemented by the CPU 101 and has the function of storing various data in the storage unit 1000 or reading various data from the storage unit 1000. The storage unit 1000 also stores the image capture data acquired by the image capture processing unit 13, the image capture device ID, etc. The image capture data stored in the storage unit 1000 may be deleted after a predetermined time has elapsed since it was acquired by the image capture processing unit 13, or it may be deleted when data has been transmitted to the information processing system 50.

[0109] Furthermore, the imaging device 10 has an application (also called a plug-in) installed to support the communication system 1a. This application is used when associating the imaging device 10 with a virtual room and when receiving control from external sources. Some of the functions shown in Figure 13 (for example, the registration request unit 15) are provided by this application. Alternatively, the application for supporting the communication system 1a may be placed on the communication network N, and the same functions may be achieved by accessing the application using a web browser or the like on the imaging device 10.

[0110] <<Communication terminal function configuration>> Next, the functional configuration of the communication terminal 30 will be explained using Figure 13. The communication terminal 30 includes a communication unit 31, a reception unit 32, a display control unit 33, an imaging unit 34, a connection unit 36, and a storage / reading unit 39. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 301 following a program (which may be a web browser or a dedicated application) deployed on the RAM 303. The communication terminal 30 also has a storage unit 3000 constructed from the ROM 302 or recording medium 315 shown in Figure 4.

[0111] The communication unit 31 is primarily implemented by the processing performed by the CPU 301 on the network interface 309, and has the function of connecting to the communication network N and sending and receiving various data or information with other devices.

[0112] The reception unit 32 is mainly implemented by the processing of the keyboard 311 and pointing device 312 by the CPU 301, and has the function of receiving various selections or operation inputs to the communication terminal 30. The display control unit 33 has the function of displaying wide-field images, normal-angle images, and various screens on the display 306 of the communication terminal 30.

[0113] The display control unit 33 is mainly implemented by the processing of the CPU 301, and for example, it displays a two-dimensional code transmitted from the information processing system 50 on the display 306. The two-dimensional code is, for example, QR code (registered trademark), DataMatrix (DataCode), MaxiCode, or PDF417. The two-dimensional code may also be a barcode.

[0114] The connection section 36 is primarily realized by the processing performed by the CPU 301 on the short-range communication circuit 320, and its function is to supply power to the imaging device 10 and to perform data communication.

[0115] The storage / reading unit 39 is mainly executed by the CPU 301 and has the function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. The storage unit 3000 has an image management information storage unit 3001. The image management information storage unit 3001 will be explained in the description of the information processing system 50.

[0116] <<Functional Configuration of the Information Processing System>> Next, the functional configuration of the information processing system 50 will be described. The information processing system 50 includes a communication unit 51, a screen generation unit 52, an association processing unit 53, an image distribution unit 54, an authentication unit 55, a communication group management unit 56, a communication control unit 57, a connection management unit 58, a storage / reading unit 59, and an API management unit 60. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with a program deployed on the RAM 503. The information processing system also includes a storage unit 5000 constructed from a ROM 502, HDD 504, or recording media 515, as shown in Figure 4.

[0117] The communication unit 51 primarily performs the function of sending and receiving various data or information with other devices via the communication network N, which is processed by the CPU 501 for the network interface 509.

[0118] The screen generation unit 52 is mainly implemented by the CPU 501 and generates screen information to be displayed by the communication terminal 30. When the communication terminal 30 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc. When the communication terminal 30 runs a native application, the screen information is held by the communication terminal 30, and the information to be displayed is transmitted in XML, etc. The screen generation unit 52 generates screen information on which wide-field images, etc., distributed by the image distribution unit 54 via the communication unit 51 are placed.

[0119] The association processing unit 53 is mainly implemented by the CPU 501 and controls the association and sharing of viewpoint information for wide-field images. When the association processing unit 53 receives viewpoint information and an imaging request from the communication terminal 30, it requests imaging from the imaging device 10 and performs processing to associate the acquired wide-field image with the viewpoint information. Furthermore, the associated wide-field image and viewpoint information are stored in the image management information storage unit 5001 by the storage / reading unit 59. The association processing unit 53 also sends storage location information (e.g., URL) to the communication terminal 30 as information indicating the storage location where the associated wide-field image and viewpoint information are stored. Note that the information processing system 50 does not need to receive viewpoint information and an imaging request from the communication terminal 30 simultaneously; it may receive them separately and then perform the association processing. Also, the URL is just one example of storage location information, and other formats such as URI may be used.

[0120] The image distribution unit 54 is primarily implemented by the CPU 501 and distributes images such as wide-field images transmitted by the imaging device 10, which is associated with the same virtual room, to the communication terminal 30 operated by the user who is in the virtual room, via the communication unit 51. Images of the normal field of view captured by the camera of the communication terminal 30 or connected cameras 8 and 9 are also distributed in the same manner. The distributed images include streaming video, video, still images, etc.

[0121] The authentication unit 55 is primarily implemented by the CPU 501 and performs authentication of the requester based on the authentication request received by the communication unit 51. The authentication unit 55 authenticates the user, for example, by determining whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches pre-held authentication information. The authentication information may include the card number of an IC card, biometric authentication information such as a face, fingerprint or voiceprint, device ID, passcode, access token, security key, ticket, etc. The authentication unit 55 may also authenticate using an external authentication system or authentication method such as OAuth. Furthermore, the authentication unit 55 may authenticate not only users but also devices such as imaging devices.

[0122] The communication group management unit 56 is primarily implemented by the CPU 501 and manages the entry of communication terminals 30 and users into virtual rooms, as well as the mapping of devices. When authentication by the authentication unit 55 is successful, the communication group management unit 56 registers the user ID and the IP address of the communication terminal 30 in the virtual room information storage unit 5002, and associates the imaging device 10 with the virtual room.

[0123] The communication control unit 57 is primarily implemented by the CPU 501 and manages the initiation, establishment, and termination of communication with the imaging device 10 associated with each virtual room. The communication control unit 57 also manages the initiation, establishment, and termination of communication for distributing wide-field images and audio in response to communication terminals 30 entering or leaving virtual rooms.

[0124] The connection management unit 58 is primarily implemented by the CPU 501 and manages the communications (connections) established between the communication terminal 30 and the imaging device 10 and the information processing system 50, associating them with virtual rooms.

[0125] The API management unit 60 is primarily implemented by the CPU 501 and manages APIs used by platform subscribers when providing wide-field image distribution services. When using APIs, platform subscribers only need to develop separate software to call the APIs. The developed software may run on a server or on a client such as a communication terminal. Any function provided by the information processing system 50, such as the image distribution unit 54, association processing unit 53, and communication control unit 57, can be provided as an API. It is also possible to provide functions added to the information processing system 50 later as APIs. Whether or not to provide an API is determined by the communication terminal operated by the platform provider accessing the information processing system 50 and accepting the API disclosure setting, allowing the API management unit 60 to control the API based on the disclosure setting. The API management unit 60 may also perform authentication processing to verify whether the requesting software requesting the API call is software developed by a legitimate platform subscriber. Authentication processing can be performed by comparing information pre-registered and stored as platform subscriber information in the storage unit 5000 with information transmitted from the requesting software.

[0126] As a specific example of the authentication process, the information processing system 50 receives an application ID, which has been previously issued by the API management unit 60, from the requesting software to the software developed by the platform subscriber. If the API management unit 60 determines that the application ID is stored in the storage unit 5000, the API management unit 60 performs control to permit the provision of the API as legitimate software. On the other hand, if it cannot determine that the software is legitimate, the API management unit 60 performs control to deny permission to provide the API.

[0127] The app ID is just one example of authentication information used to determine legitimacy, and the API management unit 60 may verify the legitimacy of the requester using authentication information such as an access token, ticket, security key, password, or PIN code issued in advance by the API management unit 60 of the information processing system or an external system. In this embodiment, the form in which the functions provided by the information processing system 50 are used as an API is not described, but the processing flow is the same except that the software such as an application developed by the platform subscriber uses the functions provided by the information processing system 50 via a decision made by the API management unit 60.

[0128] The memory / read unit 59 is mainly implemented by the CPU 501 and has the function of storing various data in the memory unit 5000 or reading various data from the memory unit 5000.

[0129] "Image Management Information Storage Unit 5001" The memory unit 5000 includes an image management information storage unit 5001. Figure 14(a) is a conceptual diagram showing the image management information stored in the image management information storage unit 5001. While the information is managed in a table format here, it does not necessarily have to be managed in a table format.

[0130] The image management information storage unit 5001 stores image management information as shown in Figure 14. Image management information is information for managing wide-field images captured in response to an imaging request. When a user sends an imaging request from the communication terminal 30, one record of image management information is generated. The items contained in the image management information are described below.

[0131] The data ID for wide-field video is identification information used to identify data in wide-field video. The data ID is assigned by the information processing system 50. ID is an abbreviation for Identification and means identifier or identification information. An ID is a name, code, string, number, or a combination of one or more of these used to uniquely distinguish a particular object from multiple objects. Note that the data ID may be associated not only with wide-field video but also with images captured at a normal field of view by the imaging device 10 associated with the virtual room.

[0132] The data name is the name of the wide-field video set by the user of the communication terminal 30. The data name can be set by the user, but it may also be set automatically.

[0133] The imaging date and time information is information used to identify the imaging date and time of the wide-field video or other captured image, such as the date and time when the user entered an imaging request into the communication terminal 30, and the date and time when the imaging device 10 captured the image, such as the wide-field video. The imaging date and time information may be substituted with the timestamp information of the image, such as the wide-field video.

[0134] The imager information is the identification information (including user ID and username) of the user who entered the imaging request into the communication terminal 30. Since the user enters the imaging request into the communication terminal 30 while inside the virtual room, the user registered in the imager information is identified by authentication to the information processing system 50 or the virtual room. The imager information is transmitted to the information processing system 50 along with the imaging request. Note that the imaging request and the imager information do not necessarily have to be transmitted to the information processing system 50 at the same time; they may be transmitted to the information processing system 50 at different times.

[0135] The imaging device information is the identification information (imaging device ID) of the imaging device 10 that captured images and generated a wide-field video. The imaging device ID is assigned by the information processing system 50 and shared with the imaging device 10, but information unique to the imaging device 10, such as a MAC address or serial number, may also be used. The imaging device ID is transmitted to the information processing system 50 along with the wide-field video. Note that the imaging device ID and the wide-field video do not necessarily have to be transmitted to the information processing system 50 at the same time; they may be transmitted to the information processing system 50 at different times.

[0136] The viewpoint information of the imager is the viewpoint information specified in the imager's communication terminal 30. For example, the viewpoint information indicates the center coordinates (center point in Figure 11) of the wide-field image displayed by the communication terminal 30, and is parameter information used to identify a predetermined region T (see Figures 9 and 10) of the wide-field image displayed by the communication terminal 30. Here, as an example of parameter information, the radial angle (r), polar angle (θ), and azimuth angle (φ) are shown, but other parameter information as described in Figures 10 to 12 may also be used. The viewpoint information is transmitted from the communication terminal 30 requesting imaging. The viewpoint information may also include information specifying the width and height of the display range of the predetermined region T. Alternatively, the viewpoint information may only include the width and height of the display range.

[0137] The virtual room ID used during imaging is the identification information of the virtual room to which the imaging device 10 is associated.

[0138] The storage location information (storage location information) for wide-field video data is information indicating where the wide-field video is stored, such as a URL or file path. The storage location identified by the storage location information may also be information indicating a predetermined folder. The folder may be a folder associated with the virtual room used during imaging. Alternatively, it may be a folder associated with identification information (additional information such as a name) indicating one or more combinations of classifications such as imaging date and time, imaging device, imager, and virtual room used during imaging. Furthermore, the storage location information of the data may be combined with information such as data ID and data name to identify the storage location of the data.

[0139] Figure 14(b) is a conceptual diagram showing image management information as a variation of Figure 14(a). While the information is managed in a table format here, it does not necessarily have to be. In Figure 14(b), wide-field videos with the same virtual room ID at the time of acquisition are stored. Thus, image management information may be classified by virtual room.

[0140] "Virtual Room Information Storage Unit 5002" A virtual room information storage unit 5002 is constructed within the memory unit 5000. Figure 15(a) is a conceptual diagram showing the virtual room information stored in the virtual room information storage unit 5002. The virtual room information storage unit 5002 stores virtual room information as shown in Figure 15(a). Virtual room information is information about a virtual room and is maintained for each virtual room. The items contained in the virtual room information will be explained below. Note that although virtual rooms are registered with tenants here, registration with tenants is not mandatory, and information on temporarily created virtual rooms and virtual rooms that can be used for sharing is also stored in the virtual room information storage unit 5002.

[0141] The virtual room ID is identification information that identifies the virtual room. In this embodiment, virtual rooms can be created at the user's discretion.

[0142] The virtual room name is a name used by users to identify a virtual room and can be set by the user at will. The virtual room ID and virtual room name may be the same information.

[0143] The device information is the identification information (device ID) of the device, including the imaging device 10, which is associated with the virtual room.

[0144] The user currently in the virtual room is the user ID of the user currently in the virtual room. This user is able to view images such as wide-field images distributed to the virtual room entrants. The method of entering the room will be described later. In addition, the user ID may be associated with the IP address of the communication terminal 30 operated by that user. In addition, the username may be associated with and stored as the user ID.

[0145] "Tenant Information Storage Unit 5003" A tenant information storage unit 5003 is constructed within the storage unit 5000. Figure 15(b) is a conceptual diagram showing the tenant information stored in the tenant information storage unit 5003. The tenant information storage unit 5003 stores tenant information as shown in Figure 15(b). Tenant information is information about tenants (user groups) and is maintained for each tenant. The items contained in the tenant information will be explained below. Note that various other information is registered in the tenant information, such as user information, and Figure 15(b) is only a part of it. • The tenant ID is identification information that identifies the tenant. The tenant name is a name used by users to identify the tenant. Note that the tenant ID and tenant name may be the same information. The tenant registration virtual room ID is the identification information for the virtual room registered with the tenant. • Tenant registration devices are information about devices registered with the tenant. Furthermore, the tenant information storage unit, tenant ID, tenant name, tenant registration virtual room ID, and tenant registration device can be rephrased as the user group information storage unit, user group ID, user group name, user group registration virtual room ID, and user group registration device, respectively.

[0146] "Replay-related information storage unit 5004" A playback-related information storage unit 5004 is constructed within the memory unit 5000. Figure 16 is a conceptual diagram showing the playback-related information stored in the playback-related information storage unit 5004. In this case, the information is managed in a table format, but it does not have to be managed in a table format.

[0147] The playback-related information includes the still image ID of the wide-field still image, the video ID of the wide-field video, the playback elapsed time, the designated area information (viewpoint information), and the storage location information of the wide-field still image. Note that the video ID and designated area information of the wide-field video have the same meaning as the video ID and designated area information of the wide-field video in Figures 14(a) and (b), respectively.

[0148] The still image ID of a wide-field still image is an example of identification information used to identify a wide-field still image extracted as a frame from a wide-field video, which is shown in Figure 16 by the video ID of the wide-field video in the same record.

[0149] The playback elapsed time is the playback elapsed time (timestamp) in the wide-field video when a wide-field still image is extracted as a frame from the wide-field video. This playback elapsed time corresponds to the playback elapsed time te in Figure 51. For example, in the case of a video with a total playback time of 10 minutes, the midpoint of the total playback time is at the 5-minute mark. The storage location information for wide-field still images is the storage location information for wide-field still images, which are extracted as frames from the wide-field video indicated by the video ID of the wide-field video in the same record, and are indicated by the still image ID in the same record.

[0150] <Entering a virtual room using a communication terminal> Next, referring to Figures 17 and 18, the process of user b entering the virtual room will be explained. It is assumed that the imaging device 10 has already been associated with the virtual room, and that the communication terminal 30A has transmitted wide-field images and normal-angle images to the information processing system 50 (the association of the imaging device 10 with the virtual room will be explained in Figure 19 and later). Furthermore, in the following explanation, there will be no particular distinction made between user b entering the virtual room and the communication terminal 30B operated by user b entering the virtual room.

[0151] Figure 17 shows an example of a screen displayed by the communication terminal 30B when user b enters a virtual room. Figure 17(a) is an example of the entry screen 200. Prior to the display of the entry screen 200, user b is logged into the information processing system 50. Logging in identifies the tenant to which user b belongs. Virtual rooms are associated with tenants. User b displays a list of virtual rooms associated with the tenant on the communication terminal 30B (see Figure 22) and selects a virtual room to enter from the list. Figure 17(a) is the entry screen 200 for the virtual room selected by user b in this way. Note that temporarily created virtual rooms or shared virtual rooms not associated with a tenant may also be displayed on the screen in Figure 17(a).

[0152] Alternatively, the creator of the virtual room may request the information processing system 50 to issue a URL corresponding to the virtual room and send this URL to user b via email or other means. When user b clicks the URL displayed on the communication terminal 30B, the communication terminal 30B displays the room entry screen 200 shown in Figure 17(a).

[0153] The entry screen 200 has a virtual room name 201, a participant name input field 202, and an entry button 203. The virtual room name 201 is the same as the one stored in the virtual room information storage unit 5002. The participant name input field 202 is a field for entering the username to be displayed in the virtual room, which may be a nickname or other designation for user b. When user b logs in, the username associated with user b's user ID is identified, and this identified username may be displayed automatically. The entry button 203 is a button for user b to request entry into the virtual room.

[0154] Furthermore, authentication for entering the virtual room may be required separately from the tenant login process.

[0155] Figure 17(b) shows the image viewing screen 210 displayed by the communication terminal 30B when user b enters the virtual room. In the image viewing screen 210 of Figure 17(b), the imaging device 10 has already started distributing wide-field images via the information processing system 50, and the communication terminal 30A has already started distributing images with a normal field of view. Therefore, the image viewing screen 210 has a first image section 211 and a second image section 212. The first image section 211 displays the wide-field image, and the second image section 212 displays the image with a normal field of view. If there are three or more locations transmitting images, the image viewing screen 210 is divided according to the number of transmitting locations.

[0156] The first image field 211 displays a wide-field image mark 213. The wide-field image mark 213 is set by the screen generation unit 52 of the information processing system 50 when it determines that the image to be displayed in the first image field 211 is a wide-field image. The communication terminal 30B may also determine and display it. By seeing the wide-field image mark 213, user b can understand that a wide-field image that allows the viewpoint to be changed is being delivered. In addition, the first image field 211 displays the device name 214 (transmitted from the imaging device 10 along with the wide-field image). The device name 214 is information set by user a, etc., as will be described later (see Figure 20).

[0157] The second image field 212 displays the participant name 215. The participant name 215 is a username, and the participant name of a user who has already entered the room (in this case, user a has already entered, so "AAA" entered by user a in the participant name input field 202) is displayed in the participant name input field 202.

[0158] Figure 18 is a sequence diagram illustrating the process by which user b (or communication terminal 30B) enters a virtual room.

[0159] S1: First, user b at site B performs an operation to display the virtual room list screen. The communication terminal 30B has previously accessed the information processing system 50 in response to the operation by user b and has received virtual room information stored in the virtual room information storage unit 5002 for displaying the virtual room list screen from the information processing system 50. At this time, the communication terminal 30B may be authenticated by the authentication unit 55 of the information processing system 50 by sending authentication information necessary for login, etc., to the information processing system 50. The authentication information may be authentication information associated with user b or authentication information associated with the communication terminal 30B. In such cases, the virtual rooms displayed on the list screen may be virtual rooms registered in the tenant associated with user b or virtual rooms registered in the tenant associated with the communication terminal 30B. When the reception unit 32 receives the operation to display the list screen, the display control unit 33 of the communication terminal 30B displays the selection screen on the display 306.

[0160] S2: When user b selects a virtual room selection button, the reception unit 32 of the communication terminal 30B accepts the selection of the virtual room. The display control unit 33 of the communication terminal 30B displays the room entry screen 200 shown in Figure 17(a) on the display 306.

[0161] S3: User b enters the required information and presses the enter button 203. Upon receiving the press from the reception unit 32, the communication unit 31 of the communication terminal 30B sends an entry request to the information processing system 50. This entry request includes information such as the virtual room ID indicating the virtual room selected in step S2, the user ID of user b authenticated by login, etc., and the IP address of the requesting terminal, communication terminal 30B. As a result, the communication unit 51 of the information processing system 50 receives the entry request.

[0162] S4: The communication group management unit 56 registers the user ID and IP address authenticated through login, etc., in the virtual room information identified by the virtual room ID in the virtual room information storage unit 5002.

[0163] S5: The communication unit 51 of the information processing system 50 then sends a response to the communication terminal 30B indicating that it has entered the room. As a result, the communication unit 31 of the communication terminal 30B receives the response indicating that it has entered the room. Following S5, the display control unit 33 of the communication terminal 30B receives the screen information generated by the screen generation unit 52 of the information processing system 50 and the image information distributed by the image distribution unit 54, and displays the image viewing screen 210 shown in Figure 17(b) based on the received information.

[0164] <Mapping of imaging equipment to rooms> Next, the mapping of the imaging device 10 to the virtual room will be explained with reference to Figures 19 to 26. Note that the mapping of the imaging device 10 to the virtual room will be explained assuming that user a at site A performs this task, but it may also be performed by a system administrator, tenant administrator, etc.

[0165] Figure 19 shows an example of a device registration screen 220 displayed by the communication terminal 30A. User a is authenticated by logging into the information processing system 50. Logging in identifies the tenant to which user a belongs. User a requests the information processing system 50 to display the device registration screen 220, and the communication terminal 30A displays the device registration screen 220 received from the information processing system 50. The device is first registered with the tenant via the device registration screen 220.

[0166] The device registration screen 220 has buttons for registering an imaging device 221, a VR goggles 222, and smart glasses 223. Buttons are provided for each type of device because there are differences in the presence or absence of a camera, the information used for registration, etc. Furthermore, since devices are registered according to their type, the information processing system 50 can also determine the type of device.

[0167] The imaging device registration button 221 is for user a to register the imaging device 10, the VR goggles registration button 222 is for registering the VR goggles 89, and the smart glasses registration button 223 is for registering the smart glasses 88.

[0168] Figure 20 shows an example of a screen displayed when the imaging device registration button 221 is pressed. Figure 20(a) shows the imaging device registration dialog 230. The imaging device registration dialog 230 has a name field 231 for the imaging device 10, a description field 232, and a next button 233. User a sets an arbitrary name in the name field 231 for the imaging device 10 so that it is clear that it is the imaging device 10 to be registered, and sets a description in the description field 232.

[0169] When user a presses the next button 233, the communication terminal 30A requests a two-dimensional code from the information processing system 50, and the communication terminal 30A displays the two-dimensional code.

[0170] Figure 20(b) shows an example of a two-dimensional code screen 240 displayed by the communication terminal 30A. The two-dimensional code screen 240 contains a message 241 stating, "Please scan the following two-dimensional code to register the device named XX (the name entered in the name field)," and a two-dimensional code 242. User a scans the two-dimensional code 242 with the imaging device 10 that they wish to register. The two-dimensional code 242 contains authentication information necessary for registration, such as the URL to which the imaging device 10 connects for its own registration, and a temporary ID and password.

[0171] When user a scans the two-dimensional code 242 with the imaging device 10, the imaging device 10 connects to a URL and is authenticated with a temporary ID and password. If authentication is successful, the official imaging device ID is exchanged, and the name, description, and imaging device ID of the imaging device 10 are registered with the tenant. The imaging device 10 also retains this imaging device ID, name, and description. The imaging device 10 registered with the tenant is associated with a virtual room according to the operations of user a, as described later. Note that the two-dimensional code 242 is just one example of code information; any code with similar authentication information embedded may be used, and other forms of codes such as barcodes may also be used.

[0172] Next, with reference to Figure 21, an example of how to register communication terminals such as VR goggles 89 and smart glasses 88 to a tenant will be explained. Figure 21 is an example of the VR goggles registration screen 250 that is displayed when the VR goggles registration button 222 is pressed. The VR goggles registration screen 250 has a temporary code input field 251 and a secret input field 252.

[0173] If the VR goggles 89 do not have a camera, they cannot capture the QR code. For this reason, user a may have the VR goggles 89 output (display) a temporary code (temporary ID) and a secret (password), and enter them into the temporary code input field 251 and the secret input field 252. The communication terminal 30A registers the VR goggles 89 with the tenant by transmitting the temporary code and secret to the information processing system 50. The VR goggles 89 connects to the information processing system 50 and receives authentication by transmitting the temporary code and secret. If authentication is successful, a formal VR goggle ID is exchanged, and the VR goggle ID is registered with the tenant. The VR goggles 89 also holds this VR goggle ID. The VR goggles 89 registered with the tenant is associated with a virtual room according to the operation of user a, which will be described later. The smart glasses 88 will be explained in detail later, but user a can register them in the same way as the imaging device 10 or the VR goggles 89. Note that the temporary code and secret are just examples of authentication information, and other information may be used as authentication information. The imaging device ID, VR goggles ID, and smart glasses ID are examples of device IDs and can therefore be referred to simply as "device ID." Thus, when registering devices other than the imaging device 10, VR goggles, and smart glasses, the same procedure can be used to associate the device ID with virtual rooms and tenants. The device ID may also be identification information linked to the device owner.

[0174] Figure 22 shows an example of a virtual room mapping screen (part 1) 260 for associating the imaging device 10 with a virtual room. The screen configuration can be the same for VR goggles 89 and smart glasses 88. The virtual room mapping screen (part 1) 260 has a virtual room list 261. The virtual room list 261 displays individual virtual room fields 262 to 264 based on the virtual rooms created in the tenant. Each individual virtual room field 262 to 264 has a link issuance button 265, an entry button 266, a settings button 267, and a virtual room name 268. The link issuance button 265 is a button for issuing a link to the virtual room (URL for invitation) and a passcode. The entry button 266 is a button for user a to enter the virtual room. The settings button 267 is a button for associating the imaging device 10 with the virtual room. The virtual room name 268 is the same as that stored in the virtual room information storage unit 5002. Therefore, user a presses the settings button 267. Pressing the settings button 267 causes the communication terminal 30A to display the virtual room mapping screen (part 2) 270.

[0175] Additionally, if a device is already associated with a virtual room, the device name 269 will be displayed in the individual virtual room section (individual virtual room section 264 in the diagram).

[0176] Figure 23 shows an example of the virtual room mapping screen (part 2) 270. Note that the virtual room mapping screen (part 2) 270 is displayed as a pop-up on the virtual room mapping screen (part 1) 260. The screen transition from the virtual room mapping screen (part 1) 260 to the virtual room mapping screen (part 2) 270 does not go through the information processing system 50, but a screen transition that does go through the information processing system 50 is also possible.

[0177] The virtual room mapping screen (part 2) 270 has the name 271 of the imaging device 10 currently (already) mapped to the virtual room (not yet registered, so not shown in the diagram), a connection button 272, and a storage button 273. The connection button 272 is a button that displays a list of devices registered in the tenant as mapping candidates for mapping a device to a virtual room. The storage button 273 is a button that displays a list of storage 90s that store wide-field images and normal-angle images captured by the imaging device 10 mapped to the virtual room. The list of storage 90s may include not only a list of storage 90s mapped to the virtual room, but also a list of specific storage locations such as folders on the storage 90. By selecting a predetermined storage 90 or a specific storage location such as a folder on the storage 90, the storage 90 can be mapped to the virtual room. The information of the storage 90 mapped in this way (address information for accessing the storage 90 and storage locations such as folders on the storage 90) can be stored in the virtual room information storage unit 5002 in association with the virtual room ID. When the connect button 272 is pressed, the communication terminal 30A displays the virtual room mapping screen (part 3).

[0178] The communication terminal 30A transmits the virtual room ID to the information processing system 50 and obtains the name of the device registered in the tenant where the virtual room is created (including the device ID, etc.), and the name of the device associated with the virtual room (including the device ID, etc.).

[0179] Figure 24 shows an example of the virtual room mapping screen (part 3) 280. The virtual room mapping screen (part 3) 280 has the name 281 of the imaging device 10 currently (already) mapped to the virtual room, a list of additional devices 282, and a save button 283. User a selects a device from the list of additional devices 282 that they want to map to the virtual room and presses the save button 283. This maps the device to the virtual room (the device ID, such as the imaging device ID, is registered in the virtual room information storage unit 5002). Note that, as shown in Figure 24, the number of imaging devices that can be mapped to a virtual room may be limited. For example, if the upper limit is two devices, the number of remaining devices that can be registered may be displayed on the virtual room mapping screen (part 3) by referring to the number of imaging device IDs already registered in the virtual room information storage unit 5002.

[0180] <Processing to initiate transmission of wide-field images to the imaging device> With the above steps, devices such as the imaging device 10 are associated with the virtual room, but user a needs to initiate image transmission to the device.

[0181] For the VR goggles 89 and smart glasses 88, user a can turn image transmission on and off by operating the device itself. This is because currently, there is no dedicated application running on the communication system 1a for the VR goggles 89 and smart glasses 88. If a dedicated application were running on the communication system 1a for the VR goggles 89 and smart glasses 88, user a would be able to remotely turn image transmission on and off.

[0182] In the case of the imaging device 10, if the application is enabled, user a can enter the virtual room and turn the transmission of wide-field images on or off from the menu.

[0183] Figure 25 shows an example of a wide-field image transmission control dialog 290 displayed by the communication terminal 30A. The wide-field image transmission control dialog 290 is displayed as a pop-up on the image viewing screen 210. Assume that user a has entered a virtual room associated with the imaging device 10 by operating the communication terminal 30A. The wide-field image transmission control dialog 290 displays the name 292 of the imaging device 10 associated with this virtual room. A toggle button 291 is displayed near the name 292, and user a can operate the toggle button 291 to turn on (start transmission) or off (stop transmission) the transmission of wide-field images by the imaging device 10. Note that the method of setting on or off using the toggle button is just one example, and it is sufficient if it can be set according to user input. For example, it may be set by selecting radio buttons or predetermined icons, or by menu operation. Alternatively, the transmission of wide-field images may start automatically after the imaging device 10 enters the room, without requiring user operation. Alternatively, certain conditions such as the date and time, the number of users who entered the room, or whether a specific user participated may be predetermined, and the transmission of wide-field images may begin when it is determined that these conditions have been met.

[0184] The communication terminal 30A transmits transmission control setting information to the information processing system 50 by operating the toggle button 291. The information processing system 50 transmits a transmission start request or a transmission stop request to the imaging device 10 according to the transmission control setting information.

[0185] Figure 25(a) shows the state where the toggle button 291 is set to OFF. Therefore, the wide-field image is not displayed in Figure 25(a). On the other hand, in Figure 25(a), when the communication terminal 30A enters the room, the image of the normal field of view captured by the camera 9 of the communication terminal 30A has already been shared and is displayed on the image viewing screen 210.

[0186] Figure 25(b) shows the state where the toggle button 291 is set to ON. When the toggle button 291 is turned ON, the information processing system 50 sends a transmission start request to the imaging device 10, and the imaging device 10 starts transmitting the wide-field image. As a result, two images are shared in one virtual room, and the image viewing screen 210 is divided into two. Also, when the setting is changed from ON to OFF, the communication terminal 30A sends OFF setting information, and the information processing system 50, upon receiving the OFF setting information, sends a transmission stop request to the imaging device 10, and the imaging device 10 stops transmitting the wide-field image.

[0187] As explained in Figure 26, even if a user is on-site, the imaging device 10 can be associated with a virtual room through a simple operation such as capturing code information with the imaging device 10. Since on-site users may not have a PC, the ability to perform the association process on the spot, using only pre-issued code information and the imaging device 10, is particularly useful for on-site users. Furthermore, by performing the association process in advance, the user can connect the imaging device 10 to a designated virtual room without having to select a virtual room, and the start or stop of transmission can be instructed from a remote location, thus reducing the burden on users who want to concentrate on their work on-site. Therefore, a system can be provided that enables efficient communication between the on-site and remote locations even during the preparation process.

[0188] <<Procedure for registering the imaging device in the virtual room>> Next, referring to Figure 26, the procedure for registering the imaging device 10 to the virtual room, as described in the series of screen transitions in Figures 19 to 25, will be explained. Figure 26 is an example of a sequence diagram showing the procedure for user a to register the imaging device 10 to the virtual room.

[0189] S11: First, user a connects the communication terminal 30A to the information processing system 50, enters authentication information (user ID, password, etc.), and requests login. The reception unit 32 of the communication terminal 30A accepts the operation.

[0190] S12: The communication unit 31 of the communication terminal 30A sends a login request to the information processing system 50, specifying authentication information. The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 55 performs authentication based on the specified authentication information. Here, we assume that the authentication was successful. At this time, the information processing system 50 can also identify the tenant ID associated with the authenticated user ID by referring to the tenant information storage unit 5003.

[0191] S13: In response to user operation, the screen generation unit 52 of the information processing system 50 generates a device registration screen 220, and the communication unit 51 transmits the screen information of the device registration screen 220 to the communication terminal 30A.

[0192] S14: The communication unit 31 of the communication terminal 30A receives screen information from the device registration screen 220, and the display control unit 33 displays the device registration screen 220 shown in Figure 19. User a selects the type of device (here, the imaging device 10 (for example, a 360-degree camera) is selected), and then enters the name and description of the imaging device 10 as shown in Figure 20. The reception unit 32 accepts the input.

[0193] S15: The communication unit 31 of the communication terminal 30A sends a request for code information (e.g., a two-dimensional code) to the information processing system 50, specifying the name and description entered by user a.

[0194] S16: The communication unit 51 of the information processing system 50 receives a request for code information (e.g., a two-dimensional code). The communication group management unit 56 generates a URL (a connection destination for registration) associated with the name and description, and generates code information (e.g., a two-dimensional code) including the URL, a temporary ID, and a password. The communication unit 51 of the information processing system 50 transmits the code information (e.g., a two-dimensional code) to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the code information (e.g., a two-dimensional code), and the display control unit 33 displays the code information (e.g., a two-dimensional code) as shown in Figure 20.

[0195] S17: Next, user a operates the imaging device 10 that they want to associate with the virtual room to capture code information (e.g., a two-dimensional code). The reception unit 12 of the imaging device 10 receives the operation.

[0196] S18: The imaging processing unit 13 of the imaging device 10 generates image data by performing imaging processing on the target to be imaged, which includes code information (e.g., a two-dimensional code), and the analysis unit 14 analyzes the image data to extract a URL, a temporary ID, and a password. As a result, the registration request unit 15 connects to the URL via the connection unit 16 and sends a registration request from the imaging device 10 to the information processing system 50, specifying the temporary ID and password. Note that if the registration method using the registration screen described in Figure 21 is implemented, the code information is not imaged, so the imaging device 10 is replaced by a communication terminal such as VR goggles 89 or smart glasses 88, and steps S15 to S17 can be omitted.

[0197] S19: The communication unit 51 of the information processing system 50 receives a temporary ID and password, and the authentication unit 55 determines whether they match the temporary ID and password associated with the connected URL. Here, we assume they match.

[0198] S20: The communication group management unit 56 of the information processing system 50 receives a request to register the imaging device 10. As an example of a device ID, it generates an imaging device ID and registers it in the tenant corresponding to the tenant ID identified when user a logged in. The imaging device ID is associated with a name and description. Specifically, the communication group management unit 56 refers to the tenant information storage unit 5003 and adds the imaging device ID to the tenant registration devices associated with the identified tenant ID and registers it. Although the communication group management unit 56 generates and registers the imaging device ID here, it may also register the imaging device ID received from the imaging device 10. If a communication terminal such as VR goggles 89 or smart glasses 88 is to be registered in the tenant instead of the imaging device 10, the corresponding device ID can be registered in the tenant information storage unit 5003 using the same procedure.

[0199] S21: The communication unit 51 of the information processing system 50 transmits the imaging device ID to the imaging device 10. The connection unit 16 of the imaging device 10 receives the imaging device ID and stores it in the storage unit 1000.

[0200] S22: The communication terminal 30A is notified of the completion of registration from the communication unit 51 of the information processing system 50, which allows user a to start associating the imaging device 10 with a virtual room. User a displays the virtual room association screen (part 1) 260 on the communication terminal 30A and selects the virtual room to which they want to associate the imaging device 10 registered with the tenant. The reception unit 32 of the communication terminal 30A receives the operation input indicating the selection. Specifically, when the reception unit 32 of the communication terminal 30A receives the operation input from user a, the display control unit 33 displays the virtual room association screen (part 1) 260. At this time, the communication unit 31 may send a screen update request to the communication unit 51 of the information processing system 50. When the information processing system 50 receives the update request, it refers to the tenant information storage unit 5003 and identifies the virtual room ID registered with the tenant associated with the authenticated user ID. Subsequently, it refers to the virtual room information storage unit 5002 and obtains the virtual room name associated with the identified virtual room ID. The information processing system 50's communication unit 51 transmits the identified virtual room ID and corresponding virtual room name information (which may also be information from a screen generated by the screen generation unit 52 based on this information) to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the virtual room ID and virtual room name information, and the display control unit 33 can update and display the virtual room mapping screen (part 1) 260 based on the received information. Note that such information can be identified based on the user ID, so it may also be received in S13 after authentication. The reception unit 32 receives an operation input from user a indicating a selection for the virtual room mapping screen (part 1) 260 displayed in this way, allowing the communication terminal 30A to identify the selected virtual room ID.

[0201] S23: Next, user a displays the virtual room mapping screen (part 2) 270 on the communication terminal 30A and presses the connect button 272 to map an additional device to a virtual room. The reception unit 32 of the communication terminal 30A receives the input indicating the press. Specifically, the display control unit 33 of the communication terminal 30A displays the virtual room mapping screen (part 2) 270 corresponding to the selected virtual room ID identified in S22. Furthermore, the reception unit 32 receives the instruction from user a to map an additional device to a virtual room (press of connect button 272).

[0202] S24: In response to the operation input in S23, the communication unit 31 of the communication terminal 30A requests information from the information processing system 50 regarding devices registered with tenants that are candidates for devices to be associated with the virtual room, and devices that have already been associated with the virtual room ID selected in step S22.

[0203] S25: The communication unit 51 of the information processing system 50 receives a request for information on devices registered with the tenant and devices associated with the selected virtual room ID, and the screen generation unit 52 generates a virtual room mapping screen (part 3) 280 that includes the device IDs of the devices registered with the tenant and devices associated with the selected virtual room ID. The communication unit 51 of the information processing system 50 transmits the screen information of the virtual room mapping screen (part 3) 280 to the communication terminal 30A.

[0204] S26: The communication unit 31 of the communication terminal 30A receives screen information from the virtual room mapping screen (part 3) 280, and the display control unit 33 displays the virtual room mapping screen (part 3) 280. User a selects a device to associate with the virtual room (here, the imaging device 10 is used as an example). The reception unit 32 of the communication terminal 30A accepts the selection, and the imaging device ID is identified as the device ID of the selected device.

[0205] S27: The communication unit 31 of the communication terminal 30A sends an association request to the information processing system 50, specifying the virtual room ID selected in step S22 and the device ID (e.g., imaging device ID) selected in S26.

[0206] S28: The communication unit 51 of the information processing system 50 receives the mapping request, and the communication group management unit 56 registers the device (e.g., imaging device 10) in the virtual room. That is, the communication group management unit 56 refers to the virtual room information storage unit 5002 and registers the device ID (e.g., imaging device ID) associated with the virtual room ID specified in the request of S27.

[0207] S29: Since a device ID (e.g., imaging device ID) has been associated with a virtual room, the communication unit 51 of the information processing system 50 transmits the virtual room ID, name, and description to the imaging device 10. The information processing system 50 may use push notifications or transmit the information by utilizing polling by the imaging device 10. The connection unit 16 of the imaging device 10 receives the virtual room ID, name, and description and stores them in the storage unit 1000. This allows the imaging device 10 to add the imaging device ID, virtual room ID, name, description, etc. when transmitting wide-field images. Other devices besides the imaging device 10 can also be associated with virtual rooms using the same procedure. Furthermore, the communication unit 51 of the information processing system 50 may send a notification to the communication terminal 30A indicating that the association is complete. From this step onward, devices registered with a virtual room (imaging device 10) can connect to the associated virtual room. Here, we will continue the explanation assuming that the imaging device 10 has connected to the virtual room by sending a connection request to the information processing system 50, specifying the virtual room ID received in S29. However, the timing of the imaging device 10 connecting to the virtual room can be changed by user operation.

[0208] S30: The communication terminal 30A and the information processing system 50 perform the entry process described in Figure 18, thereby allowing the communication terminal 30A to enter the virtual room associated with the device (imaging device 10).

[0209] S31: After entering the room, user a turns on the toggle button 291 of the imaging device 10 associated with the virtual room on the image viewing screen 210. The reception unit 32 of the communication terminal 30A receives the ON signal.

[0210] S32: The communication unit 31 of the communication terminal 30A sends a request to the information processing system 50 to start transmitting wide-field images, specifying the device ID (imaging device ID). Alternatively, user a may directly start transmitting wide-field images by operating a button on the imaging device 10. In addition, user a may also cause the communication unit 31 of the communication terminal 30A to send a request to stop transmission to the information processing system 50.

[0211] S33: The communication unit 51 of the information processing system 50 receives a transmission start request and requests the imaging device 10, identified by its device ID (imaging device ID), to start transmission. The information processing system 50 may use push notifications or the imaging device 10 may use polling. The connection unit 16 of the imaging device 10 receives the transmission start request and the imaging processing unit 13 starts imaging. The image transmission control unit 18 repeatedly transmits wide-field images via the connection unit 16 at a fixed FPS or an FPS corresponding to the bandwidth. Therefore, the communication terminal 30 that has entered the virtual room can display the status of site A on the image viewing screen 210 in real time.

[0212] <Distribution of wide-field images, etc.> Referring to Figure 27, the process by which wide-field images and normal-angle images are shared will be explained. Figure 27 is an example of a sequence diagram illustrating the process of sharing wide-field images. In Figure 27, communication terminals 30A and 30B have completed the entry process described in Figure 18 and have entered the virtual room. Communication terminal 30A also has a normal-angle camera 9, which is shared with communication terminal 30B. Instead of the camera 9 of communication terminal 30A, images captured by smart glasses 88 associated with the virtual room may also be shared. Also in Figure 27, the imaging device 10 has already been connected to the same virtual room through the registration procedure described in Figure 26.

[0213] S41: The imaging unit 34 of the communication terminal 30A takes an image of the surroundings, and the communication unit 31 specifies the virtual room ID in which the user is staying and transmits the video and audio, including the image obtained from the imaging, to the information processing system 50.

[0214] S42, S43: When the communication unit 51 of the information processing system 50 receives video and audio including images, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A and 30B that are in the same virtual room from the virtual room information storage unit 5002 and transmits the video and audio including images via the communication unit 51. In Figure 27, the communication unit 31 of the communication terminal 30A receives and displays an image with a normal field of view from the information processing system 50, but it may also display an image with a normal field of view captured by the imaging unit 34 without receiving it.

[0215] S44: Next, in response to a transmission start request based on the transmission start setting, the imaging device 10 captures a wide-field image, and the image transmission control unit 18 transmits the video and audio, including the wide-field image, to the information processing system 50 via the connection unit 16, specifying the virtual room ID, imaging device ID, name, and description to which the device is registered.

[0216] S45, S46: When the communication unit 51 of the information processing system 50 receives video and audio including wide-field images, the image distribution unit 54 obtains the IP addresses of communication terminals 30A and 30B that are in the same virtual room from the virtual room information storage unit 5002 and transmits video and audio including wide-field images via the communication unit 51.

[0217] S47: Next, the communication terminal 30C equipped with camera 9 entered the new virtual room by performing the entry process described in Figure 18.

[0218] S48: The communication unit 31 of the communication terminal 30C transmits video and audio, including images with a normal field of view, to the information processing system 50.

[0219] S49~S51: The communication unit 51 of the information processing system 50 receives video and audio, including images with a normal field of view, from the communication terminal 30C, obtains the IP addresses of the communication terminals 30A~30C that are in the same virtual room from the virtual room information storage unit 5002, and the image distribution unit 54 transmits video and audio, including images with a normal field of view.

[0220] S52: In addition, the communication unit 51 of the information processing system 50 transmits video, including wide-field images, and audio to the communication terminal 30C that has entered the same virtual room.

[0221] In this way, users a and b who have entered the same virtual room can share video, including wide-field images, captured by the imaging device 10 associated with the virtual room, in real time. Note that the transmission order of each image shown in Figure 27 is just an example; the wide-field images may be shared first, or the images with a normal field of view may be shared first.

[0222] Here, we will provide some additional information about the smart glasses 88 and VR goggles 89. The smart glasses 88 have a camera and display function with a normal field of view. Images with a normal field of view obtained by the camera held by the smart glasses 88 are distributed in the same way as cameras 8 and 9. The display function held by the smart glasses 88 is flat, like a normal display, so a portion of the wide-field image is displayed at the viewpoint indicated by the user. The VR goggles 89 have a display function (they may also have a camera with a normal field of view). The display function held by the smart glasses 88 projects a wide-field image at a viewpoint determined by the orientation of the user's head, so a predetermined region image Q of a predetermined region T in the wide-field image is displayed at a viewpoint corresponding to the orientation of the user's head. While viewing a wide-field image with the smart glasses 88 or VR goggles 89, the user can send an imaging request specifying the viewpoint information being viewed to the information processing system 50.

[0223] <Image capture by the imaging device based on instructions from a communication terminal> Next, referring to Figures 28 to 34, we will explain the process by which the imaging device 10 takes an image based on instructions from any communication terminal 30.

[0224] First, we will explain the screen used when a user requests imaging from the imaging device 10 by operating the communication terminal 30 during the sharing of wide-field images. While any user can instruct imaging, here we will assume that user b instructs imaging. Furthermore, since this is during the sharing of wide-field images, strictly speaking, the imaging device 10 is already "imaging" in real time. An imaging request during the sharing of wide-field images refers to capturing the wide-field image for saving purposes. Simply put, when wide-field images are being distributed in real time, the wide-field images are not saved and therefore cannot be reviewed later (a certain scene may not be saved arbitrarily).

[0225] Figure 28 shows an example of the image viewing screen 400 displayed by the communication terminal 30B. The explanation of Figure 28 primarily focuses on the differences from Figure 25(b). Communication terminals 30A and 30B each display video, including wide-field images, which are being streamed in real time. Both terminals also display video, including images with a normal field of view captured by the camera 9 of communication terminal 30A. The wide-field image video allows users at each location to arbitrarily change the virtual viewpoint (see Figure 11), thereby changing the predetermined area T (see Figures 9 and 10) of the displayed wide-field image. This allows for checking the situation on site in areas that cannot be seen with a normal field of view.

[0226] As shown in Figure 28, the first image panel 211, which displays a wide-field image, shows two types of imaging buttons: a still image save button 401a and a video recording button 401b. Hereafter, "imaging button 401" will be used as a collective term for the save button 401a and the video recording button 401b.

[0227] The save button 401a is a button used by the communication terminal 30B to send a save request (a type of capture request) for the wide-field image to the imaging device 10, using viewpoint information to identify a predetermined area of ​​the video, including the wide-field image (in this case, a wide-field still image) currently displayed in the first image field 211.

[0228] The recording button 401b is a button used by the communication terminal 30B to send a recording request (a type of shooting request) to the imaging device 10 to start recording a wide-field image (in this case, a wide-field video) from the present moment, using viewpoint information to identify a predetermined area of ​​the video, including the wide-field image currently displayed in the first image field 211.

[0229] Note that the imaging button 401 can be any GUI component capable of instructing imaging, and is not limited to the example shown in Figure 28.

[0230] Figure 29 shows the image viewing screen 400 on the communication terminal 30B, where the download button 402 is displayed after the image capture button 401 is pressed. In Figure 29, below the first image section 211, two types of download buttons (an example of display components) are displayed: a still image download button 402a, which is pressed when downloading a saved wide-field still image, and a video download button 402b, which is pressed when downloading a recorded wide-field video. Hereafter, "download button 402" will be used as a collective term for the still image download button 402a and the video download button 402b.

[0231] The still image download button 402a has embedded information about the storage location (e.g., URL) to which the wide-field image captured (saved) in response to the capture (save) request sent from the imaging device 10 when the user presses the save button 401a has been uploaded. When user b presses the still image download button 402a, the communication terminal 30B connects to the URL and can download the wide-field still image. Since viewpoint information is associated with the wide-field still image, when the communication terminal 30B displays the downloaded wide-field still image, it sets a virtual viewpoint so that a predetermined area of ​​the wide-field still image, identified by the viewpoint information, aligns with the center of the first image field 211 and displays it.

[0232] The video download button 402b has embedded information about the save location (e.g., URL) to which the wide-field video, captured (recorded) in response to an imaging (recording) request sent from the imaging device 10 when the user presses the save button 401b, is uploaded. When user b presses the video download button 402b, the communication terminal 30B connects to the URL and can download the wide-field video. Since viewpoint information is associated with the wide-field video, when the communication terminal 30B displays the downloaded wide-field video, it sets a virtual viewpoint so that a predetermined area of ​​the wide-field video, identified by the viewpoint information, aligns with the center of the first image field 211 and displays it.

[0233] It is not necessary to perfectly align the viewpoint with the center; the viewpoint may be set to be within the vicinity of the center of the first image field 211 for display. Here, we have explained an example in which a downloaded wide-field image is displayed in the first image field 211, which was displaying video containing a wide-field image being streamed in real time. However, it is also possible to continue displaying the video containing the wide-field image being streamed in real time in the first image field 211, add a new image field to the image viewing screen 400, and display the downloaded wide-field image in the new image field. In this way, it is possible to check the changes in the situation at the site using the video of the wide-field image being streamed in real time, while simultaneously checking a wide-field image that captures a specific situation at the site.

[0234] The appearance of the download button 402 is just one example; the download button 402 may display a message such as "360° image URL." Alternatively, instead of a button, a link corresponding to save location information (e.g., a URL) may be displayed. The user may also be able to download by clicking the link.

[0235] Furthermore, the communication terminal 30B may automatically receive and display the saved wide-field image and associated viewpoint information without displaying the download button 402.

[0236] Furthermore, Figure 29 shows the image viewing screen 400 displayed by the communication terminal 30B that requested the image capture, but the download button 402 may also be displayed on the image viewing screen 400 displayed by the communication terminal 30A that is displaying video including wide-field images being delivered in real time. In one embodiment, when user b performs an operation to instruct the sharing of the wide-field image captured by the image capture request, the download button 402 may be displayed on the communication terminals 30A of participants who are in the same virtual room. By doing so, the person who requested the capture (user b) can check the image they captured on their own device before sharing it with other participants, thus preventing the sharing of accidentally captured wide-field images or wide-field images that need to be shared. However, all communication terminals 30 in the virtual room may automatically display the download button 402 when saving wide-field images.

[0237] Referring to Figure 30, the explanation will assume that the communication terminal 30A displays the download button 402. Figure 30(a) is an example of the image viewing screen 410 that is displayed before user a presses the download button 402. Before user a presses the download button 402, the communication terminal 30A displays a wide-field image in the first image field 211 with a field of view that includes, for example, a work machine equipped with a drill, as an arbitrary virtual viewpoint specified by user a. In this state, user a presses the download button 402.

[0238] Figure 30(b) is an example of the image viewing screen 420 displayed on user a's side after user b presses the download button 402. The viewpoint information when user b presses the capture button 401 is the first image field 211 in Figure 28, so as shown in Figure 30(b), the downloaded wide-field image is displayed in the first image field 211 of the communication terminal 30A with the same viewpoint as the first image field 211 in Figure 28. Therefore, it displays a wide-field image with a field of view that allows confirmation of a scene where building materials are being lifted from the roof of a building by a crane, rather than a work machine equipped with a drill. In addition, user a can arbitrarily change the virtual viewpoint of the downloaded wide-field image displayed in the first image field 211 in Figure 30(b), just as with the video of the wide-field image delivered in real time. Therefore, even for a wide-field image that captures a specific scene, the viewpoint of the user who requested the capture can be reflected as the initial value, and specific scenes of the site can be confirmed in a range that cannot be confirmed with a normal field of view.

[0239] In this way, users at different locations can later share a virtual viewpoint of a wide-field image related to a specific scene. Note that the wide-field image saved via save request may be displayed as a pop-up instead of in the first image pane 211 in Figure 30(b), or it may be displayed in a separate window. This allows for efficient communication between users by using wide-field images of specific scenes while simultaneously monitoring the ever-changing situation on-site in real-time video feeds.

[0240] As shown in Figure 30(b), the imaging button 401 is not displayed while the communication terminal 30A is displaying a downloaded wide-field image. This is because the real-time wide-field image is not displayed in the first image field 211. In addition, the device name 214 in the first image field 211 displays the name of the imaging device 10 along with a statement indicating that it is a shared image. The device name 214 can be identified from the tenant registration device information stored in the tenant information storage unit 5003. Alternatively, the device name 214 may be a device ID.

[0241] Furthermore, on communication terminals 30A and 30B where the download button 402 is displayed, users a and b can arbitrarily delete the download button 402.

[0242] Furthermore, as shown in Figure 31, the communication terminals 30A and 30B may display a thumbnail image 404 of the wide-field image along with (and also function as) the download button 402. The thumbnail image 404 may be created by the information processing system 50 or by the communication terminal 30. The viewpoint of the thumbnail image 404 is determined by viewpoint information.

[0243] Furthermore, it is preferable that users a and b each be able to set a data name (registered in the image management information storage unit 3001 of each communication terminal 30) for the wide-field image captured in response to the imaging request.

[0244] Figure 32 shows an example of an image viewing screen 430 when there are three image fields. In the image viewing screen 430 of Figure 32, for example, a first image field 211 displays a wide-field image captured by the imaging device 10, a second image field 212 displays a normal-angle image captured by the camera 9 of the communication terminal 30A, and a third image field 431 displays a normal-angle image captured by the camera 8 of the communication terminal 30C. Any of users a, b, and c can press the capture button, and users a, b, and c at each location can share the wide-field image from the same viewpoint.

[0245] <<Actions or processing in response to imaging requests when sharing wide-field images>> Next, referring to Figure 33, the operation or processing in response to an imaging request from the communication system 1b during the sharing of a wide-field image will be explained. Figure 33 is an example of a sequence diagram explaining the process in which user b requests imaging from the imaging device 10 from the communication terminal 30B during the sharing of a wide-field image. In the explanation of Figure 33, it is assumed that communication terminals 30A and 30B have entered the virtual room through the entry process explained in Figure 18. Also, in the explanation of Figure 33, the case in which user b presses the imaging button 401 and shares viewpoint information with user a is described, but user a can also press the imaging button 401 and share viewpoint information with user b. In addition, in Figure 33, the imaging device 10 is already connected to the same virtual room through the registration procedure explained in Figure 26.

[0246] S61: For example, user b performs an operation on the image viewing screen 400 in Figure 28 to move (change) the virtual viewpoint of a wide-field image included in the video being delivered in real time from the imaging device 10 via the information processing system 50. The reception unit 32 of the communication terminal 30B receives an operation input indicating a viewpoint movement (change) operation, and the display control unit 33 displays a predetermined area of ​​the wide-field image identified based on the viewpoint information after the movement (change) in the first image field 211 as a display range corresponding to the viewpoint after the movement (change).

[0247] S62: Next, if user b wants to save a wide-field image (still image) of a specific scene, they press the save button 401a. Alternatively, if user b wants to start recording a wide-field image (video) from a specific scene, they press the record button 401b. Upon receiving the press, the reception unit 32 of the communication terminal 30B identifies the viewpoint information corresponding to the viewpoint currently set as the viewpoint information when instructing imaging (saving or recording). Pressing the imaging button 401 is one example of a method for sending an imaging request, and imaging requests may also be sent in response to other inputs to the user interface. The user interface includes not only GUIs but also voice and gesture interfaces.

[0248] S63: The communication unit 31 of the communication terminal 30B transmits an imaging request to the information processing system 50 by specifying the identified viewpoint information, the virtual room ID in which the user has entered, and the imaging device ID corresponding to the imaging device that instructed the imaging request.

[0249] S64: The communication unit 51 of the information processing system 50 receives the imaging request, and the association processing unit 53 transmits the imaging request by specifying the received viewpoint information to the imaging device 10 identified by the imaging device ID. This imaging request may be a response to the imaging device 10 that transmitted the wide - field - of - view image, or may be a push notification.

[0250] S65: The connection unit 16 of the imaging device 10 receives the imaging request, and the imaging processing unit 13 performs imaging according to the received imaging request. When the imaging request indicates a request to save a wide - field - of - view image (still image), the imaging processing unit 13 starts saving. On the other hand, when the imaging request indicates a request to record a wide - field - of - view image (video), the imaging processing unit 13 starts recording.

[0251] Since the imaging device 10 captures images in real - time and distributes the video of the wide - field - of - view image, if it is unable to perform imaging according to the imaging request in parallel, the imaging processing unit 13 may control to temporarily stop the video distribution of the wide - field - of - view image, perform imaging according to the imaging request, and then start the video distribution of the wide - field - of - view image again. The imaging device 10 can perform imaging to obtain a wide - field - of - view image without using the viewpoint information. The imaging processing unit 13 may capture a still image or a video. In the case of a video, it may capture images for a predetermined time (e.g., 10 seconds), or the user b may specify the recording time. Also, when capturing a video, the imaging device 10 may capture the video by recording the video being distributed while distributing the video of the wide - field - of - view image in real - time.

[0252] S66: The storage processing unit 17 of the imaging device 10 transmits a request to inquire of the information processing system 50 about the storage destination information (e.g., URL) of the upload destination of the wide - field - of - view image via the connection unit 16.

[0253] S67: The communication unit 51 of the information processing system 50 receives the query request, and the association processing unit 53 creates destination information (e.g., a URL) targeting the storage 90 configured in the virtual room. Specifically, the association processing unit 53 refers to the virtual room information storage unit 5002 and identifies the virtual room ID associated with the device ID (imaging device ID) of the imaging device 10, which is the querying device. Furthermore, the association processing unit 53 obtains information about the storage 90 associated with the identified virtual room ID. As information about the storage 90, for example, address information of a predetermined storage 90 or information about folders on the storage can be obtained as destination information (e.g., a URL). For services for construction sites, information about folders classified appropriately for the site to which the service is applied may be prepared in advance, such as information about folders associated with construction work. The communication unit 51 of the information processing system 50 transmits the obtained destination information (e.g., a URL) to the imaging device 10. The communication unit 51 may also transmit the destination information (e.g., a URL) to the imaging device 10 together with the imaging request in S64.

[0254] S68: The connection unit 16 of the imaging device 10 receives destination information (e.g., URL), and the storage processing unit 17, via the connection unit 16, saves the wide-field image data (still image or video) captured in S65 and the viewpoint information corresponding to the virtual viewpoint set in the communication terminal 30B to the destination indicated by the destination information (e.g., URL), associating them. Note that the association process here does not need to be anything other than a process that links the saved wide-field image so that the viewpoint information can be reflected when viewing it. The viewpoint information may be added to the wide-field image as metadata, or the viewpoint information may be managed as separate data from the wide-field image and a common identification information may be assigned so that the correspondence can be identified, or the storage location of the viewpoint information may be assigned to the wide-field image as metadata. Alternatively, the association process may be performed by embedding the viewpoint information in a predetermined data item specified in a standardized wide-field image format. Furthermore, since the viewpoint information is also stored in the image management information storage unit 5001 as described later, only the wide-field image may be stored in the storage 90.

[0255] S69: The storage processing unit 17 of the imaging device 10 sends a notification to the information processing system 50 via the connection unit 16 indicating that saving to the storage destination information (e.g., URL) is complete. At this time, the imaging device 10 may also receive information indicating the date and time of acquisition of the wide-field image captured by the imaging device 10.

[0256] S70: When the communication unit 51 of the information processing system 50 receives a notification indicating that saving is complete, the association processing unit 53 performs a process to associate and manage the series of information received in step S63 and the information received from the imaging device 10, etc., with the image management information storage unit 5001 and saves them. That is, the association processing unit 53 assigns a data ID and manages the imaging date and time information (which may be the date and time information when the imaging device 10 actually took an image, or the date and time information when the communication terminal sent the imaging request), imager information (for example, linked to the IP address of the communication terminal 30B), imaging device information (obtained in step S63), imager's viewpoint information, virtual room ID at the time of imaging (obtained in step S63), and data storage location information (URL) with the image management information storage unit 5001. Note that the virtual room ID at the time of imaging, imaging device information, and imager information may be identified by the association processing unit 53 by referring to the virtual room information storage unit 5002. The virtual room ID at the time of imaging is the virtual room ID of the virtual room in which the communication terminal that requested the imaging is located. The imaging device information is the imaging device information associated with the identified virtual room ID. The imaging user information is the user information associated with the communication terminal that sent the imaging request, from among the user information currently in the virtual room associated with the identified virtual room ID. The data name may be arbitrarily set by user b or automatically set by the information processing system 50.

[0257] S71: The communication unit 51 of the information processing system 50 sends a notification to the communication terminal 30B indicating that saving is complete, specifying the storage destination information (e.g., URL). The communication unit 51 also sends the information saved in association with the image management information storage unit 5001 to the communication terminal 30B. Based on the received information, the communication terminal 30B can construct the image management information storage unit 3001 in the storage unit 3000.

[0258] S72: When the communication unit 31 of the communication terminal 30B receives a notification indicating that saving is complete, the display control unit 33 displays the download button 402 in Figure 29. If the save location information is not automatically shared with all users currently in the room, user b instructs the communication terminal 30B to perform a sharing operation to share the save location information. In this way, user b can share only the wide-field images that are deemed to be worth sharing with user a who is in the virtual room. Specifically, the reception unit 32 of the communication terminal 30B receives an operation input indicating a sharing operation.

[0259] S73: In response to receiving an operation input indicating a sharing operation, the communication unit 31 of the communication terminal 30B sends a sharing request to the information processing system 50, specifying the storage destination information (e.g., URL) and the virtual room ID.

[0260] S74: When the communication unit 51 of the information processing system 50 receives a sharing request, the association processing unit 53 refers to the virtual room information storage unit 5002, identifies the communication terminal 30A currently in the virtual room based on the virtual room ID, and the communication unit 51 sends the storage destination information (e.g., URL) to the communication terminal 30A. When the communication unit 31 of the communication terminal 30A receives the storage destination information (e.g., URL), the display control unit 33 displays the download button 402. If the sharing operation is not required, steps S72 and S73 may be omitted, and step S74 may be executed without receiving a sharing request.

[0261] The communication unit 51 transmits the information stored in association with the image management information storage unit 5001 to the communication terminal 30A. Based on the received information, the communication terminal 30A can construct the image management information storage unit 3001 in the storage unit 3000.

[0262] S75, S76: Users a and b press the download button 402. The reception units 32 of the communication terminals 30A and B receive the input indicating that the button has been pressed. Note that if user b is already viewing a wide-field image from the viewpoint they wish to share, they do not need to press the download button 402 again; it is sufficient to simply display a notification on the screen indicating that the save location information has been shared with the other user.

[0263] S77, S78: The communication units 31 of the communication terminals 30A and B connect to the destination information (e.g., URL) embedded in the download button 402 and request the wide-field image and viewpoint information associated with the destination information (e.g., URL) from the storage 90.

[0264] S79,80: The communication units 31 of the communication terminals 30A and B receive wide-field images from the storage 90. The display control units 33 of the communication terminals 30A and B, while receiving video including wide-field images distributed in real time from the imaging device 10, display the wide-field images received from the storage 90 in the first image field 211 or as a pop-up display. In the initial state for displaying the wide-field image, the display control unit 33 sets the viewpoint where a predetermined area of ​​the wide-field image, identified by viewpoint information, is displayed to coincide with the center of the first image field 211 and displays it. Note that it is not necessary to perfectly coincide the viewpoint with the center; the viewpoint may be set to be included in the range near the center of the image field and displayed. The field of view may be within a predetermined range. The user may set an initial value for the field of view. Also, if the wide-field image is a video, the entire file may be downloaded or it may be streamed from the storage 90.

[0265] Similarly, when the smart glasses 88 are the communication terminal 30, in the initial state for displaying the received wide-field image, the viewpoint where a predetermined area of ​​the wide-field image identified by the viewpoint information is displayed is set to coincide with the center of the display function in front of the eyes. Similarly, when the VR goggles 89 are the communication terminal 30, in the initial state for displaying the received wide-field image, the viewpoint of the wide-field image identified by the viewpoint information is set to coincide with the center of the display function in front of the eyes, regardless of the orientation of the head. In either case, however, it is not necessary to perfectly align the viewpoint with the center; the viewpoint may be set to be included in the range near the center of the display function for display.

[0266] In Figure 33, communication terminals 30A and 30B press the download button 402 (URL for uploading wide-field images) received from the information processing system 50 to obtain wide-field images from storage 90. However, as shown in Figure 34, the information processing system 50 may directly transmit the wide-field images and viewpoint information to communication terminals 30A and 30B.

[0267] Figure 34 is a modified sequence diagram illustrating the process in which user b requests imaging from the imaging device 10 via the communication terminal 30 when sharing a wide-field image. Note that the explanation of Figure 34 will primarily focus on the differences from Figure 33.

[0268] In Figure 34, in step S91, the imaging device 10 returns the wide-field image and viewpoint information to the information processing system 50. The communication unit 51 of the information processing system 50 performs a process to associate the wide-field image and viewpoint information received by the association processing unit 53, and then transmits the received wide-field image and associated viewpoint information to the communication terminals 30A and 30B in the same virtual room as the imaging device 10 (S92, S93). Note that the viewpoint information to be associated may be the viewpoint information received in step S63, not the viewpoint information received in step S91. In this way, users a and B can display the wide-field image from the same viewpoint without having to press the download button 402 to display the wide-field image. In the case of Figure 34, each communication terminal 30A and 30B may first display the wide-field image as a thumbnail and then enlarge the wide-field image according to user operation.

[0269] In Figure 34, the storage location information (e.g., URL) is not sent to the communication terminals 30A and 30B, but the information processing system 50 may send the storage location information (e.g., URL) to the communication terminals 30A and 30B. Even if the storage location information (e.g., URL) is not sent to the communication terminals 30A and 30B, the storage 90 is associated with each virtual room (see Figure 23), so the user can connect the communication terminal 30 to the storage 90 and view the saved wide-field images with the associated viewpoint information reflected at any time.

[0270] <Viewing wide-field images registered in external storage> Next, referring to Figures 35 to 37, we will explain how to view information stored in the external storage 90, including wide-field images registered in the storage 90. Figure 35 is an example of a classification selection screen 600 displayed when any user connects a communication terminal 30 to the information processing system 50. The classification selection screen 600 is created using the image management information stored in the image management information storage unit 5001 in Figure 14, the virtual room information storage unit 5002 in Figure 15, the tenant information storage unit 5003, etc. When a user logs in to a tenant, a classification selection screen classified by the information associated with this tenant is displayed. Note that it is not always necessary to log in to a tenant; users can also log in as an account not associated with a tenant. In such cases, a selection screen classified based on the data for which the logged-in user has access rights among the image management information stored in the image management information storage unit 5001 may be displayed. A user's access rights to the image management information may be updated by connecting a predetermined communication terminal 30 to the information processing system 50 and sending an access rights update instruction from the communication terminal 30 to the information processing system 50. Various classifications can be applied to the selectable classifications. Figure 35 illustrates an example of a classification, showing a selection screen classified based on the virtual room ID stored in the image management information storage unit 5001 during imaging.

[0271] The classification selection screen 600 has a header field 601 and an image selection field 602. The header field 601 displays the login username 603 and the classification list (list) 604. The login username 603 is information that identifies the user logged into the tenant and includes the user's name and user ID. The classification list 604 is a list of classifications based on the information stored in the image management information storage unit 5001. Here, as an example of classification, data including wide-field images stored in storage 90 can be classified according to which virtual room they are associated with. For example, each wide-field image can be identified by the virtual room ID at the time of acquisition, which is stored in the image management information storage unit 5001, indicating which virtual room it was captured in during remote communication. In the classification list 604 of Figure 35, in addition to the name of the virtual room where remote communication took place, the date on which the remote communication took place (corresponding to the date and time information on when the image was captured) is also displayed. In the example in Figure 35, for instance, remote communication takes place on the date "October 12, 2021" in a virtual room named "Construction Site A," and the images captured during this remote communication are stored in the classification of the virtual room named "Construction Site A" on the date "October 12, 2021." If information on images captured on a date other than "October 12, 2021" in the virtual room named "Construction Site A" is stored in the image management information storage unit 5001, then the classification list 604 can display classification options that combine a date other than "October 12, 2021" (for example, October 13, 2021) with the virtual room name "Construction Site A." Users can edit the virtual room name by accessing the virtual room information storage unit 5002 of the information processing system 50 using the communication terminal 30, and the virtual room name is set when the virtual room is created and stored in the virtual room information storage unit 5002. By referring to the virtual room information storage unit 5002 based on the virtual room ID stored in the image management information storage unit 50001 at the time of imaging, the corresponding virtual room name can be identified.The date can be specified from the imaging date and time of the image management information stored in the image management information storage unit 5001 (if there are multiple, the first date and time). In FIG. 35, it is classified by virtual rooms, but it may also be classified by other classifications such as date, date and time, imager information, imaging device information, or data name. Also, it may be classified by a combination of two or more of these classifications.

[0272] The image selection column 602 has image management information 605 and thumbnail images 606. When the user selects an arbitrary classification (in this example, a virtual room) from the classification list 604, the thumbnail image 606 is displayed in the image selection column 602 as information on the wide-view image captured during remote communication in the selected virtual room. In the example of FIG. 35, it shows that the classifications of the date "October 12, 2021" and the virtual room name "Construction Site A" are in the selected state. In the example of FIG. 35, it is classified by a combination of the classification of the virtual room and the date on which remote communication was performed in the virtual room, but the classification list 604 may be created by classifying only by the virtual room regardless of the date, or the classification list 604 may be created by classifying only by the date. Also, the wide-view image may be a still image or a moving image. In the case of a moving image, the thumbnail image 606 may be a wide-view image at any timing from the start to the end of the recording.

[0273] Also, the thumbnail image 606 is displayed with the viewpoint of the wide-view image specified by the viewpoint information aligned with the center of the thumbnail image. Therefore, the user viewing the classification selection screen 600 can view the thumbnail image with the viewpoint information that the imager wanted to share. Note that the viewpoint does not necessarily have to be aligned with the center, and it may be displayed with a viewpoint included in the range near the center.

[0274] The user can select a predetermined thumbnail image 607 from among multiple thumbnail images 606 using the cursor c1. When the user clicks or taps while the image is selected with the cursor c1, the classification selection screen 600 transitions to the wide-field image viewing screen 610. Note that selection using the cursor c1 is merely an example; selection can also be made using other user inputs such as touch input on a touch panel or key input via a keyboard. Furthermore, the thumbnail image 606 is just an example of image information used to identify the image to be viewed. Image information such as the image data name and the date and time of capture may be displayed in the image selection field 602, and the user can transition to the wide-field image viewing screen by selecting this image information.

[0275] Figure 36 shows an example of the wide-field image viewing screen 610. The explanation of Figure 36 mainly describes the differences from Figure 35. The wide-field image viewing screen 610 has a wide-field image display unit 614. The wide-field image display unit 614 displays the acquisition date and time 611 and the data name 612. The communication terminal 30 can edit the data name 612 by receiving editing operation input from the user. The initial value of the data name 612 may be left blank.

[0276] In Figure 36, the wide-field image display unit 614 is shown in an example where a wide-field image is selected that shows a field worker measuring a construction site with a measuring tape. In this way, by viewing the content displayed on the wide-field image display unit 614, it is possible to understand the situation at the site even without participating in remote communication. Therefore, the content displayed on the wide-field image display unit 614 itself can be used as a field work report, educational content, or entertainment content.

[0277] The wide-field image display unit 614 displays a wide-field image 613 corresponding to the image information selected by the user in Figure 35 (thumbnail image 606 in this example). Immediately after the wide-field image 613 is displayed on the wide-field image display unit 614, the viewpoint of the wide-field image, specified by the viewpoint information, is aligned with the center of the wide-field image display unit 614 and displayed. The communication terminal 30 then receives operation input from the user and performs display control to arbitrarily change the viewpoint of the wide-field image 613 displayed on the wide-field image display unit 614 in the up, down, left, and right directions, allowing areas not displayed in the thumbnail image 606 to be displayed for viewing. Note that the viewpoint does not necessarily have to be aligned with the center; it may be displayed with a viewpoint that is included in the range near the center. Also, changes to the viewpoint may be accepted only in the up and down direction, or only in the left and right direction.

[0278] Furthermore, if the wide-field image 613 selected by the user in Figure 35 is a video, a menu will be displayed with options such as play, stop, fast forward, and rewind.

[0279] Furthermore, the lower left area of ​​the wide-view image display unit 614 displays the "frame extraction mode" button b1, the "still image playback mode" button b2, and the "video playback mode" button b3.

[0280] The "Frame Extraction Mode" button b1 is pressed to switch to the frame extraction mode screen (see Figures 39 and 40) for extracting wide-field still images as frames from recorded wide-field video. When this button b1 is pressed, viewpoint information is also generated to identify a predetermined area in the displayed wide-field video, and this viewpoint information is carried over to the still image playback mode and video playback mode.

[0281] The "Still Image Playback Mode" button b2 is pressed when switching to the still image playback mode screen (see Figures 41 to 44), which displays a still image of a predetermined area based on viewpoint information from the wide-field still images extracted by the frame extraction mode.

[0282] The "Video Playback Mode" button b3 is pressed when switching to a video playback mode screen (see Figures 45 to 48) that plays back and displays a recorded wide-field video, and then displays a still image of a predetermined area (a predetermined area in a wide-field still image extracted in frame extraction mode) from this wide-field video based on viewpoint information.

[0283] Furthermore, in the state shown in the thumbnail image 606 of Figure 35, the user may be able to arbitrarily change the viewpoint of the wide-field image.

[0284] Figure 37 is an example of a sequence diagram illustrating the process by which a user views a wide-field image stored on external storage.

[0285] S101: With the user accessing the information processing system 50 via the communication terminal 30, the communication terminal 30 accepts authentication information (user ID, password, etc.) from the user and accepts the user's request to log in to the tenant to which they belong. The reception unit 32 of the communication terminal 30 accepts these inputs.

[0286] S102: The communication unit 31 of the communication terminal 30 sends a login request to the information processing system 50, specifying the authentication information. The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 55 performs authentication based on the authentication information. Here, we assume that the authentication was successful.

[0287] S103: The screen generation unit 52 of the information processing system 50 generates a classification selection screen 600 that classifies wide-field images according to predetermined classifications, and the communication unit 51 transmits the screen information of the generated classification selection screen 600 to the communication terminal 30. Specifically, it generates a classification selection screen 600 that displays a list of classifications as selection candidates based on the image management information stored in the image management information storage unit 5001 in Figure 14, the virtual room information storage unit 5002 in Figure 15, the tenant information storage unit 5003, etc. In the example in Figure 35, as an example of classification, it generates a classification selection screen 600 that includes a list (classification list 604) classified according to which virtual room the data containing wide-field images stored in storage 90 is associated with. First, the tenant ID of the tenant to which the user ID authenticated in S102 belongs is identified by referring to the tenant information storage unit 5003 shown in Figure 15. In the tenant information storage unit 5003, virtual rooms are registered for each tenant. Wide-field images captured and saved by an imaging device associated with a virtual room in response to a participant's request during remote communication in the virtual room are stored in the image management information storage unit 5001, associated with the virtual room ID of the virtual room where the remote communication took place, the person who requested the image (participant), the date and time of capture, the imaging device, viewpoint information, storage location, etc. Therefore, the screen generation unit 52 can identify the virtual room ID and corresponding virtual room name associated with the tenant ID of the tenant to which the logged-in user belongs by referring to the tenant information storage unit 5003. In this way, the virtual room name to be displayed in the classification list 604 shown in Figure 35 can be identified. Furthermore, when displaying the classification of the virtual room associated with the saved data in combination with the date of capture, as shown in Figure 35, the screen generation unit 52 can identify the data ID associated with each identified virtual room ID by referring to the image management information storage unit 5001. Furthermore, the screen generation unit 52 can identify the date and time of capture information associated with the identified data ID by referring to the image management information storage unit 5001.The screen generation unit 52 can identify the date from the information of the imaging date and time identified in this way, and can generate a classification selection screen 600 that displays the identified date associated with the virtual room to be displayed in the classification list 604, as shown in Figure 35. By combining the date in this way, for example, if multiple wide-field images taken in a virtual room named "Construction Site A" are stored, and the dates on which they were taken are different, they can be displayed in the classification list as different classifications. This is useful when you want to narrow down the candidates for wide-field images that can be selected based on the date on which the work was performed, for example, because even wide-field images taken at the same site may have different work performed on different dates. Similarly, instead of the date, information such as the photographer, imaging device, or storage location can be combined and displayed with the classification of the virtual room by referring to the image management information storage unit 5001. Displaying it in this way is useful when you want to narrow down the candidates for wide-field images that can be selected based on conditions such as the photographer, imaging device, or storage location, even if they were taken at the same site. Furthermore, these classification conditions can be used to generate a classification list 604, each classified as a single condition, or, as shown in Figure 35, a classification list 604 can be generated by combining two or more conditions. In this example, the screen generation unit 52 generates the classification selection screen 600 (classification list 604), but the screen generation unit 52 may also transmit classification information to be displayed in the classification list 604 (in this example, the virtual room name and date that match the classification conditions) to the communication terminal 30 via the communication unit 51, thereby causing the communication terminal 30 to generate a classification selection screen 600 with a classification list 604 based on the classification information. Therefore, the screen information transmitted in S103 may include the classification selection screen 600 generated by the screen generation unit 52, or it may include classification information to be displayed in the classification list 604 instead of the generated screen. Alternatively, both the generated classification selection screen 600 and the classification information to be displayed in the classification list 604 may be included in the screen information.

[0288] S104: The communication unit 31 of the communication terminal 30 receives screen information from the classification selection screen 600, and the display control unit 33 displays the classification selection screen 600. Specifically, the selectable classifications (in this example, virtual rooms associated with dates) are displayed in the header column 601 of Figure 35. The communication terminal 30 receives operation input from the user corresponding to the selection of a classification (in this example, a virtual room associated with a specific date) associated with the wide-view image they wish to view. The reception unit 32 receives these operation inputs.

[0289] S105: The communication unit 31 of the communication terminal 30 sends a request for a list of wide-field images to the information processing system 50, specifying identification information that identifies the classification selected by the user (in this example, the virtual room ID and date information of the selected virtual room) as conditions. Note that the conditions such as the virtual room ID and date do not necessarily have to be specified at the same time; they may be specified separately and the request sent.

[0290] S106, S107: The communication unit 51 of the information processing system 50 receives a request for a list of wide-field images. The screen generation unit 52 identifies the data IDs associated with the virtual room IDs included in the conditions specified in the request from the image management information storage unit 5001. Furthermore, since the date is also specified as a condition in the received request, the image management information storage unit 5001 identifies the data IDs associated with the date of acquisition that matches the date specified in the image management information storage unit 5001 from among the identified data IDs. For each data ID that matches the specified conditions in this way, the image management information storage unit 5001 obtains information on the storage location of the associated data. The screen generation unit 52 accesses each storage location in the external storage 90 using the obtained storage location information via the communication unit 51 and obtains the wide-field images and viewpoint information from the storage 90. The screen generation unit 52 may also obtain and use the viewpoint information stored in the image management information storage unit 5001.

[0291] S108: The screen generation unit 52 of the information processing system 50 creates thumbnail images using the wide-field images and viewpoint information acquired in S106 and S107. Furthermore, it acquires image information (information such as imaging device, imager, and virtual room participants) related to data IDs that match the conditions specified in S105 from the image management information storage unit 5001, and generates a classification selection screen 600 on which the thumbnail images and image information are arranged based on the acquired image information and the created thumbnail images. In this example, the screen generation unit 52 generates the classification selection screen 600, but the screen generation unit 52 may also transmit the thumbnail images and image information to the communication terminal 30 via the communication unit 51, causing the communication terminal 30 to generate the classification selection screen 600 based on the thumbnail images and image information.

[0292] S109: The communication unit 51 transmits the screen information of the classification selection screen 600 to the communication terminal 30. The screen information here may be the classification selection screen 600 generated in S108, or it may be the information necessary for the communication terminal 30 to generate the classification selection screen 600 (such as the thumbnail images generated in S106 to S108 and the acquired image information). The screen information also includes identification information (data ID in this example) used to identify the data corresponding to each of the created thumbnail images, which was identified in S106 and S107.

[0293] S110: The communication unit 31 of the communication terminal 30 receives screen information of the classification selection screen 600, and the display control unit 33 displays the updated classification selection screen 600 based on the received screen information. If the received screen information is necessary for the communication terminal 30 to generate the classification selection screen 600, the communication terminal 30 executes a screen update process to update the classification selection screen 600 displayed by the communication terminal 30 based on the received screen information. Specifically, the image selection field 602 shown in Figure 35 is updated with image information and thumbnail images associated with the selected classification. The communication terminal 30 receives operation input from the user to select a thumbnail image of a wide-view image that they wish to view from the updated image selection field 602. The reception unit 32 receives these operation inputs.

[0294] S111: The communication unit 31 of the communication terminal 30 identifies the data ID of the wide-field image corresponding to the thumbnail image selected by the user based on the operation input received in S110, and sends a request for the wide-field image to the information processing system 50, specifying the identified data ID.

[0295] S112, S113: The communication unit 51 of the information processing system 50 receives a request for a wide-field image. The screen generation unit 52 obtains information (e.g., URL) about the storage location of the data associated with the data ID specified in the request from the image management information storage unit 5001. The screen generation unit 52 obtains the wide-field image and viewpoint information from the external storage 90 using the storage location information via the communication unit 51. Note that the viewpoint information from the image management information storage unit 5001 may be used. Also, if the wide-field image and viewpoint information corresponding to the data ID specified in the request have already been obtained in S107, the wide-field image and viewpoint information obtained in S107 may be used.

[0296] S114: The screen generation unit 52 of the information processing system 50 uses the acquired wide-field image and viewpoint information to generate an image in which a predetermined area of ​​the wide-field image identified based on the viewpoint information is aligned with the center of the wide-field image display unit 614 so as to reflect the viewpoint of the wide-field image identified by the viewpoint information, and generates a wide-field image viewing screen 610 by placing the generated image on the wide-field image display unit 614. Note that the viewpoint does not necessarily have to be aligned with the center, and may be displayed with a viewpoint included in the range near the center. In this example, the screen generation unit 52 of the information processing system 50 generates the wide-field image viewing screen 610, but the wide-field image and viewpoint information may be transmitted to the communication terminal 30 by the communication unit 51 as information necessary to generate the wide-field image viewing screen 610, so that the communication terminal 30 generates the wide-field image viewing screen 610.

[0297] S115: The communication unit 51 transmits screen information of the wide-field image viewing screen 610 to the communication terminal 30. The screen information here may be the wide-field image viewing screen 610 generated in S114, or it may be information necessary for the communication terminal 30 to generate the wide-field image viewing screen 610 (such as the wide-field image and viewpoint information acquired in S113). The screen information also includes image information related to the data ID corresponding to the selected thumbnail image (information such as the date and time of acquisition, the imaging device, the person who took the image, and the participants in the virtual room). Such image information can be identified by referring to the image management information storage unit 5001 based on the specified data ID.

[0298] S116: The communication unit 31 of the communication terminal 30 receives screen information of the wide-field image viewing screen 610, and the display control unit 33 displays the wide-field image viewing screen 610 based on the received screen information. If the received screen information is information necessary for the communication terminal 30 to generate the classification selection screen 600, the display control unit 33 performs a screen update process to update the classification selection screen 600 displayed by the communication terminal 30 to the classification selection screen 600 based on the received screen information. Specifically, the wide-field image display unit 614 shown in Figure 36 generates an image that displays a predetermined area of ​​the wide-field image identified based on the received viewpoint information, so that the viewpoint identified based on the received viewpoint information is reflected in the display, and the display control unit 33 displays the generated image. The display control unit 33 of the communication terminal 30 receives an operation input from the user via the reception unit 32 to arbitrarily change the viewpoint of the wide-field image displayed on the wide-field image viewing screen 610. Based on the received wide-field image and viewpoint information corresponding to the changed viewpoint, the display control unit 33 performs a screen update process to update a predetermined area of ​​the wide-field image displayed on the wide-field image display unit 614 so that a predetermined area of ​​the wide-field image corresponding to the changed viewpoint is displayed. If the information processing system 50 also performs the update process of the wide-field image viewing screen 610, this can be achieved by the communication terminal 30 sending a request for screen update processing to the information processing system 50 specifying the changed viewpoint information, and the communication terminal 30 receiving the wide-field image viewing screen 610 in which the predetermined area of ​​the wide-field image displayed by the screen generation unit 52 of the information processing system 50 has been updated based on the changed viewpoint information.

[0299] <Processing for each mode> Next, we will explain each mode using Figures 36, 38, and 48.

[0300] <<Frame extraction mode>> The frame extraction mode will be explained using Figures 36, 38, and 40. Figure 38 is a sequence diagram showing the processing of the frame extraction mode. Figure 39 is a conceptual diagram of the frame extraction mode screen. Figure 40 is a conceptual diagram of the frame extraction mode screen. In this case, the communication terminal 30 has an image editing application (editing app) installed, and the display control unit 33 is realized by the editing app. The communication terminal 30 in this case can also be referred to as the display terminal.

[0301] S201: When any user (in this case, user b) presses the "frame extraction mode" button b1 shown in Figure 36 with a mouse or the like, the reception unit 32 accepts the transition to frame extraction mode.

[0302] S202: The display control unit 33 displays a frame extraction mode screen on the display 306, as shown in Figure 39. The left area of ​​this screen displays the predetermined area video. Also displayed is the extraction button (camera button) b11, which is pressed when extracting a wide-field still image from the original wide-field video of the predetermined area video currently being played. At the bottom are the play button b12, which is pressed when playing the predetermined area video, and the stop button b13, which is pressed when stopping the predetermined area video that is being played. In addition, the right area of ​​the frame extraction mode screen displays the extracted wide-field still image when the extraction button b11 is pressed. However, instead of the entire wide-field still image, the predetermined area still image of the same predetermined area as the predetermined area video currently being played in the wide-field video is displayed.

[0303] S203: When user b moves the cursor c1 to the play button b12 with a mouse or the like and presses it, the reception unit 32 receives the playback request.

[0304] S204: The display control unit 33 plays and displays a predetermined area of ​​video in the left area of ​​the frame extraction mode screen.

[0305] S205: When user b performs an operation to move the virtual viewpoint over a predetermined area of ​​video using a mouse or the like, the reception unit 32 receives the operation to move the virtual viewpoint.

[0306] S206: The display control unit 33 performs a modified display of the predetermined region video after the viewpoint movement. This is a process in which, in Figure 10, the viewpoint of the virtual camera IC in Figure 10(a) is moved (changed) as in Figure 10(c), causing the display control unit 33 to change the predetermined region image in Figure 10(b) to the predetermined region image in Figure 10(d).

[0307] S207: When user b places the cursor c1 on the crop button b11 with a mouse or the like and presses it, the reception unit 32 accepts the cropping of a wide-field still image.

[0308] S208: The display control unit 33 extracts a wide-field still image, including the currently displayed predetermined region still image, from the wide-field video which is the source of the currently displayed predetermined region video. At this time, the display control unit 33 generates a still image ID to identify the extracted wide-field still image. The display control unit 33 also obtains the elapsed playback time of the currently displayed video out of the total playback time of the video, and viewpoint information (predetermined region information) to identify the predetermined region in the wide-field video of the currently displayed predetermined region video. This process is as described above using Figure 51.

[0309] S209: The communication unit 31 transmits playback-related information and data of the wide-field still image extracted in step S208 to the information processing system 50. The transmitted playback-related information includes the information obtained in step S208, namely the generated still image ID, the video ID of the wide-field video from which the image was extracted, information on the playback elapsed time indicating the time of extraction, and viewpoint information (predetermined region information) for identifying a predetermined region in the extracted predetermined region video. As a result, the communication unit 51 of the information processing system 50 receives the playback-related information and the wide-field still image.

[0310] S210: In the information processing system 50, the association processing unit 53 determines the storage location information (URL, etc.) for the wide-field still image in the storage 90. The storage and reading unit 59 stores the playback-related information, including the storage location information for the wide-field still image determined by the association processing unit 53, as one record in the playback-related information storage unit 5004.

[0311] S211: The communication unit 51 accesses the storage location information for the wide-field still image determined by the association processing unit 53 and saves the data of the wide-field still image received in step S209.

[0312] With the above steps, the processing of the frame extraction mode shown in Figure 38 is completed.

[0313] <<Still Image Playback Mode>> Next, the still image playback mode will be explained using Figures 36 and 41 to 44. Figure 41 is a flowchart illustrating the main processing of the communication terminal in still image playback mode. Figure 42 is a sequence diagram showing the detailed processing of the still image playback mode. Figure 43 is a conceptual diagram of the screen that plays a predetermined area of ​​video from a predetermined area of ​​still images in still image playback mode. Figure 44 is a conceptual diagram of the screen that returns from a predetermined area of ​​video to a predetermined area of ​​still images in still image playback mode. In this case, the communication terminal 30 has a still image viewer 30a and a video viewer 30b installed, and the communication unit 31, the reception unit 32, and the display control unit 33 execute processing according to the instructions of the still image viewer 30a and the video viewer 30b. The communication terminal 30 in this case can also be referred to as a display terminal. It is not necessary to install the still image viewer 30a and video viewer 30b in advance; the communication terminal 30 may have equivalent functionality, or the display terminal 30 may receive a program with equivalent functionality to the still image viewer 30a and video viewer 30b from the information processing system 50 or other servers, thereby allowing the communication terminal 30 to execute the processing described in Figures 42 and 46. Furthermore, the still image viewer 30a and video viewer 30b of the communication terminal 30 may be implemented using a web browser.

[0314] (Main processes) S251: In still image playback mode, upon operation by any user (here, user b), the display control unit 33 displays a predetermined area still image, which is a predetermined area in a wide-view still image, on the display 306 of the communication terminal 30B.

[0315] S252: The reception unit 32 receives the input for the video playback instruction.

[0316] S253: The communication unit 31 (receiving unit) acquires the wide-field video associated with the wide-field still image from the information processing system 50 (or storage 90).

[0317] S254: The display control unit 33 plays and displays a predetermined region video, which is a predetermined area in the wide-view video acquired in step S253, on the display 306.

[0318] (Detailed processing) S301: When any user (in this case, user b) presses the "still image playback mode" button b2 shown in Figure 36 with a mouse or the like, the reception unit 32 accepts the transition to still image playback mode.

[0319] S302: In response to instructions from the still image viewer 30a, the communication unit 31 transmits a request for wide-field still images to the information processing system 50. This request includes an ID for each wide-field still image to identify the wide-field still images extracted in frame extraction mode. As a result, the communication unit 51 of the information processing system 50 receives the request for wide-field still images.

[0320] S303: The storage and reading unit 59 of the information processing system 50 searches the playback-related information storage unit 5004 using each still image ID received in step S302 as a search key, and reads the storage location information of the corresponding wide-field still image. Then, the communication unit 51 accesses the storage location information in the storage 90 that has been read.

[0321] S304: The communications unit 51 acquires data for each wide-field still image from the storage location information.

[0322] S305: The communication unit 51 transmits the data of each wide-field still image acquired in step S304 to the still image viewer 30a of the requesting communication terminal 30. At this time, the data of each wide-field still image includes the still image ID of that wide-field still image. As a result, the communication unit 31 receives the data of each wide-field still image according to the instructions of the still image viewer 30a.

[0323] S306: In response to an instruction from the still image viewer 30a, the display control unit 33 displays a still image playback mode screen containing each wide-field still image on the display 306, as shown in Figure 43. This screen displays the predetermined region still images from each wide-field still image acquired in step S305. A video playback button b21 is also displayed to continue playing the video from the elapsed playback time of the currently playing still image. At the bottom, a rightward forward button b22 for scrolling the displayed predetermined region still images to the right and a leftward forward button b23 for scrolling them to the left are displayed. In Figure 43, three predetermined region still images are displayed, with the middle image being larger than the left and right images. Note that the display of each predetermined region still image in Figure 43 can be changed by moving the virtual viewpoint.

[0324] S307: When user b places the cursor c1 over the video playback button b21 with a mouse or the like and presses it, the reception unit 32 receives a command from the still image viewer 30a to play a video associated with a predetermined area still image in a wide-field still image that is displayed relatively large in the center of the still image playback mode screen.

[0325] S308: In response to instructions from the still image viewer 30a, the communication unit 31 transmits a specific still image ID to the information processing system 50 to identify the predetermined area still image that was displayed in the center when the video playback button b21 was pressed in step S307. As a result, the communication unit 51 of the information processing system 50 receives the specific still image ID.

[0326] S309: The storage and reading unit 59 of the information processing system 50 searches the playback-related information storage unit 5004 using the specific still image ID received in step S308 as a search key, and reads the corresponding video ID of the wide-field video, playback elapsed time information, and viewpoint information.

[0327] S310: The communication unit 51 transmits the video ID of the wide-field video, the playback elapsed time information, and the viewpoint information, which were read in step S309, to the still image viewer 30a of the communication terminal 30. As a result, the communication unit 31 receives the video ID of the wide-field video, the playback elapsed time information, and the viewpoint information.

[0328] S311: The still image viewer 30a passes to the video viewer 30b the video ID of the wide-field video, playback elapsed time information, and viewpoint information received in step S309.

[0329] S312: In response to instructions from the video viewer 30b, the communication unit 31 sends a request to the information processing system 50 for a wide-field video in which a predetermined area still image that was displayed in the center of the still image playback mode screen when pressed in step S307 has been framed and extracted. This request includes the video ID read in step S309. As a result, the communication unit 51 of the information processing system 50 receives the request for a wide-field video.

[0330] S313: The storage and reading unit 59 of the information processing system 50 searches the image management information storage unit 5001 using the video ID received in step S312 as a search key, and reads the specific storage location information of the corresponding wide-field video. Then, the communication unit 51 accesses the specific storage location information in the storage 90 that has been read.

[0331] S314: The communications unit 51 acquires wide-field video data from specific storage location information.

[0332] S315: The communication unit 51 transmits the wide-field video data acquired in step S314 to the video viewer 30b of the requesting communication terminal 30. As a result, the communication unit 31 receives the wide-field video data according to the instructions of the video viewer 30b.

[0333] S316: In response to instructions from the video viewer 30b, the display control unit 33 displays a specific predetermined region video on the display 306 in still image playback mode, as shown in Figure 44. At this time, the display control unit 33 displays the predetermined region video by determining a predetermined region in the wide-view video using viewpoint information obtained from the information processing system 50 via steps S310 and S311. Furthermore, the display control unit 33 uses the playback elapsed time information obtained from the information processing system 50 via steps S310 and S311 to play and display the predetermined region video from a specific playback elapsed time in the total playback time of the wide-view video. Note that the display control unit 33 may also display the predetermined region video from a predetermined time before the specific playback elapsed time (for example, 3 seconds before) or from a predetermined time after the specific playback elapsed time (for example, 0.5 seconds after). In this way, in response to a playback instruction for a video associated with a predetermined region still image in a displayed wide-field still image, the predetermined region video in the wide-field video containing the wide-field still image can be played and displayed. Therefore, for example, if you want to check a scene of interest using a wide-field still image and then check detailed information related to that scene in a video, you can play the wide-field video while retaining the viewpoint information from the wide-field still image. This makes it easier to understand the relationship between the still image and the video, even with wide-field images where the virtual viewpoint can be shifted.

[0334] In step S309, the information processing system 50 sent the video ID, etc., to the still image viewer 30a, but it may also send the video ID to the video viewer 30b. In this case, the processing in step S310 is omitted.

[0335] Furthermore, if the storage 90 has the same functions and data as the image management information storage unit 5001 and the playback-related information storage unit 5004 of the information processing system 50, the processing in steps S302, S305, S308, S310, S312, and S315 may be performed by the communication terminal 30 communicating directly with the storage 90 without going through the information processing system 50. In such a case, the steps S303 to S305, S309, S310, and S313 to S315, which are performed by the information processing system 50, may be performed by the storage 90.

[0336] With the above steps, the processing for the still image playback mode shown in Figure 42 is complete.

[0337] <<Video Playback Mode>> Next, the video playback mode will be explained using Figures 36, 45 to 48. Figure 41 is a flowchart illustrating the main processing of the communication terminal during video playback mode. Figure 46 is a sequence diagram showing the detailed processing of the video playback mode. Figure 47 is a conceptual diagram of the screen that plays a predetermined area of ​​still images from a predetermined area of ​​video in video playback mode. Figure 48 is a conceptual diagram of the screen that returns to displaying a predetermined area of ​​video from a predetermined area of ​​still images in video playback mode. In this case, the communication terminal 30 also has a still image viewer 30a and a video viewer 30b installed, and the communication unit 31, the reception unit 32, and the display control unit 33 execute processing according to the instructions of the still image viewer 30a and the video viewer 30b. The communication terminal 30 in this case can also be referred to as a display terminal.

[0338] (Main processes) S351: In video playback mode, upon operation by any user (here, user b), the display control unit 33 displays a predetermined area video, which is a predetermined area in a wide-view video, on the display 306 of the communication terminal 30B.

[0339] S352: The reception unit 32 receives the operation input for the still image playback instruction.

[0340] S353: The communication unit 31 (receiving unit) acquires wide-field still images associated with wide-field video from the information processing system 50 (or storage 90).

[0341] S354: The display control unit 33 plays back and displays a predetermined region still image, which is a predetermined area of ​​the wide-view still image acquired in step S253, on the display 306.

[0342] (Detailed processing) S401: When any user (in this case, user b) presses the "video playback mode" button b3 shown in Figure 36 with a mouse or the like, the reception unit 32 accepts the transition to video playback mode.

[0343] S402: In response to instructions from the video viewer 30b, the communication unit 31 sends a request for wide-field video to the information processing system 50. This request includes a video ID to identify the wide-field video recorded in step S65 of Figure 33. As a result, the communication unit 51 of the information processing system 50 receives the request for wide-field video.

[0344] S403: The storage and reading unit 59 of the information processing system 50 searches the image management information storage unit 5001 using the video ID received in step S402 as a search key, and reads the storage location information of the corresponding wide-field video. Then, the communication unit 51 accesses the storage location information in the storage 90 that has been read.

[0345] S404: The communications unit 51 acquires data for each wide-field video from the storage location information.

[0346] S405: The communication unit 51 transmits the wide-field video data acquired in step S404 to the video viewer 30b of the requesting communication terminal 30. At this time, the data for each wide-field video includes the video ID of that wide-field video. As a result, the communication unit 31 receives the wide-field video data according to the instructions of the video viewer 30b.

[0347] S406: In response to instructions from the video viewer 30b, the display control unit 33 displays a video playback mode screen including the wide-view video on the display 306, as shown in Figure 44. This screen displays the predetermined region video from the wide-view video acquired in step S405. A still image playback button b31 is also displayed for playing a still image extracted at the nearest playback time of the currently playing video. At the bottom, a playback display button b32 for displaying the displayed predetermined region video and a reverse playback display button b33 for displaying reverse playback are also displayed. Note that the predetermined region video in Figure 47 can be changed by moving the virtual viewpoint.

[0348] S407: When user b places the cursor c1 on the still image playback button b31 with a mouse or the like and presses it, the reception unit 32 receives a playback instruction from the video viewer 30b for a still image associated with a predetermined area video in the wide-field video currently being displayed on the video playback mode screen.

[0349] S408: In response to instructions from the video viewer 30b, the communication unit 31 transmits to the information processing system 50 the video ID of the original wide-field video of the predetermined area video being played back, and information about the elapsed playback time when the still image playback button b31 was pressed. As a result, the communication unit 51 of the information processing system 50 receives the video ID and the elapsed playback time information.

[0350] S409: The storage and reading unit 59 of the information processing system 50 searches the playback-related information storage unit 5004 using the specific video ID received in step S308 as a search key, and extracts a record containing the corresponding still image ID. Furthermore, the storage and reading unit 59 obtains the still image ID and viewpoint information of the wide-field still image from the record containing the nearest playback elapsed time received in step S308.

[0351] S410: The communication unit 51 transmits the still image ID and viewpoint information of the wide-field still image, which were read in step S409, to the still image viewer 30a of the communication terminal 30. As a result, the communication unit 31 receives the still image ID and viewpoint information of the wide-field still image.

[0352] S411: In response to an instruction from the still image viewer 30a, the communication unit 31 sends a request to the information processing system 50 for a wide-field still image that was framed and extracted at the nearest playback time of the predetermined area video that was displayed on the video playback mode screen when pressed in step S407. This request includes the still image ID obtained in step S411. As a result, the communication unit 51 of the information processing system 50 receives the request for a wide-field still image.

[0353] S412: The storage and reading unit 59 of the information processing system 50 searches the playback-related information storage unit 5004 using the still image ID received in step S411 as a search key, and reads the specific storage location information of the corresponding wide-field still image. The communication unit 51 then accesses the specific storage location information in the storage 90 that has been read.

[0354] S413: The communications unit 51 acquires wide-field still image data from specific storage location information.

[0355] S414: The communication unit 51 transmits the wide-field still image data acquired in step S413 to the still image viewer 30a of the requesting communication terminal 30. As a result, the communication unit 31 receives the wide-field still image data according to the instructions of the still image viewer 30a.

[0356] S415: In response to an instruction from the still image viewer 30a, the display control unit 33 displays a specific predetermined area still image on the display 306 in video playback mode, as shown in Figure 48. At this time, in step S410, the display control unit 33 displays the predetermined area still image by defining a predetermined area in the wide-field still image using viewpoint information acquired from the information processing system 50. In this way, in response to a playback instruction for a still image associated with a predetermined area video in the displayed wide-field video, the predetermined area still image in the wide-field still image included in the wide-field video can be played back and displayed. Therefore, for example, if you want to check detailed information in the wide-field video and then check a scene of interest related to the video, you can play back the wide-field still image by inheriting the viewpoint information associated with the scene of interest from among the wide-field still images, making it easier to grasp the relationship between still images and videos, even with wide-field images where the virtual viewpoint can be moved.

[0357] In step S410, the information processing system 50 transmitted the still image ID, etc., to the still image viewer 30a, but it may also transmit the still image ID to the video viewer 30b. In this case, the video viewer 30b provides the still image ID and viewpoint information to the still image viewer 30a.

[0358] Furthermore, if the storage 90 has the same functions and data as the image management information storage unit 5001 and the playback-related information storage unit 5004 of the information processing system 50, the processes in steps S402, S405, S408, S410, S411, and S414 may be executed by the communication terminal 30 communicating directly with the storage 90 without going through the information processing system 50. In such a case, the steps S403 to S405, S409, S410, and S412 to S414, which are executed by the information processing system 50, may be executed by the storage 90.

[0359] This completes the processing for the video playback mode shown in Figure 46.

[0360] <Examples of communication system applications in telemedicine> Figure 49 illustrates an example of remote communication where communication system 1a is applied to telemedicine. Note that the explanation of Figure 49 will primarily focus on the differences from Figure 1. In Figure 49, location A is an operating room, but the processing flow from (1) to (6) is the same as in Figure 1. In Figure 49, the patient is placed on an operating table 355 and undergoes surgery performed by a medical professional such as a doctor. The medical professional (corresponding to the user) uses various surgical instruments 354, such as forceps and scalpels, to operate on the patient. The medical professional can also wear smart glasses 88 and transmit images of the surgical field to the communication network N. Furthermore, various cameras, such as an operating room camera 351, a surgical field camera 352, and an endoscope 353, are arranged in the operating room as imaging devices similar to the imaging device 10. These imaging devices may also have imaging functions to capture images for generating wide-field images. All imaging devices and smart glasses 88 in the operating room are described as being associated with a virtual room.

[0361] A main unit 356 is located in the operating room to monitor the patient's vital signs and the operating status of medical equipment. The main unit 356 corresponds to the communication terminal 30 in this embodiment. In addition to the functions shown in Figure 1, the communication terminal 30 (main unit 356) in the operating room may also have the function of receiving images from the endoscope 353 and the surgical field camera 352. The communication terminal 30 can display the received images, including wide-field images, on the display 306 and can transmit them to the information processing system 50 as images from the communication terminal 30's location. The operation panel 357 is an input interface that accepts various operations, and may allow medical personnel to operate equipment in the operating room via the operation panel 357. The endoscope 353, surgical field camera 351, and surgical field camera 352 may also communicate directly with the information processing system 50 without going through the communication terminal 30. In this way, multiple imaging devices 10 can be associated with the same virtual room, so users at remote locations can request recordings of wide-field images capturing various scenes from the site at location A. For example, if you want to record images taken inside a patient's body, you can send a recording request to the imaging device corresponding to the endoscope 353, and if you want to record the overall situation in the operating room, you can send a recording request to the imaging device corresponding to the operating room camera 351.

[0362] Furthermore, the communication terminal 30 may have the functionality of an electronic medical record system, or it may have the functionality to communicate with an electronic medical record system. The communication terminal 30 may display electronic medical record information on the display 306. Also, the storage 90 may be an electronic medical record system. In such cases, the recording data of wide-field images (and associated viewpoint information) recorded in response to a recording request may be associated with the patient's electronic medical record and saved by the association processing unit 53. Also, the folders indicated by the storage location of the storage 90 may be classified by patient or surgery. Also, the virtual room information storage unit 5002 may store associated information indicating the patient and surgery. In this way, information related to the patient and surgery can always be displayed on the viewing screen of the communication terminal 30.

[0363] Figure 50 shows an example of a virtual room mapping screen (part 1) 360 for associating an imaging device with a virtual room in the case of telemedicine. The explanation of Figure 50 mainly focuses on the differences from Figure 22.

[0364] In the case of telemedicine, the virtual room mapping screen (part 1) 360 displays a list of virtual rooms 361 associated with, for example, remote surgeries or consultations. Medical cameras 362, including 360-degree cameras, are associated with each virtual room. Medical cameras 362 include endoscopes, surgical field cameras used for imaging the surgical field in the operating room, and cameras for capturing microscopic images.

[0365] <Main effects> As described above, this embodiment has the effect of making it easier to understand the relationship between wide-field video and still images contained within the video, based on the associated viewpoint information.

[0366] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0367] For example, the configuration example shown in Figure 13 is divided according to its main function in order to facilitate understanding of the processing performed by the information processing system 50, the imaging device 10, and the communication terminal 30. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the information processing system 50, the imaging device 10, and the communication terminal 30 can be further divided into more processing units depending on the processing content. Furthermore, each processing unit can be divided to include even more processing.

[0368] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0369] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0370] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figures 26, 27, 33, and 34, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices. [Explanation of symbols]

[0371] 10 Imaging device 11. Communication section (an example of a transmitting section, an example of a receiving section) 30. Communication terminal (an example of a display terminal) 31. Communication section (an example of a transmitting section, an example of a receiving section) 32 Reception Department 33 Display Control Unit 50 Information Processing Systems 51. Communication section (an example of a transmitting section, an example of a receiving section) 90 storage 306 Display (Example of a display unit) [Prior art documents] [Patent Documents]

[0372] [Patent Document 1] Japanese Patent Publication No. 2020-140600

Claims

1. A display terminal that displays images, A receiving unit that receives a playback instruction for a video associated with a predetermined region still image, which is a predetermined region in a wide-view still image having a wide viewing angle displayed on the display unit, A display control unit, in response to the playback instruction, causes the display unit to play back and display the predetermined region video, which is the predetermined region in the wide-field video that includes the wide-field still image, based on viewpoint information for identifying the predetermined region relating to the predetermined region still image. A display terminal characterized by having the following features.

2. The display terminal according to claim 1, characterized in that the wide-field video containing the wide-field still images is a wide-field video from which the wide-field still images have been extracted as frames.

3. The display terminal according to claim 1 or 2, characterized in that the display control unit starts playback display of the predetermined region video in the wide-field video based on the playback elapsed time in the wide-field video when the wide-field still image is extracted from the wide-field video.

4. The display terminal according to claim 3, characterized in that the display control unit starts playback and display of the predetermined area video in the wide-view video from the elapsed playback time.

5. The reception unit receives screen operations on the predetermined area video that is being played back, The display terminal according to claim 1 or 2, characterized in that the display control unit moves the virtual viewpoint over the wide-view video based on the screen operation, and then changes and displays the predetermined region video based on the modified viewpoint information corresponding to the viewpoint after the move.

6. The display control unit causes a plurality of predetermined area still images to be displayed on the display unit. The reception unit receives a designation for a specific predetermined region still image from among a plurality of predetermined region still images. The display control unit causes the predetermined region video to be played back and displayed on the display unit based on the viewpoint information used to identify the predetermined region relating to the specified still image of the predetermined region. The display terminal according to claim 1 or 2, characterized by the above.

7. The display terminal according to any one of claims 1 to 6, characterized in that the wide-field still image and the wide-field video have a viewing angle wider than the display range that can be displayed on the display unit at one time.

8. An information processing system that transmits an image to a display terminal that displays an image, A receiving unit that receives still image identification information transmitted by the display terminal for identifying wide-view still images having a wide viewing angle, A transmission unit transmits to the display terminal viewpoint information associated with the still image identification information for identifying a predetermined area in the wide-field still image, and a wide-field video in which the wide-field still image is included. An information processing system characterized by having the following features.

9. The information processing system according to claim 8, characterized in that the transmitting unit transmits to the display terminal the playback elapsed time in the wide-field video when the wide-field still image is extracted from the wide-field video.

10. A communication system comprising a display terminal that displays an image, and an information processing system that transmits an image to the display terminal, The aforementioned display terminal is We accept designations for a specified region still image, which is a specified area within a wide-field still image having a wide field of view. The still image identification information for identifying the wide-field still image is transmitted to the information processing system. The aforementioned information processing system is The display terminal receives the still image identification information transmitted by the display terminal, The display terminal is transmitted viewpoint information associated with the still image identification information for identifying a predetermined area in the wide-field still image, and a wide-field video in which the wide-field still image is extracted as a frame. The aforementioned display terminal is The viewpoint information and the wide-field video are received. Based on the viewpoint information, the predetermined region video, which is the predetermined region in the wide-view video, is played back and displayed on the display unit. A communication system characterized by the following features.

11. A display method performed by a display terminal that displays an image, The aforementioned display terminal is The display unit receives a playback instruction for a video associated with a predetermined region still image, which is a predetermined region within a wide-view still image having a wide viewing angle. In response to the playback instruction, the display unit plays and displays the predetermined region video, which is the predetermined region in the wide-field video that includes the wide-field still image, based on viewpoint information for identifying the predetermined region related to the predetermined region still image. A method of display characterized by the following features.

12. An information processing method performed by an information processing system that transmits an image to a display terminal that displays an image, The aforementioned information processing system is The display terminal receives still image identification information transmitted for identifying wide-field still images having a wide viewing angle, The display terminal is transmitted viewpoint information associated with the still image identification information for identifying a predetermined area in the wide-field still image, and a wide-field video containing the wide-field still image. An information processing method characterized by the following:

13. A communication method performed by a communication system having a display terminal that displays an image and an information processing system that transmits an image to the display terminal, The aforementioned display terminal is We accept designations for a specified region still image, which is a specified area within a wide-field still image having a wide field of view. The still image identification information for identifying the wide-field still image is transmitted to the information processing system. The aforementioned information processing system is The display terminal receives the still image identification information transmitted by the display terminal, The display terminal is transmitted viewpoint information associated with the still image identification information for identifying a predetermined area in the wide-field still image, and a wide-field video in which the wide-field still image is extracted as a frame. The aforementioned display terminal is The viewpoint information and the wide-field video are received. Based on the viewpoint information, the predetermined region video, which is the predetermined region in the wide-view video, is played back and displayed on the display unit. A communication method characterized by the following features.

14. On the computer, The display unit receives a playback instruction for a video associated with a predetermined region still image, which is a predetermined region within a wide-view still image having a wide viewing angle. In response to the playback instruction, the display unit plays and displays the predetermined region video, which is the predetermined region in the wide-field video that includes the wide-field still image, based on viewpoint information for identifying the predetermined region related to the predetermined region still image. A program characterized by the following features.

15. A program that causes a computer to send an image to a display terminal that displays images, To the aforementioned computer, The display terminal transmits still image identification information for identifying wide-field still images having a wide viewing angle, and receives the said still image identification information. The display terminal is instructed to transmit viewpoint information associated with the still image identification information for identifying a predetermined area in the wide-field still image, and a wide-field video in which the wide-field still image is extracted as a frame. A program characterized by the following features.

16. A display terminal that displays images, A receiving unit that receives a playback instruction for a still image associated with a predetermined region video, which is a predetermined region in a wide-view video having a wide viewing angle displayed on the display unit, A display control unit, in response to the playback instruction, causes the display unit to display a predetermined region still image, which is a predetermined region in a wide-field still image, that is a wide-field still image associated with the wide-field video, based on viewpoint information for identifying a predetermined region in the wide-field still image included in the wide-field video. A display terminal characterized by having the following features.

17. The display terminal according to claim 16, characterized in that the wide-field still images included in the wide-field video are wide-field still images extracted as frames of the wide-field video.

18. The display terminal according to claim 16 or 17, characterized in that the display control unit starts playback display of the predetermined region still image in the wide-field still image based on the playback elapsed time in the wide-field video when the wide-field still image is extracted from the wide-field video.

19. The display terminal according to claim 18, characterized in that the display control unit starts playback display of the predetermined region still image in the wide-field still image that was extracted at the nearest playback time of the playback time.

20. The reception unit receives screen operations on the predetermined area still image that is being played back, The display terminal according to claim 16 or 17, characterized in that the display control unit moves the virtual viewpoint over the wide-view still image based on the screen operation, and then changes and displays the predetermined area still image based on the modified viewpoint information corresponding to the viewpoint after the move.

21. The display control unit causes a plurality of predetermined area videos to be displayed on the display unit. The reception unit receives a designation for a specific predetermined region video from among a plurality of predetermined region videos. The display control unit causes the display unit to play and display a still image of the predetermined region based on the viewpoint information used to identify the predetermined region related to the specified predetermined region video. The display terminal according to claim 16 or 17.

22. The display terminal according to any one of claims 16 to 21, characterized in that the wide-field still image and the wide-field video have a viewing angle wider than the display range that can be displayed on the display unit at once.

23. An information processing system that transmits an image to a display terminal that displays an image, A receiving unit that receives video identification information transmitted by the display terminal for identifying wide-field video having a wide viewing angle, A transmission unit that transmits to the display terminal viewpoint information associated with the video identification information for identifying a predetermined area in the wide-field video, and wide-field still images included in the wide-field video, An information processing system characterized by having the following features.

24. The information processing system according to claim 23, characterized in that the transmitting unit transmits to the display terminal the playback elapsed time in the wide-field video when the wide-field still image is extracted from the wide-field video as one frame.

25. A communication system comprising a display terminal that displays an image, and an information processing system that transmits an image to the display terminal, The aforementioned display terminal is The system accepts a designation for a predetermined region video, which is a predetermined area within a frame that constitutes a wide-field video having a wide field of view. The system transmits video identification information for identifying the wide-field video to the aforementioned information processing system. The aforementioned information processing system is The display terminal receives the video identification information transmitted by the display terminal, To the display terminal, the following are transmitted: viewpoint information for identifying a predetermined region in the frame constituting the wide-field video, associated with the video identification information, and a wide-field still image included in the wide-field video. The aforementioned display terminal is The viewpoint information and the wide-field still image are received. Based on the viewpoint information, the display unit plays back and displays the predetermined region still image, which is the predetermined region in the wide-field still image. A communication system characterized by the following features.

26. A display method performed by a display terminal that displays an image, The aforementioned display terminal is The display unit receives a playback instruction for a still image associated with a predetermined region video, which is a predetermined region in a wide-view video with a wide field of view that is displayed on the display unit. In response to the playback instruction, the display unit displays a predetermined region still image, which is a predetermined region in a wide-field still image, that is a wide-field still image associated with the wide-field video, based on viewpoint information for identifying a predetermined region in the wide-field still image included in the wide-field video. A method of display characterized by the following features.

27. An information processing method performed by an information processing system that transmits an image to a display terminal that displays an image, The aforementioned information processing system is The display terminal receives video identification information transmitted for identifying wide-field video having a wide viewing angle, The display terminal is transmitted viewpoint information associated with the video identification information for identifying a predetermined area in the wide-field video, and wide-field still images included in the wide-field video. An information processing method characterized by the following:

28. A communication method performed by a communication system having a display terminal that displays an image and an information processing system that transmits an image to the display terminal, The aforementioned display terminal is The system accepts a designation for a predetermined region video, which is a predetermined area within a frame that constitutes a wide-field video having a wide field of view. The system transmits video identification information for identifying the wide-field video to the aforementioned information processing system. The aforementioned information processing system is The display terminal receives the video identification information transmitted by the display terminal, To the display terminal, the following are transmitted: viewpoint information for identifying a predetermined region in the frame constituting the wide-field video, associated with the video identification information, and a wide-field still image included in the wide-field video. The aforementioned display terminal is The viewpoint information and the wide-field still image are received. Based on the viewpoint information, the display unit plays back and displays the predetermined region still image, which is the predetermined region in the wide-field still image. A communication method characterized by the following features.

29. On the computer, The display unit receives a playback instruction for a still image associated with a predetermined region video, which is a predetermined region in a wide-view video having a wide field of view. In response to the playback instruction, the display unit displays a predetermined region still image, which is a predetermined region in a wide-field still image, that is a wide-field still image associated with the wide-field video, based on viewpoint information for identifying a predetermined region in the wide-field still image included in the wide-field video. A program characterized by the following features.

30. A program that causes a computer to send an image to a display terminal that displays images, To the aforementioned computer, The display terminal transmits video identification information for identifying wide-field video having a wide viewing angle, and receives said video identification information. The display terminal is instructed to transmit viewpoint information associated with the video identification information for identifying a predetermined area in the wide-field video, and wide-field still images included in the wide-field video. A program characterized by the following features.

Images