Communication systems, information processing systems, information processing methods, programs

Abstract

The purpose is to understand the viewing status of the distributed image data. [Solution] A communication system comprising: a plurality of communication terminals and an information processing system that distributes distributed image data, which includes a plurality of image data captured at a plurality of locations, to the plurality of communication terminals, wherein the system includes: a viewing status recording unit that stores viewing status information received from each of the plurality of communication terminals in association with the distributed image data, in response to operations on the plurality of image data included in the distributed image data at each of the plurality of communication terminals; and a display control unit that, after the distributed image data has been distributed, receives a request from the communication terminal to play back the distributed image data and displays the viewing result based on the viewing status information on the communication terminal that made the playback request.

Description

[Technical Field]

[0001] The present invention relates to a communication system, an information processing system, an information processing method, and a program. [Background Art]

[0002] Conventionally, there is known a system that, based on captured image data transmitted from an imaging device and time-series information of a predetermined region of the captured image data, generates image data reflecting the viewing status of each user in the captured image data and distributes the generated image data to communication terminals of each user. With this system, when a user reviews the distributed image data later, the user can be made aware of the portions that the user themselves paid attention to, and the like. [Summary of the Invention] [Problem to be Solved by the Invention]

[0003] However, in the conventional system described above, when image data captured by imaging devices disposed at each of a plurality of bases is distributed to a communication terminal, the viewing status of the distributed image data cannot be grasped.

[0004] In view of the above problem, it is an object of the present invention to enable grasping of the viewing status of distributed image data. [Means for Solving the Problem]

[0005] The disclosed technology is a communication system comprising: a plurality of communication terminals; and an information processing system that distributes distributed image data, which includes a plurality of image data captured at a plurality of locations, to the plurality of communication terminals, the system comprising: a viewing status recording unit that stores viewing status information received from each of the plurality of communication terminals in association with the distributed image data, in response to operations on the plurality of image data included in the distributed image data at each of the plurality of communication terminals; and a display control unit that, after the distributed image data has been distributed, receives a request from the communication terminal to play back the distributed image data and displays the viewing result based on the viewing status information on the communication terminal that made the playback request. [Effects of the Invention]

[0006] It is possible to track the viewing status of the distributed image data. [Brief explanation of the drawing]

[0007] [Figure 1] This figure illustrates an example of remote communication using wide-field imaging. [Figure 2] This is an example of a schematic diagram of a communication system configuration. [Figure 3] This is an example of a hardware configuration diagram for an imaging device. [Figure 4] This is an example of a hardware configuration diagram for a communication terminal and an information processing system. [Figure 5] (a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a top view of the imaging device. [Figure 6] This is a diagram illustrating the use of the imaging device. [Figure 7] (a) is a hemispherical image (front) captured by the imaging device, (b) is a hemispherical image (back) captured by the imaging device, and (c) is an image represented using equirectangular projection. [Figure 8] (a) A conceptual diagram showing the state of covering the sphere with an equirectangular projection image, and (b) A diagram showing a full-sphere image. [Figure 9]This diagram shows the positions of a virtual camera and a predetermined region when a 360-degree spherical image is treated as a three-dimensional sphere. [Figure 10] (a) is a stereoscopic perspective view of Figure 5, and (b) is a diagram showing the state in which an image of a predetermined area is displayed on the display. [Figure 11] This figure shows the relationship between information from a predetermined region and an image of a predetermined region T. [Figure 12] This is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. [Figure 13] This is an example of a functional configuration diagram for a communication system. [Figure 14] This is a conceptual diagram showing the image management information stored in the image management information storage unit. [Figure 15] This is a conceptual diagram showing the virtual room information stored in the virtual room information storage unit. [Figure 16] This is an example of a sequence diagram illustrating the process of a user (or communication terminal) entering a virtual room. [Figure 17] This figure shows an example of a virtual room mapping screen (part 1) for associating imaging devices with virtual rooms. [Figure 18] This figure shows an example of the virtual room mapping screen (part 2). [Figure 19] This figure shows an example of the virtual room mapping screen (part 3). [Figure 20] This figure shows an example of a wide-field image transmission start / stop dialog displayed by a communication terminal. [Figure 21] This is an example of a sequence diagram showing the procedure for a user to register an imaging device in a virtual room. [Figure 22] This is an example of a sequence diagram illustrating the process by which wide-field images are shared. [Figure 23] This figure shows an example of a viewing status memory unit. [Figure 24] This figure shows an example of a storage unit for aggregated results. [Figure 25] This is the first sequence diagram showing the operation of communication system 1. [Figure 26]It is a first diagram showing a display example of a communication terminal. [Figure 27] It is a second diagram showing a display example of a communication terminal. [Figure 28] It is a third diagram showing a display example of a communication terminal. [Figure 29] It is a fourth diagram showing a display example of a communication terminal. [Figure 30] It is a second sequence diagram showing an operation of the communication system 1. [Figure 31] It is a diagram illustrating an example of remote communication in which the communication system 1 is applied to telemedicine. [Figure 32] It is a diagram showing an example of a virtual room association screen (part 1) 360 for associating an imaging device with a virtual room in the case of telemedicine. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, as an example of an embodiment for carrying out the present invention, an information processing system and an image sharing method performed in the information processing system will be described.

[0009] <An Example of Remote Communication> FIG. 1 is a diagram illustrating an example of a communication system that realizes remote communication using a wide-field image. In the communication system 1 shown in FIG. 1, three bases A to C communicate with each other via the information processing system 50. The number of bases is merely an example, and it may be two bases or four or more bases. Remote communication refers to achieving communication through images and voice with partners who are physically separated by utilizing IT tools.

[0010] Locations A, B, and C can be anywhere, but they should be places where wide-field images can be transmitted, such as an office. Location A is equipped with an imaging device 10 that can capture wide-angle images, such as a 360-degree spherical image, or wide-angle images with a horizontal range of 180 to 360 degrees. Hereafter, such wide-angle images will simply be referred to as wide-field images. Locations A to C are equipped with various communication terminals 30A to 30C that can view wide-field images. Hereafter, any of the communication terminals 30A to 30C will be referred to as "communication terminal 30".

[0011] In this embodiment, while capturing the entirety of site A with a wide-field image, users at each site A to C can arbitrarily change their viewpoint to view the wide-field image if they wish to focus on a particular area. The viewpoint refers to the central position or range displayed on the displays of communication terminals 30A to 30C within the entire wide-field image.

[0012] The imaging device 10 can be mounted on a tripod 86 or on an arm 85 via a gimbal 87. A relay device (in Figure 1, the communication terminal 30A also serves as the relay device) is installed at base A, and the communication terminal 30A transmits the wide-field images received from the imaging device 10 via wired or wireless connection to the information processing system 50. The communication terminal 30A can also serve as a terminal for viewing the wide-field images. A camera 9 is connected to the communication terminal 30A (it can also be built-in), and images of the normal field of view (360-degree images can also be captured) captured by the camera 9 can also be transmitted to the information processing system 50. In addition, user A (for example, a worker) can wear smart glasses 88, and images of the normal field of view (360-degree images can also be captured) captured by the smart glasses 88 may be transmitted to the information processing system 50. Smart glasses 88 are information terminals that display information acquired via the internet on a display while maintaining a field of view. Smart glasses 88 may be placed at any base.

[0013] At site B, a PC (Personal Computer) or smartphone is provided as an example of a communication terminal 30B. Furthermore, any device capable of communicating with the information processing system 50 via the communication network N may be used as a communication terminal 30B; in addition to these, display devices such as tablet terminals, PDAs (Personal Digital Assistants), electronic whiteboards, and projectors may also be used. The communication terminal 30B may also have a built-in or connected camera.

[0014] At base C, a PC, smartphone, VR goggles 89, etc., are installed as examples of communication terminals 30C. In Figure 1, an imaging device 8 is built into or connected to the communication terminal 30C. VR goggles 89 are information terminals that display an artificial world on a computer or a 360-degree image according to the direction of movement of the head and body. The imaging device 8 may be for wide-angle or normal field of view. Furthermore, the communication terminal 30C can be any device that can communicate with the information processing system 50 via the communication network N, and may be a display device such as a tablet terminal, PDA, electronic whiteboard, or projector. VR goggles 89 may be installed at any base.

[0015] In this embodiment, the imaging device 10 and the communication terminal 30 are managed in a communication group called a virtual room. The imaging device 10 is associated with the virtual room, and the communication terminal 30 (the user operating the communication terminal 30) can enter this virtual room to receive the wide-field image transmitted by the imaging device 10 and view the wide-field image. Smart glasses 88 and VR goggles 89 can also be associated with the virtual room. The imaging devices 8 and 9 enter the virtual room together with the communication terminal 30.

[0016] Users A to C at locations A to C can arbitrarily change their viewpoint on the wide-field image. Therefore, each user A to C viewing the wide-field image in real time may be seeing a different viewpoint, which could make communication difficult. To address this, in this embodiment, viewpoint information indicating the viewpoint of a user at any given location's communication terminal 30 is shared by users of communication terminals 30 at other locations. In other words, for example, viewpoint information indicating the viewpoint of a user at communication terminal 30 at location A is used as viewpoint information for users at communication terminals 30 at locations B and C. The outline of viewpoint information sharing will be explained below. For the sake of explanation, it will be assumed that viewpoint information indicating the viewpoint of a user at location B is shared by users at locations A and C.

[0017] (1) Communication terminals 30A to 30C share a wide-field image (an example of a first wide-field image) captured by the imaging device 10. When user B requests the capture of a wide-field image while viewing from an arbitrary viewpoint on communication terminal 30B, communication terminal 30B (an example of a first communication terminal) transmits viewpoint information and the capture request to the information processing system 50.

[0018] (2) In response to the imaging request, the information processing system 50 specifies viewpoint information and requests the imaging device 10 to take an image (either a still image or a video).

[0019] (3) The imaging device 10 takes an image in response to an imaging request and saves the wide-field image (an example of a second wide-field image) and viewpoint information to a URL (an example of storage location information; in Figure 1, the imaging device 10 is stored in storage 90) notified by the information processing system 50. The wide-field image saved in storage 90 can be downloaded and displayed by any communication terminal 30.

[0020] (4) The information processing system 50 sends a URL to the communication terminal 30B.

[0021] (5) The information processing system 50 also automatically or at the request of user B transmits the URL to the communication terminals 30A and 30C (an example of a second communication terminal) that are in the same virtual room.

[0022] (6) The communication terminals 30A and 30C connect to a URL to receive viewpoint information and a wide-field image, and set the viewpoint of the wide-field image identified by the viewpoint information to align with the center of the image field and display it. Note that it is not necessary to perfectly align the viewpoint with the center; the viewpoint can also be set to be included in the area near the center of the image field and displayed.

[0023] The same applies when users at locations B and C share the perspective of users at location A, and when users at locations A and B share the perspective of users at location C.

[0024] As described above, in this embodiment, even when wide-field images are distributed, viewpoint information is shared at each location, making it easier for users to communicate their intentions.

[0025] In (3), the imaging device 10 can transmit the wide-field image itself to the information processing system 50, and in (4), the information processing system 50 can transmit the wide-field image to the communication terminals 30A to 30C.

[0026] Furthermore, the communication system 1 shown in Figure 1 can also be applied to VR (Virtual Reality) education, event distribution, remote customer service, etc. In VR education, the imaging device 10 is placed at a site such as a laboratory, and students can view blackboards, equipment, samples, experimental results, etc. from a remote location by arbitrarily changing their viewpoint. In event distribution, the imaging device 10 is placed at the event venue, and event participants such as spectators can view the venue online from a remote location by arbitrarily changing their viewpoint. The venue includes images of performers, contestants, presenters, products and exhibits explained at the event, images of materials explained at the event, and images of the venue's condition. The event venue may be indoors or outdoors and includes venues for sports, concerts, plays, etc. In remote customer service, for example, when applied to customer service at a travel agency, the imaging device 10 is placed at the travel destination, and customers can review their itinerary from a remote location by arbitrarily changing their viewpoint.

[0027] The locations where images are captured are not limited to these; any space where users (viewers) at the viewing location have a need to understand the situation at a remote location, such as schools, factories, warehouses, construction sites, server rooms, or stores, is acceptable.

[0028] <About Terminology> A tenant is a company or organization that has contracted with a service provider (in this embodiment, an information processing system) to receive image distribution services. While a user is typically affiliated with a tenant, individuals can also subscribe to the service. In addition to users, tenants also include registered imaging devices, virtual rooms, and other elements.

[0029] A base of operations refers to a place that serves as the foundation for activities. In this embodiment, a conference room is used as an example of a base of operations. A conference room is a room set up primarily for the purpose of holding meetings. Meetings are also called gatherings, meetings, discussions, assemblies, and other similar terms.

[0030] Furthermore, the base of operations does not necessarily have to have users actively participating; it could include locations where only imaging equipment for distributing captured video and still images is installed.

[0031] A device refers to an apparatus other than a general-purpose communication terminal 30 such as a PC or smartphone, and is an imaging device or a wide-field image viewing device. In this embodiment, examples include an imaging device 10, smart glasses 88, and VR goggles 89.

[0032] Viewpoint information refers to parameter information that specifies which predetermined region of a wide-field image to display on the screen. In this embodiment, the radial, polar, and azimuth angles corresponding to the center of the wide-field image displayed on the screen are described as examples of viewpoint information, but it can also be specified using other parameter information such as the coordinates of diagonal vertices.

[0033] A wide-field image is an image with a wide field of view captured over a wide imaging range, including 360-degree images (also called spherical images, omnidirectional images, or all-around images) that capture the entire 360 ​​degrees surrounding the image. A wide-field image has a display range of up to 360 degrees vertically and horizontally, but any image with a wide field of view that is wider than the display range that can be displayed on a screen at once, even if it is less than 360 degrees vertically or horizontally, is included as a wide-field image. For example, an image with a display range wider than the range that a human can see at once is also included as a wide-field image. In addition, depending on the display method, an image that can be displayed on a screen at once is also included as a wide-field image if it has a wide field of view in a given display method. In this embodiment, an equirectangular spherical image is used as an example of a wide-field image, but omnidirectional images, hemispherical images, 3D panoramic images, 2D panoramic images, and VR (Virtual Reality) images are also included as wide-field images.

[0034] Images with a normal field of view are not wide-field images, but in this embodiment, they will be described as non-wide-field images (planar images).

[0035] A communication group is a collection of users from whom wide-field images are shared (distributed). In a normal space, a communication group is described as a virtual room, meaning that when each user enters the same room, each user can share the wide-field image. "Virtual" means that it is realized through information processing via a network.

[0036] Remote communication refers to communicating with someone who is physically far away, using IT tools to convey information through images and audio.

[0037] VR education refers to educational tools that utilize VR (Virtual Reality) to create an unrealistic space before the viewer's eyes, immersing them in a realistic experience and deepening their understanding and interest.

[0038] Remote customer service refers to customer service where staff in remote locations provide product information to users via video conferencing using tablets or displays.

[0039] An image refers to a visual representation of an event or phenomenon on a medium, while a video refers to an image displayed on a display device. Videos are included within the category of images.

[0040] Users at each location communicate remotely with each other. Remote communication is a meeting held at a remote location. A meeting is when people gather together for consultation, discussion, etc. Meetings can take various forms, such as customer service, conferences, gatherings, meetings, study groups, classes, seminars, and presentations. It does not necessarily have to be two-way communication. Therefore, a virtual room can also be called a virtual conference room.

[0041] <Example of a communication system configuration> Figure 2 shows an example of a schematic diagram of the communication system 1. The communication system 1 is a system that transmits and receives wide-field images and normal-angle images captured by the imaging device 10 bidirectionally between multiple locations. By displaying images distributed from one location at other locations, users at other locations can view the images. As an example of a wide-field image, a 360-degree spherical image captured by the imaging device 10 is distributed. The communication system 1 can, for example, allow a wide-field image captured at a predetermined location to be viewed at other locations in remote locations.

[0042] As shown in Figure 2, in the communication system 1, the imaging device 10, communication terminal 30A, and information processing system 50 located at base A, and communication terminals 30B and 30C located at each of the multiple bases (bases B and C), are connected in a way that enables communication.

[0043] If the imaging device 10 can be directly connected to the communication network N, a communication terminal 30A acting as a relay device (e.g., a router) is not necessary. In this case, the communication terminal 30A is connected to the communication network N without the imaging device 10. However, if the communication terminal 30A is located at site A, the communication terminal 30A also functions as a relay device, allowing user A to view wide-field images in the same way as communication terminals 30B and 30C. It is also possible to place additional imaging devices 10 at sites other than site A, or to place multiple imaging devices 10 at site A.

[0044] Each communication terminal 30 and information processing system 50 can communicate via the communication network N. The communication network N is constructed using the Internet, mobile communication networks, LANs (Local Area Networks), etc. The communication network N may also include wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution), in addition to wired communication.

[0045] As described later, the imaging device 10 is a special digital camera capable of capturing a full-sphere image by capturing a subject, landscape, etc., to obtain two hemispherical images as the basis. The wide-field image obtained by the imaging device 10 can be a video, a still image, or both. In the following description, the information including the captured image data and the audio data acquired during the capture of the image indicated by the captured image data may be referred to as "video."

[0046] The communication terminal 30 is a computer such as a PC used by users at each location. The communication terminal 30 displays images captured at its own location, wide-field images (still images or videos) distributed from other locations, and images with a normal field of view. For example, the communication terminal 30 acquires wide-field images captured by the imaging device 10 via the communication network N. The communication terminal 30 also has image processing software such as OpenGL ES installed, and can display images based on viewpoint information that identifies a portion of the wide-field image. Note that OpenGL ES is just one example of image processing software, and other software can be used. Furthermore, even if image processing software is not installed, image processing can be performed by software received from an external source, or image display can be performed by receiving the results of image processing performed by external software. In other words, the communication terminal 30 can display a predetermined portion of the wide-field image.

[0047] The communication terminal 30 can arbitrarily change the viewpoint relative to the display range of a wide-field image in response to user operation. The communication terminal 30 can change the field of view (predetermined area) based on viewpoint information corresponding to the changed viewpoint by moving the virtual viewpoint in response to user operation input (including key input, drag, scrolling, etc.) to a touch panel, directional buttons, mouse, keyboard, touchpad, etc. Furthermore, if the communication terminal 30 is a communication terminal worn by a user, such as VR goggles, it can detect the posture information of the communication terminal 30 that has been changed in response to changes in the movements of the user wearing it, and can change the field of view (predetermined area) based on viewpoint information corresponding to the changed viewpoint by moving the virtual viewpoint in response to the detected posture information.

[0048] The communication terminal 30A distributes wide-field images acquired from the imaging device 10 via a wired cable such as a USB (Universal Serial Bus) cable to communication terminals 30 at other locations via the information processing system 50. The connection between the imaging device 10 and the communication terminal 30A can also be a wireless connection using short-range wireless communication, etc., instead of a wired connection using a wired cable. Multiple communication terminals 30A can also be placed at location A.

[0049] In some cases, a user at location A may wear smart glasses 88, and the smart glasses 88 may connect to the communication network N. In this case, the images captured by the smart glasses 88 are transmitted to the information processing system 50 via the communication network N, and the information processing system 50 can distribute them to the communication terminals 30 at each location.

[0050] Communication terminal 30B is located at site B where user B is located, and communication terminal 30C is located at site C where user C is located. Multiple communication terminals 30B and 30C can be located at sites B and C. Furthermore, communication terminals 30B and 30C can be carried around by user B and user C, respectively.

[0051] Furthermore, the communication terminals 30A to 30C at locations A to C can either have built-in imaging devices 8 and 9, or they can be terminals to which imaging devices 8 and 9 can be attached externally. The communication terminals 30A to 30C can distribute images of their own location captured by their own imaging devices 8 and 9 to other locations. In addition, any devices can be placed at locations A to C.

[0052] The arrangement of each terminal and device (communication terminal 30 and imaging device) and the user shown in Figure 2 is just one example, and other examples are possible. Furthermore, the communication terminal 30 is not limited to a PC, but can also be, for example, a tablet terminal, smartphone, wearable device, projector, electronic whiteboard (a whiteboard with electronic blackboard functionality that enables mutual communication), or autonomous mobile robot. The communication terminal 30 can be any computer running a dedicated application for a web browser or image distribution service.

[0053] Furthermore, if the imaging device 10 has a display, it can be configured to display images distributed from other locations.

[0054] The information processing system 50 has one or more information processing devices. The information processing system 50 manages and controls communication between the imaging devices 10 and communication terminals 30 at each location, and manages the wide-field images that are transmitted and received. The information processing system 50 provides a platform that allows users to utilize the functions necessary to provide an image distribution service that distributes wide-field images. This platform can be used by service providers such as individuals and companies that wish to provide image distribution services through a contract. Hereinafter, in order to distinguish them from tenants that receive image distribution services, service providers that provide image distribution services to users using the platform will be referred to as platform subscribers.

[0055] Therefore, the information processing system 50, acting as a platform, exposes an API (Application Programming Interface), and platform subscribers can use this API to provide various image distribution services. Platform subscribers only need to develop software such as applications that handle screens displayed by the communication terminal 30 and API calls, and do not need to develop functions provided by the API, such as image distribution, from scratch.

[0056] The information processing system 50 can be constructed using a single computer, or it can be constructed using multiple computers, each part (function or means) of which is arbitrarily assigned. Furthermore, all or part of the functions of the information processing system 50 can be server computers located in a cloud environment, or server computers located in an on-premises environment.

[0057] Storage 90 is a storage device separate from the information processing system 50. Storage 90 can be external storage (including storage located on the cloud and storage located on-premises) or it can be storage included in the information processing system 50.

[0058] <Example Hardware Configuration> Next, the hardware configuration of each device or terminal in the image communication system according to this embodiment will be described using Figures 3 and 4. Note that the hardware configuration shown in Figures 3 and 4 may have components added or removed as needed.

[0059] <<Hardware configuration of the imaging device>> First, the hardware configuration of the imaging device 10 will be explained using Figure 3. Figure 3 is a diagram showing an example of the hardware configuration of the imaging device 10. In the following, the imaging device 10 will be described as a 360-degree (omnidirectional) imaging device using two image sensors, but there may be more than two image sensors. Furthermore, it is not necessarily a device dedicated to omnidirectional imaging; by attaching an omnidirectional imaging unit to a regular digital camera or smartphone, the device will have essentially the same functionality as the imaging device 10.

[0060] As shown in Figure 3, the imaging device 10 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output interface 116, a short-range communication circuit 117, an antenna 117a for the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network interface 121.

[0061] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lens 102 unless otherwise specified) each having a field of view of 180° or more for forming a hemispherical image, and two image sensors 103a and 103b corresponding to each lens. The image sensors 103a and 103b have an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the lenses 102a and 102b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers for setting various commands or parameters necessary for the operation of the image sensors. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example; it can also have only one, or three or more.

[0062] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, operation unit 115, input / output I / F 116, short-range communication circuit 117, electronic compass 118, gyro sensor 119, acceleration sensor 120, and network I / F 121, etc.

[0063] The image processing unit 104 receives image data output from image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create the data for the equirectangular projection image described later.

[0064] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 acting as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0065] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 10, there may also be functions to display a preview or video on a display (for example, the display of an external terminal such as a smartphone that communicates with the imaging device 10 via a short-range communication circuit 117). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0066] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit can be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0067] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress of processing by the image processing unit 104 and data of completed equirectangular projection images.

[0068] The control unit 115 is a collective term for various operation buttons, a power switch, a shutter button, and a touch panel that combines display and operation functions. The user inputs various imaging modes, imaging conditions, etc., by operating the control unit 115.

[0069] The input / output interface (I / F) 116 is a general term for interface circuits (such as USB I / F) to external media such as SD cards or personal computers. The I / F 116 can be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded to external media via the I / F 116, or transmitted to an external terminal (device) via the I / F 116 as needed.

[0070] The short-range communication circuit 117 communicates with an external terminal (device) via an antenna 117a provided on the imaging device 10 using a short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi. The short-range communication circuit 117 can transmit equirectangular projection image data to the external terminal (device).

[0071] The electronic compass 118 calculates the orientation of the imaging device 10 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) according to Exif and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data. The gyro sensor 119 is a sensor that detects changes in angle (Roll angle, Pitch angle, Yaw angle) associated with the movement of the imaging device 10. The changes in angle are an example of related information (metadata) according to Exif and are used for image processing such as image correction of captured images. Furthermore, the acceleration sensor 120 is a sensor that detects acceleration in three axes. The imaging device 10 calculates its own orientation (angle relative to the direction of gravity) based on the acceleration detected by the acceleration sensor 120. By providing the acceleration sensor 120, the accuracy of image correction of the imaging device 10 is improved.

[0072] Network I / F121 is an interface for data communication using a communication network N, such as the Internet, via a router or similar device.

[0073] <<Hardware configuration of the communication terminal>> Figure 4 shows an example of the hardware configuration of the communication terminal 30 and the information processing system 50. First, the communication terminal 30 will be described. The hardware configuration of the communication terminal 30 is indicated by codes in the 300 series. The communication terminal 30 is built by a computer and, as shown in Figure 4, is equipped with a CPU 301, ROM 302, RAM 303, HD (Hard Disk) 304, HDD controller 305, display 306, external device connection I / F 308, network I / F 309, bus line 310, keyboard 311, pointing device 312, DVD-RW (Digital Versatile Disk Rewritable) drive 314, media I / F 316, sound input / output I / F 317, microphone 318, speaker 319, short-range communication circuit 320, and camera 321.

[0074] Of these components, the CPU 301 controls the overall operation of the communication terminal 30. The ROM 302 stores programs used to drive the CPU 301, such as IPLs. The RAM 303 is used as the work area for the CPU 301. The HD 304 stores various data, such as programs. The HDD controller 305 controls the reading or writing of various data to the HD 304 according to the control of the CPU 301. The display 306 displays various information such as cursors, menus, windows, characters, or images. The display 306 is an example of a display unit. The display 306 can also be a touch panel display equipped with input means. The external device connection I / F 308 is an interface for connecting various external devices. In this case, external devices include, for example, USB memory or printers. The network I / F 309 is an interface for data communication using the communication network N. The bus line 310 is an address bus or data bus, etc., for electrically connecting each component, such as the CPU 301 shown in Figure 4.

[0075] The keyboard 311 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting or executing various instructions, selecting processing targets, or moving a cursor, etc. Note that the input means may not be limited to the keyboard 311 and the pointing device 312, but may also be a touch panel or an audio input device, etc. The DVD-RW drive 314 controls the reading or writing of various data to the DVD-RW 313, which is an example of a removable recording medium. Note that the DVD-RW 313 may be a DVD-R or a Blu-ray® Disc, etc. The media I / F 316 controls the reading or writing (storage) of data to the recording medium 315, such as flash memory. The microphone 318 is a type of built-in sound collection means for inputting sound. The sound input / output I / F 317 is a circuit that processes the input and output of sound signals between the microphone 318 and the speaker 319 according to the control of the CPU 301. The short-range communication circuit 320 is a communication circuit for communicating with an external terminal (device) using short-range wireless communication technology such as NFC, Bluetooth (registered trademark), or Wi-Fi. The camera 321 is a type of built-in imaging means that captures an image of a subject and obtains image data. Note that the microphone 318, speaker 319, and camera 321 may be external devices rather than being built into the communication terminal 30.

[0076] <<Hardware configuration of the information processing system>> As shown in Figure 4, each hardware component of the information processing system 50 is indicated by a 500-series code in parentheses. The information processing system 50 is built using a computer and has a configuration similar to that of the communication terminal 30, as shown in Figure 4; therefore, a description of each hardware component is omitted.

[0077] The above programs can be distributed as installable or executable files recorded on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc (registered trademark), SD card, and USB memory. The recording media can also be provided domestically or internationally as a program product. For example, the communication terminal 30 realizes the image display method according to the present invention when the program according to the present invention is executed.

[0078] <Regarding wide-field images and viewpoint information> The following section explains how to generate wide-field images (spherical images) using Figures 5 to 12.

[0079] First, the external appearance of the imaging device 10 will be described using Figure 5. The imaging device 10 is a digital camera for obtaining captured images that will form the basis of a 360° spherical image. Figure 5(a) is a left side view of the imaging device, Figure 5(b) is a front view of the imaging device, and Figure 5(c) is a top view of the imaging device. This external view is merely one example of the imaging device 10, and other external appearances are possible.

[0080] As shown in Figure 5(a), the imaging device 10 is small enough to be held in one hand, but this shape is merely an example, and it can take on other shapes. Also, as shown in Figures 5(a), 5(b), and 5(c), the imaging device 10 has an image sensor 103a on the front side and an image sensor 103b on the back side at the top. These image sensors 103a and 103b are used in conjunction with optical components (for example, lenses 102a and 102b, which will be described later) that can capture hemispherical images (angle of view of 180° or more). Also, as shown in Figure 5(b), an operating section 115 such as a shutter button is provided on the side of the imaging device 10 opposite to the front side. As mentioned above, the imaging device can have only one image sensor, or it can have three or more.

[0081] Next, the usage of the imaging device 10 will be explained using Figure 6. Figure 6 is an illustrative diagram of the imaging device in use. As shown in Figure 6, the imaging device 10 is used, for example, to image a subject around the imaging device 10. In this case, two hemispherical images can be obtained by imaging the subject around the imaging device 10 using the image sensors 103a and 103b shown in Figure 5, respectively.

[0082] Next, using Figures 7 and 8, we will outline the process from the image captured by the imaging device 10 to the creation of a full-sphere image. Figure 7(a) shows the hemispherical image (front side) captured by the imaging device, Figure 7(b) shows the hemispherical image (back side) captured by the imaging device, and Figure 7(c) shows the image represented by equirectangular projection (hereinafter referred to as "equistratic projection image"). Figure 8(a) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 8(b) shows the full-sphere image.

[0083] As shown in Figure 7(a), the image obtained by the image sensor 103a is a curved hemispherical image (front side) by the lens 102a described later. Similarly, as shown in Figure 7(b), the image obtained by the image sensor 103b is a curved hemispherical image (rear side) by the lens 102b described later. The imaging device 10 then combines the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) to create an equirectangular projection image EC as shown in Figure 7(c).

[0084] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to overlay an equirectangular projection image EC so that it covers the sphere, as shown in Figure 8(a), and creates a full-sphere image (full-sphere panoramic image) CE as shown in Figure 8(b). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. OpenGL ES is merely one example of software that performs image processing, and the full-sphere image CE can also be created using other software. Furthermore, the full-sphere image CE can be a still image or a video.

[0085] As described above, the 360-degree spherical image CE is an image pasted to cover a sphere, which can cause discomfort to the human eye. Therefore, the imaging device 10 can display a predetermined area T of the 360-degree spherical image CE (hereinafter referred to as the "predetermined area image") as a planar image with less curvature, thereby providing a display that does not cause discomfort to the human eye. This will be explained with reference to Figures 9 to 10.

[0086] Figure 9 shows the positions of the virtual camera and a predetermined region when the 360-degree image is treated as a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of the user viewing the 360-degree image CE, which is displayed as a three-dimensional sphere. Figure 10(a) is a stereoscopic perspective view of Figure 9, and Figure 10(b) shows the predetermined region image as it appears on a display. Figure 10(a) represents the 360-degree image CE shown in Figure 9 as a three-dimensional sphere CS. If the 360-degree image CE generated in this way is a sphere CS, then, as shown in Figure 9, the virtual camera IC is located inside the 360-degree image CE. The predetermined region T in the 360-degree image CE is the imaging area of ​​the virtual camera IC and is identified by predetermined region information indicating the imaging direction and field of view of the virtual camera IC in the three-dimensional virtual space including the 360-degree image CE. Furthermore, zooming in on the predetermined region T can also be represented by moving the virtual camera IC closer to or further away from the 360-degree image CE. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. Therefore, the predetermined region T can be determined by the field of view α and the distance f from the virtual camera IC to the 360-degree spherical image CE (see Figure 11).

[0087] Then, the predetermined region image Q shown in Figure 10(a) is displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in Figure 10(b). The image shown in Figure 10(b) is a predetermined region image represented by the initially set (default) predetermined region information. The following explanation will use the imaging direction (ea, aa) and field of view (α) of the virtual camera IC. Note that the predetermined region T can also be represented by the position coordinates (X, Y, Z) of the imaging area of ​​the virtual camera IC, which is the predetermined region T, rather than by the field of view α and distance f.

[0088] Next, we will explain the relationship between the predetermined region information and the image of the predetermined region T using Figure 11. Figure 11 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 11, "ea" is the elevation angle, "aa" is the azimuth angle, and "α" is the field of view (Angle). That is, the orientation of the virtual camera IC is changed so that the point of fixation of the virtual camera IC, indicated by the imaging direction (ea, aa), becomes the center point CP(x, y) of the predetermined region T, which is the imaging area of ​​the virtual camera IC. As shown in Figure 11, the center point CP(x, y) when the diagonal field of view of the predetermined region T, represented by the field of view α of the virtual camera IC, is α becomes the parameter ((x, y)) of the predetermined region information. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between any vertex of a given region T and the center point CP(x, y) (2L is the diagonal). In Figure 11, the following trigonometric function generally holds:

[0089]

number

[0090] Figure 12 shows the relationship described in Figure 11 as a point in a three-dimensional Euclidean space using spherical coordinates. Here, the position coordinates of the center point CP shown in Figure 11, expressed in spherical polar coordinates, are (r, θ, φ). (r, θ, φ) are the radial, polar angle, and azimuth angle, respectively. The radial r is the distance from the origin of the three-dimensional virtual space containing the full-sphere image to the center point CP, and is therefore equal to the distance f shown in Figure 11. Figure 12 is a diagram illustrating these relationships. Hereafter, the position coordinates (r, θ, φ) of the virtual camera IC will be used as an example of viewpoint information for explanation. The viewpoint information only needs to be parameter information that can identify a predetermined region T (predetermined region image Q) that is displayed as an image of the imaging area of ​​the virtual camera IC on the predetermined display shown in Figure 10, as described above, and includes the coordinates of the diagonal vertices of the predetermined region T. In addition, the information indicating the field of view α of the virtual camera IC and the information indicating the center point CP(x, y) described in Figure 11 can also be used as viewpoint information. Furthermore, viewpoint information includes not only position coordinate information in spherical coordinates, but also position coordinate information in Cartesian coordinates and coordinate difference values ​​from the initially set (default) predetermined area information. In addition, viewpoint information can also be information other than coordinate information, such as angles and distances, as shown in Figure 11. Also, although Figures 11 and 12 use the center point of the predetermined area T as the reference, the predetermined area T can be identified by parameter information based on any of the vertices of the predetermined area T. Note that the above explanation of viewpoint information was given using the example of a 360-degree spherical image as the wide-field image, but in the case of other wide-field images, the information that identifies the predetermined area T in that wide-field image will be the viewpoint information.

[0091] <About the features> Next, the functional configuration of the communication system 1 according to this embodiment will be described using Figure 13. Figure 13 is a diagram showing an example of the functional configuration of the communication system 1 according to this embodiment. In Figure 13, the terminals, devices, and servers shown in Figure 1 that are related to the processing or operation described later are shown.

[0092] <<Functional Configuration of Imaging Device>> First, the functional configuration of the imaging device 10 will be described with reference to Figure 13. The imaging device 10 includes a communication unit 11, a reception unit 12, an imaging processing unit 13, an analysis unit 14, a registration request unit 15, a connection unit 16, a storage processing unit 17, an image transmission control unit 18, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 111 that follow a program deployed on the SRAM 113 or DRAM 114. The imaging device 10 also has a storage unit 1000 constructed from a ROM 112, etc., as shown in Figure 3.

[0093] The communication unit 11 has the function of connecting to a communication network N using wireless communication means such as Wi-Fi and sending and receiving various data or information with other devices. In this embodiment, the connection unit 16 mainly transmits the wide-field image acquired by the imaging processing unit 13 to the information processing system 50, but it is also possible for the communication unit 11 to transmit the wide-field image to the information processing system 50.

[0094] The reception unit 12 is a function that receives user input for the imaging device 10. The reception unit 12 receives input such as power on / off, shutter button on / off (start or stop transmission of wide-field images), etc.

[0095] The imaging processing unit 13 captures images of subjects, landscapes, etc., and acquires the captured images. The captured images acquired by the imaging processing unit 13 may be videos or still images (or both), and may include sound along with the images. In addition, the imaging processing unit 13 captures images of, for example, a two-dimensional code (see Figure 19) displayed on the display 306 of the communication terminal 30.

[0096] The analysis unit 14 analyzes the two-dimensional code captured and acquired by the imaging processing unit 13 and extracts the information contained in the two-dimensional code (URL for registering the imaging device with the tenant, temporary ID and password).

[0097] The registration request unit 15 uses the information contained in the two-dimensional code read by the analysis unit 14 to send a request to the information processing system 50 to register the imaging device 10 as a tenant of the information processing system 50.

[0098] The connection section 16 is implemented, for example, by a short-range communication circuit 117, and has the function of receiving power supply from the communication terminal 30A and performing data communication.

[0099] The storage processing unit 17 processes the wide-field images captured in response to imaging requests from any location and saves them to a URL (e.g., storage 90) notified by the information processing system 50.

[0100] The image transmission control unit 18 controls the transmission of wide-field images to the information processing system 50. For example, the image transmission control unit 18 transmits captured images acquired by the imaging processing unit 13 to the information processing system 50 periodically or in response to user operation if they are still images, or at a predetermined FPS (Flame Per Second) if they are videos. The image transmission control unit 18 also switches between the communication unit 11 and the connection unit 16.

[0101] The storage / reading unit 19 has the function of storing various data in the storage unit 1000 or reading various data from the storage unit 1000. The storage unit 1000 also stores the image capture data acquired by the image processing unit 13, the image capture device ID, etc. The image capture data stored in the storage unit 1000 can be configured to be deleted after a predetermined time has elapsed since it was acquired by the image processing unit 13, or it can be configured to be deleted when data is transmitted to the information processing system 50.

[0102] Furthermore, the imaging device 10 has an application (also called a plug-in) installed to support the communication system 1. This application is not necessary for commercially available imaging devices 10, but it is used when associating the imaging device 10 with a virtual room and when receiving control from external sources. Some of the functions shown in Figure 13 (for example, the registration request unit 15) are provided by this application.

[0103] <<Communication terminal function configuration>> Next, the functional configuration of the communication terminal 30 will be explained using Figure 13. The communication terminal 30 has a communication unit 31, a reception unit 32, a display control unit 33, an imaging unit 34, a storage / reading unit 35, and a connection unit 36. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 301 following a program (which may be a web browser or a dedicated application) deployed on the RAM 303. The communication terminal 30 also has a storage unit 3000 constructed from the ROM 302 or recording medium 315 shown in Figure 4.

[0104] The communication unit 31 is implemented, for example, by a network interface 309, and is connected to a communication network N, performing functions such as sending and receiving various data or information with other devices.

[0105] The reception unit 32 has the function of receiving various selections or operation inputs to the communication terminal 30. The display control unit 33 has the function of displaying wide-field images, normal-angle images, and various screens on the display 306 of the communication terminal 30. The display control unit 33 displays, for example, a two-dimensional code transmitted from the information processing system 50 on the display 306. The two-dimensional code may be, for example, QR code (registered trademark), DataMatrix (DataCode), MaxiCode, or PDF417. The two-dimensional code may also be a barcode.

[0106] The connection section 36 is implemented, for example, by a short-range communication circuit 320, and has the function of supplying power to the imaging device 10 and performing data communication.

[0107] The storage / reading unit 35 is executed by instructions from the CPU 301 shown in Figure 4, and has the function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. The storage unit 3000 has an image management information storage unit 3001. The image management information stored in the image management information storage unit 3001 will be explained in the description of the information processing system 50.

[0108] <<Functional Configuration of the Information Processing System>> Next, the functional configuration of the information processing system 50 will be described. The information processing system 50 includes a communication unit 51, a screen generation unit 52, an association processing unit 53, an image distribution unit 54, an authentication unit 55, a communication group management unit 56, a communication control unit 57, a connection management unit 58, a storage / reading unit 59, an API management unit 60, a viewing status recording unit 61, a viewing status aggregation unit 62, a focus point identification unit 63, and a focus point determination unit 64. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with a program deployed on the RAM 503. The information processing system also includes a storage unit 5000 constructed from a ROM 502, HD 504, or recording medium 515 shown in Figure 4.

[0109] The communication unit 51 has the function of sending and receiving various data or information with other devices via the communication network N.

[0110] The screen generation unit 52 generates screen information to be displayed by the communication terminal 30. When the communication terminal 30 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc. When the communication terminal 30 runs a native application, the screen information is held by the communication terminal 30, and the information to be displayed is transmitted in XML, etc. The screen generation unit 52 generates screen information on which wide-field images, etc., distributed by the image distribution unit 54 are placed.

[0111] The association processing unit 53 controls the sharing of viewpoint information for wide-field images. When the association processing unit 53 receives an imaging request along with viewpoint information from the communication terminal 30, it requests imaging from the imaging device 10 and processes the acquisition of a wide-field image and the association of the viewpoint information. Furthermore, it stores the associated wide-field image and viewpoint information in the image management information storage unit 5001. The association processing unit 53 also sends a URL to the communication terminal 30 as information indicating the storage location where the associated wide-field image and viewpoint information are stored. Note that the information processing system 50 does not need to receive viewpoint information and imaging requests from the communication terminal 30 simultaneously; it can receive them separately and then process the association. Also, the URL is just one example of information indicating a storage location, and other formats such as URIs can be used.

[0112] The image distribution unit 54 distributes wide-field images transmitted by the imaging device 10, which is associated with the same virtual room, to the communication terminal 30 operated by the user who is in the virtual room. Images of the normal field of view captured by the camera or connected imaging devices 8 and 9 of the communication terminal 30 are also distributed in the same manner.

[0113] In the following description, images distributed from the information processing system 50 to the communication terminals 30 at each location will be referred to as "distributed images," and the image data representing the distributed images will be referred to as "distributed image data." In this embodiment, the distributed image data is a video (moving image) consisting of multiple consecutive frames. In this embodiment, the distributed image is each frame in the distributed image data. Screen information for displaying the distributed image is generated by the screen generation unit 52. Note that the distributed image data can also be a still image.

[0114] The distributed images include images from multiple locations that are in the same virtual room. The images from each location are images captured by imaging devices 8 and 9 installed at that location.

[0115] Furthermore, the images of locations included in the distributed images include images showing normal field-of-view images captured at multiple locations input to the same virtual room, and wide-field-of-view images captured by the imaging device 10 associated with this virtual room. In the following explanation, images with a normal field of view may be referred to as normal field-of-view images, and image data showing normal field-of-view images may be referred to as normal field-of-view image data. Also, in the following explanation, image data showing wide-field-of-view images may be referred to as wide-field-of-view image data.

[0116] The authentication unit 55 is a function that authenticates the requester based on the authentication request received by the communication unit 51. For example, the authentication unit 55 authenticates the user by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, facial recognition, biometric information such as fingerprints or voiceprints, etc. The authentication unit 55 can also authenticate using an external authentication system or an authentication method such as OAuth.

[0117] The communication group management unit 56 manages the entry of communication terminals 30 and users into the virtual room, as well as the mapping of devices. When authentication by the authentication unit 55 is successful, the communication group management unit 56 registers the user ID and the IP address of the communication terminal 30 in the virtual room information storage unit 5002, and maps the imaging device 10 to the virtual room.

[0118] The communication control unit 57 manages the start, establishment, and termination of communication with the imaging device 10 associated with each virtual room. The communication control unit 57 also manages the start, establishment, and termination of communication for distributing wide-field images and audio in response to the communication terminal 30 entering or leaving the virtual room.

[0119] The connection management unit manages the communications (connections) established between the communication terminal 30 and the imaging device 10 and the information processing system 50, associating them with virtual rooms.

[0120] The API management unit 60 manages the APIs used by platform subscribers when providing wide-field image distribution services. When using the APIs, platform subscribers only need to develop separate software to call the APIs. This software can run on a server or on a client such as a communication terminal.

[0121] Any function provided by the information processing system 50, such as the image distribution unit 54, the association processing unit 53, and the communication control unit 57, can be provided as an API. Furthermore, functions added to the information processing system 50 later can also be provided as APIs. Whether or not to provide an API is determined by the communication terminal operated by the platform provider accessing the information processing system 50 and receiving the API disclosure setting, allowing the API management unit 60 to control the API based on the disclosure setting. The API management unit 60 can also perform authentication processing to verify whether the requesting software making the API call is software developed by a legitimate platform subscriber.

[0122] Authentication can be verified by comparing the information stored in the storage unit 5000 with the information sent from the requesting software. As a specific example of the authentication process, the information processing system 50 receives an application ID, which has been previously issued by the API management unit 60, from the requesting software to the software developed by the platform contractor. If the API management unit 60 determines that the application ID is stored in the storage unit 5000, the API management unit 60 performs control to permit the provision of APIs as legitimate software.

[0123] On the other hand, if the API management unit 60 cannot determine that the software is legitimate, it will implement control to prevent the provision of the API. The application ID is an example of authentication information used to determine legitimacy. The API management unit 60 can also verify the legitimacy of the requester using authentication information such as an access token, ticket, security key, password, or PIN code issued in advance by the API management unit 60 of the information processing system or an external system. In this embodiment, the form in which the functions provided by the information processing system 50 are used as an API is not described, but the processing flow is the same except that the software such as an application developed by the platform subscriber uses the functions provided by the information processing system 50 after being judged by the API management unit 60.

[0124] The memory / reading unit 59 has the function of storing various data in the memory unit 5000 or reading various data from the memory unit 5000. The memory unit 5000 also includes an image management information storage unit 5001, a virtual room information storage unit 5002, a viewing status storage unit 5004, and an aggregated result storage unit 5005.

[0125] The viewing status recording unit 61 stores viewing status information in the viewing status storage unit 5004 provided in the storage unit 5000. Specifically, the viewing status recording unit 61 stores information indicating the location of interest at each location's communication terminal 30 as viewing status information, and stores it in the viewing status storage unit 5004 in association with the distributed image data transmitted to each location's communication terminal 30. The viewing status information is associated with the distributed image data, for example, by the virtual room ID of the virtual room in which each location's communication terminal 30 is entered.

[0126] A location of interest is, for example, a location that captured an enlarged image from among the images of multiple locations included in the distributed image displayed on the communication terminal 30 at each location.

[0127] Furthermore, if the distributed image includes a wide-field image, the viewing status recording unit 61 stores viewpoint information indicating the viewpoint of interest in the wide-field image as viewing status information in the viewing status storage unit 5004 in association with the distributed image data transmitted to the communication terminal 30 at each location.

[0128] The viewpoint of interest is, for example, a designated viewpoint in the wide-field image displayed on the communication terminal 30 at each location. In the following explanation, information indicating the location of interest will be referred to as "location of interest information," and viewpoint information indicating the designated viewpoint in the wide-field image will be referred to as "designated viewpoint information."

[0129] In other words, the viewing status information of this embodiment includes attention location information indicating other locations that are attracting attention at the communication terminal 30 of each location that has entered the virtual room, and designated viewpoint information indicating the viewpoint in the wide-field image designated at each location that has entered the virtual room.

[0130] The viewing status aggregation unit 62 aggregates the viewing status information stored in the viewing status storage unit 5004 and stores the aggregation result information showing the aggregation result in the aggregation result storage unit 5005. Specifically, the viewing status aggregation unit 62 aggregates the number of other locations that are paying attention to each location for each timestamp when the distributed image data is distributed, based on the attention location information included in the viewing status information. The viewing status aggregation unit 62 then stores the aggregated result as attention location number information in the aggregation result storage unit 5005.

[0131] The attention location identification unit 63 identifies the location that is attracting the most attention from the most locations at each timestamp when the distributed image data is distributed, based on the results of the aggregation by the viewing status aggregation unit 62. In the following explanation, the location that is attracting the most attention will be referred to as the most attention location.

[0132] The viewpoint determination unit 64 determines the viewpoint that is attracting the most attention in the wide-field image for each timestamp when the distributed image data is distributed, based on the specified viewpoint information included in the viewing status information. The viewpoint determined by the viewpoint determination unit 64 indicates the viewpoint that is attracting the most attention at the time indicated by the timestamp. In the following explanation, the viewpoint that is attracting the most attention will be called the most-attention viewpoint, and the viewpoint information that indicates the most-attention viewpoint will be called the most-attention viewpoint information.

[0133] The focus point determination unit 64 stores the most important focus point information as part of the aggregated result information in the aggregated result storage unit 5005. Therefore, the aggregated result information stored in the aggregated result storage unit 5005 in this embodiment includes the number of points of interest information and the most important focus point information.

[0134] The viewpoint determination unit 64 of this embodiment calculates the centroid from the viewpoints indicated by the viewpoint information of all locations included in the designated viewpoint information 5004b, and determines the viewpoint closest to the centroid as the viewpoint of most interest. Note that the method of determining the viewpoint of most interest in this embodiment is just one example and is not limited thereto.

[0135] In this embodiment, the screen generation unit 52, when generating screen information showing the distributed image, refers to the number of points of interest information stored in the aggregated result storage unit 5005 and generates screen information so that users of the communication terminal 30 at each location can identify the most important point of interest for each timestamp. Furthermore, when generating screen information showing the distributed image, the screen generation unit 52 refers to the most important viewpoint information stored in the aggregated result storage unit 5005 and generates screen information so that users of the communication terminal 30 at each location can identify the most important viewpoint for each timestamp.

[0136] In this embodiment, by generating screen information in this way, the most important location and viewpoint will be displayed in the distributed image data while the distributed image data is being distributed to the communication terminals 30 at each location. Therefore, according to this embodiment, users of the communication terminals 30 at each location can be made aware of the locations and viewpoints that users of the communication terminals 30 at other locations are paying attention to.

[0137] Furthermore, the information processing system 50 of this embodiment stores the distributed image data in the storage location indicated by the data storage location information included in the image management information described later, using the storage and reading unit 59. By storing the distributed image data in this way, users of the communication terminal 30 can review the distributed image data after it has been distributed.

[0138] Furthermore, in the information processing system 50 of this embodiment, when the distributed image data is reviewed on the communication terminal 30, the viewing status information associated with the distributed image data is used to identify, for each timestamp, the locations and viewpoints that users of communication terminals 30 at other locations were paying attention to, just as during distribution.

[0139] The image management information recording unit 5100 will be described below with reference to Figure 14. Figure 14(a) is a conceptual diagram showing the image management information stored in the image management information storage unit 5001. The image management information storage unit 5001 stores image management information as shown in Figure 14. Image management information is information for managing wide-field images captured in response to an imaging request, and when a user sends an imaging request from the communication terminal 30, one record of image management information is generated. Each item of the image management information will be described below.

[0140] • The data ID is identification information used to identify a wide-field image. The data ID is assigned by the information processing system 50. ID is an abbreviation of Identification and means identifier or identification information. An ID is a name, code, string, number, or a combination of one or more of these used to uniquely distinguish a particular object from multiple objects.

[0141] The data name is the name of the wide-field image set by the user of the communication terminal 30. The data name can be set by the user, but it can also be set automatically.

[0142] The imaging date and time information is information used to identify the date and time the wide-field image was captured, such as the date and time the user entered an imaging request into the communication terminal 30, and the date and time the imaging device 10 captured the wide-field image. The imaging date and time information can also be substituted with the timestamp of the wide-field image.

[0143] The imager information is the identification information (which may be the username) of the user who entered the imaging request into the communication terminal 30. Since the user enters the imaging request into the communication terminal 30 while inside the virtual room, the user registered in the imager information is identified by authentication to the information processing system 50 or the virtual room. The imager information is transmitted to the information processing system 50 along with the imaging request.

[0144] The imaging device information is the identification information (imaging device ID) of the imaging device 10 that captured the wide-field image. The imaging device ID is assigned by the information processing system 50 and shared with the imaging device 10, but information unique to the imaging device 10, such as the MAC address or serial number, can also be used. The imaging device ID is transmitted to the information processing system 50 along with the wide-field image.

[0145] The viewer's viewpoint information consists of radial radius, polar angle, and azimuth angle. The viewpoint information indicates the center coordinate of the wide-field image displayed by the communication terminal 30. The viewpoint information is transmitted from the communication terminal 30 requesting the image capture. In addition to radial radius, polar angle, and azimuth angle, the viewpoint information may also include information specifying the width and height of the display range. Alternatively, the viewpoint information may consist only of the width and height of the display range.

[0146] The virtual room ID used during imaging is the identification information of the virtual room to which the imaging device 10 is associated.

[0147] • Data storage location information includes the URL or file path where the wide-field images are stored.

[0148] Figure 14(b) is also a conceptual diagram showing image management information. In Figure 14(b), wide-field images with the same virtual room ID at the time of acquisition are stored. Image management information may be classified by virtual room.

[0149] The virtual room information storage unit 5002 will be described below with reference to Figure 15. Figure 15 is a conceptual diagram showing the virtual room information stored in the virtual room information storage unit 5002. The virtual room information storage unit 5002 stores virtual room information as shown in Figure 15. The virtual room information is information about a virtual room and is maintained for each virtual room. Each item of the virtual room information will be explained below. Note that virtual rooms are registered with tenants.

[0150] The virtual room ID is identification information that identifies the virtual room. In this embodiment, virtual rooms can be created at the user's discretion.

[0151] The virtual room name is a name used by users to identify the virtual room, and can be set by the user at will.

[0152] The imaging device information is the identification information (imaging device ID) of the imaging device 10 associated with the virtual room.

[0153] The user currently in the room is the user ID of the user currently in the virtual room. This user is capable of viewing wide-field images. The method of entering the room will be described later. In addition, the user ID is associated with the IP address of the communication terminal 30 operated by that user.

[0154] <Entering a virtual room using a communication terminal> Next, referring to Figure 16, the process of user B entering the virtual room will be explained. It is assumed that the imaging device 10 has already been associated with the virtual room, and that the communication terminal 30A has transmitted wide-field images and normal-angle images to the information processing system 50 (the association of the imaging device 10 with the virtual room will be explained in Figure 18 and later). Furthermore, in the following explanation, there will be no particular distinction made between user B entering the virtual room and the communication terminal 30B operated by user B entering the virtual room.

[0155] Figure 16 is a sequence diagram illustrating the process by which user B (or communication terminal 30B) enters a virtual room.

[0156] S1: First, user B at site B performs an operation to display the virtual room list screen. When the reception unit 32 receives the operation to display the list screen, the display control unit 33 of the communication terminal 30B displays the selection screen on the display 306.

[0157] S2: When user B selects a virtual room selection button, the reception unit 32 of the communication terminal 30B accepts the selection of a virtual room. The display control unit 33 of the communication terminal 30B displays the room entry screen 200 on the display 306.

[0158] S3: User B enters the required information and presses the enter button 203. Upon receiving the press from the reception unit 32, the communication unit 31 of the communication terminal 30B sends an entry request to the information processing system 50. This entry request includes the virtual room ID indicating the virtual room selected in step S2, the user ID of the logged-in user B, and the IP address of the requesting terminal, the communication terminal 30B.

[0159] S4: As a result, the communication unit 51 of the information processing system 50 receives the entry request. The communication group management unit 56 registers the logged-in user ID and IP address in the virtual room information identified by the virtual room ID in the virtual room information storage unit 5002.

[0160] S5: Then, the communication unit 51 of the information processing system 50 transmits a message to the communication terminal 30B indicating that the room has been entered. As a result, the communication unit 31 of the communication terminal 30B receives the message that the room has been entered.

[0161] Here, we will explain the process for associating the imaging device 10 with a virtual room. In the following explanation, we will assume that the communication system 1 is applied to a construction site, for example. In this case, location A is the construction site, and locations B and C can be any location that can communicate with location A, such as an office. At the construction site, various construction work is being carried out by workers at each location. By applying the communication system 1 of this embodiment to the construction site, the entire site can be captured using a wide-field image, and if there is any construction work or task of interest, users at each location A to C can arbitrarily change their virtual viewpoint to check it.

[0162] Figure 17 shows an example of the virtual room mapping screen (part 1) 260 for associating the imaging device 10 with a virtual room. The screen configuration can be the same for VR goggles 89 and smart glasses 88. The virtual room mapping screen (part 1) 260 has a virtual room list 261. The virtual room list 261 displays individual virtual room fields 262 to 264 based on the virtual rooms created for each construction site.

[0163] Each individual virtual room field 262-264 has a link issuance button 265, an entry button 266, a settings button 267, and a virtual room name 268. The link issuance button 265 is a button for issuing a link to the virtual room (a URL for invitation) and a passcode. The entry button 266 is a button for user A to enter the virtual room. The settings button 267 is a button for associating the imaging device 10 with the virtual room. The virtual room name 268 is the same as the one stored in the virtual room information storage unit 5002. Therefore, user A presses the settings button 267. Upon pressing the settings button 267, the communication terminal 30A displays the virtual room association screen (part 2) 270.

[0164] Additionally, if a device is already associated with a virtual room, the device name 269 will be displayed in the individual virtual room section (individual virtual room section 264 in the diagram).

[0165] Figure 18 shows an example of the virtual room mapping screen (part 2) 270. Note that the virtual room mapping screen (part 2) 270 is displayed as a pop-up on the virtual room mapping screen (part 1) 260. The screen transition from the virtual room mapping screen (part 1) 260 to the virtual room mapping screen (part 2) 270 does not go through the information processing system 50, but a screen transition that does go through the information processing system 50 is also possible.

[0166] The virtual room mapping screen (part 2) 270 has the name 271 of the imaging device 10 currently (already) mapped to the virtual room (not yet registered, so not shown in the diagram), a connect button 272, and a storage button 273. The connect button 272 is a button that displays a list of devices registered to the tenant. The storage button 273 is a button that displays a list of storage 90 where the imaging device 10 mapped to the virtual room saves wide-field images. When the connect button 272 is pressed, the communication terminal 30A displays the virtual room mapping screen (part 3).

[0167] The communication terminal 30A transmits a virtual room ID to the information processing system 50 and obtains the name of the device registered in the tenant where the virtual room is created (including the device ID, etc.), and the name of the device associated with the virtual room (including the device ID, etc.).

[0168] Figure 19 shows an example of the virtual room mapping screen (part 3) 280. The virtual room mapping screen (part 3) 280 has the name 281 of the imaging device 10 currently (already) mapped to the virtual room, a list of addable devices 282, and a save button 283. User A selects the device they want to map to the virtual room from the list of addable devices 282 and presses the save button 283. This maps the device to the virtual room (the imaging device ID is registered in the virtual room information storage unit 5002).

[0169] <Processing to initiate transmission of wide-field images to the imaging device> With the above steps, devices such as the imaging device 10 are now associated with the virtual room, but user A needs to initiate image transmission to the device.

[0170] For the VR goggles 89 and smart glasses 88, user A turns image transmission on and off by operating the device itself. This is because currently, there is no dedicated application running on the communication system 1 for the VR goggles 89 and smart glasses 88. If a dedicated application were running on the communication system 1 for the VR goggles 89 and smart glasses 88, user A would be able to remotely turn image transmission on and off.

[0171] In the case of the imaging device 10, if the application is enabled, user A can enter the virtual room and turn the transmission of wide-field images on or off from the menu.

[0172] Figure 20 shows an example of a wide-field image transmission start / stop dialog 290 displayed by the communication terminal 30A. The wide-field image transmission start / stop dialog 290 is displayed as a pop-up on the image viewing screen 210. Assume that user A has entered a virtual room associated with the imaging device 10 by operating the communication terminal 30A. The wide-field image transmission start / stop dialog 290 displays the name 292 of the imaging device 10 associated with this virtual room. A toggle button 291 is displayed near the name 292, and user A can operate the toggle button 291 to turn on or off the transmission of wide-field images by the imaging device 10. Note that the method of setting on or off using the toggle button is just one example, and it is sufficient if it can be set according to user input. For example, it can also be set by selecting radio buttons or predetermined icons, or by menu operation. Alternatively, the transmission of wide-field images can be started automatically after the imaging device 10 enters the room, without requiring user operation. Furthermore, it is possible to pre-determine certain conditions, such as the date and time, the number of users who entered the room, or whether a specific user participated, and then start transmitting wide-field images when it is determined that these conditions have been met.

[0173] The communication terminal 30A transmits the state of the toggle button 291 to the information processing system 50. The information processing system 50 transmits a transmission start request or a transmission stop request to the imaging device 10 according to the state of the toggle button 291.

[0174] Figure 20(a) shows the toggle button 291 in the off state. Therefore, the wide-field image is not displayed in Figure 20(a). On the other hand, in Figure 20(a), when the communication terminal 30A enters the room, the image of the normal field of view captured by the camera 9 of the communication terminal 30A has already been shared and is displayed on the image viewing screen 210.

[0175] Figure 20(b) shows the toggle button 291 in the ON state. When the toggle button 291 is turned ON, the information processing system 50 sends an ON request to the imaging device 10, causing the imaging device 10 to start transmitting the wide-field image. As a result, two images are shared in one virtual room, and the image viewing screen 210 is divided into two.

[0176] <<Procedure for registering the imaging device in the virtual room>> Next, referring to Figure 21, the procedure for registering the imaging device 10 to the virtual room, as described in the screen transition section, will be explained. Figure 21 is an example of a sequence diagram showing the procedure for user A to register the imaging device 10 to the virtual room.

[0177] S11: First, user A connects the communication terminal 30A to the information processing system 50, enters authentication information (user ID, password, etc.), and requests to log in to the tenant to which they belong. The reception unit 32 of the communication terminal 30A accepts the operation.

[0178] S12: The communication unit 31 of the communication terminal 30A sends a login request to the information processing system 50, specifying the authentication information. The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 55 performs authentication. Here, we assume that authentication was successful.

[0179] S13: In response to user operation, the screen generation unit 52 of the information processing system 50 generates a device registration screen 220, and the communication unit 51 transmits the screen information of the device registration screen 220 to the communication terminal 30A.

[0180] S14: The communication terminal 30A displays the device registration screen based on the received screen information. More specifically, the communication unit 31 of the communication terminal 30A receives the screen information of the device registration screen 220, and the display control unit 33 displays the device registration screen 220.

[0181] S15: User A selects the type of device (in this case, the imaging device 10 is selected), and then enters the name and description of the imaging device 10. The reception unit 32 receives the input.

[0182] S16: The communication unit 31 of the communication terminal 30A sends a request for a two-dimensional code to the information processing system 50, specifying the name and description entered by user A.

[0183] S17: The communication unit 51 of the information processing system 50 receives a request for a two-dimensional code. The communication group management unit 56 generates a URL (a connection destination for registration) associated with the name and description, and generates a two-dimensional code containing the URL, a temporary ID, and a password.

[0184] S18: The communication unit 51 of the information processing system 50 transmits a two-dimensional code to the communication terminal 30A.

[0185] S19: The communication unit 31 of the communication terminal 30A receives the two-dimensional code, and the display control unit 33 displays the two-dimensional code.

[0186] S20: Next, user A operates the imaging device 10 that they want to associate with the virtual room to capture a two-dimensional code. The reception unit 12 of the imaging device 10 receives the operation.

[0187] S21: The imaging processing unit 13 of the imaging device 10 generates image data by performing imaging processing including a two-dimensional code, and the analysis unit 14 analyzes the image data to extract a URL, a temporary ID, and a password. As a result, the registration request unit 15 connects to the URL via the connection unit 16, specifies the temporary ID and password, and sends a registration request from the imaging device 10 to the information processing system 50.

[0188] S22: The communication unit 51 of the information processing system 50 receives a temporary ID and password, and the authentication unit 55 determines whether they match the temporary ID and password associated with the connected URL. Here, we assume they match.

[0189] S23: The communication group management unit 56 of the information processing system 50 receives a request to register the imaging device 10, so it generates an imaging device ID and registers it in the tenant when user A logs in. The imaging device ID is associated with a name and description.

[0190] S24: The communication unit 51 of the information processing system 50 transmits the imaging device ID to the imaging device 10. The connection unit 16 of the imaging device 10 receives the imaging device ID and stores it in the storage unit 1000.

[0191] S25: The communication terminal 30A is notified that registration is complete, and user A begins associating the imaging device 10 with the virtual room.

[0192] S26: User A displays the virtual room mapping screen (part 1) 260 on the communication terminal 30A.

[0193] S27: The communication terminal 30A selects the virtual room to which it wants to associate the imaging device 10 registered with the tenant. The reception unit 32 of the communication terminal 30A accepts the selection.

[0194] S28: Next, user A displays the virtual room mapping screen (part 2) 270 on the communication terminal 30A.

[0195] S29: User A presses the Add Device button. The reception unit 32 of the communication terminal 30A receives the press.

[0196] S30: The communication unit 31 of the communication terminal 30A requests the information processing system 50 for devices registered with the tenant and devices associated with the virtual room ID selected in step S29.

[0197] S31: The communication unit 51 of the information processing system 50 receives requests from devices registered with the tenant and devices associated with the virtual room ID, and the screen generation unit 52 generates a virtual room mapping screen (part 3) 280 that includes the device IDs of the devices registered with the tenant and devices associated with the virtual room ID. The communication unit 51 of the information processing system 50 transmits the screen information of the virtual room mapping screen (part 3) 280 to the communication terminal 30A.

[0198] S32: The communication unit 31 of the communication terminal 30A receives screen information of the virtual room mapping screen (part 3) 280, and the display control unit 33 displays the virtual room mapping screen (part 3) 280.

[0199] S33: User A selects an imaging device 10 to associate with the virtual room. The reception unit 32 of the communication terminal 30A receives the selection and identifies the imaging device ID.

[0200] S34: The communication unit 31 of the communication terminal 30A sends an association request to the information processing system 50, specifying the virtual room ID selected in step S28 and the imaging device ID selected in step S30.

[0201] S35: The communication unit 51 of the information processing system 50 receives a mapping request, and the communication group management unit 56 registers the imaging device 10 in the virtual room. That is, the communication group management unit 56 registers the imaging device ID in the virtual room information storage unit 5002.

[0202] S36: Since the imaging device ID has been associated with the virtual room, the communication unit 51 of the information processing system 50 transmits the virtual room ID, name, and description to the imaging device 10. The information processing system 50 can use push notifications or transmit by utilizing the polling of the imaging device 10. The connection unit 16 of the imaging device 10 receives the virtual room ID, name, and description and stores them in the storage unit 1000. As a result, when the imaging device 10 transmits a wide-field image, it can attach the imaging device ID, virtual room ID, name, description, etc.

[0203] S37: The imaging device 10 connects to the virtual room identified by the virtual room ID in the information processing system 50.

[0204] S38: Communication terminal 30A is notified that the mapping is complete. S39: User A turns on the toggle button 291 of the imaging device 10 associated with the virtual room on the image viewing screen 210. The reception unit 32 of the communication terminal 30A receives the ON signal.

[0205] S40: The communication unit 31 of the communication terminal 30A sends a request to the information processing system 50 to start transmitting wide-field images, specifying the imaging device ID. User A can also directly start transmitting wide-field images by operating a button on the imaging device 10. In addition, the communication unit 31 of the communication terminal 30A may send a request to the information processing system 50 to stop transmission based on user A's actions.

[0206] S41: The communication unit 51 of the information processing system 50 receives a transmission start request and requests the imaging device 10, identified by the imaging device ID, to start transmission. The information processing system 50 can use push notifications or the imaging device 10 can use polling. The connection unit 16 of the imaging device 10 receives the transmission start request and the imaging processing unit 13 starts imaging. The image transmission control unit 18 repeatedly transmits wide-field images via the connection unit 16 at a fixed FPS or an FPS corresponding to the bandwidth. Therefore, the communication terminal 30 that has entered the virtual room can display the status of site A on the image viewing screen 210 in real time.

[0207] <Distribution of wide-field images, etc.> Refer to Figure 22 to explain the process of sharing wide-field images and images with a normal field of view. Figure 22 is an example of a sequence diagram illustrating the process of sharing wide-field images. In Figure 22, communication terminals 30A and 30B have already entered the virtual room. Communication terminal 30A also has a camera 9 with a normal field of view, which is shared with communication terminal 30B. Instead of using camera 9 on communication terminal 30A, it is also possible to share images captured by smart glasses 88 associated with the virtual room.

[0208] S41: The imaging unit 34 of the communication terminal 30A repeatedly captures images, and the communication unit 31 specifies the virtual room ID in which the user is present and repeatedly transmits the images and audio to the information processing system 50.

[0209] S42, S43: When the communication unit 51 of the information processing system 50 receives images and audio, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A and 30B that are in the virtual room from the virtual room information storage unit 5002 and transmits the images and audio via the communication unit 51.

[0210] S44: Next, in response to a transmission start request by the toggle button 291 being turned on, the imaging device 10 repeatedly captures wide-field images, and the image transmission control unit 18 repeatedly transmits the wide-field images and audio to the information processing system 50 via the connection unit 16, specifying the virtual room ID, imaging device ID, name, and description.

[0211] S45, S46: When the communication unit 51 of the information processing system 50 receives wide-field images and audio, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A and 30B that are in the virtual room from the virtual room information storage unit 5002 and transmits the wide-field images and audio via the communication unit 51.

[0212] S47: Next, communication terminal 30C, equipped with camera 9, entered the new virtual room.

[0213] S48: The communication unit 31 of the communication terminal 30C transmits images and audio of a normal field of view to the information processing system 50.

[0214] S49~S51: The communication unit 51 of the information processing system 50 receives images and audio of the normal field of view from the communication terminal 30C, obtains the IP addresses of the communication terminals 30A~30C that are in the virtual room from the virtual room information storage unit 5002, and the image distribution unit 54 transmits images and audio of the normal field of view.

[0215] S52: In addition, the communication unit 51 of the information processing system 50 transmits wide-field images and audio to the communication terminal 30C that has entered the same virtual room.

[0216] In this way, users A and B who enter the same virtual room can share the wide-field image captured by the imaging device 10 associated with the virtual room. Note that the transmission order of each image shown in Figure 22 is just one example; the wide-field image can be shared first, or the image with a normal field of view can be shared first.

[0217] Let's add some details about the smart glasses 88 and VR goggles 89. The smart glasses 88 have a camera and display function with a normal field of view. Images with a normal field of view captured by the camera held by the smart glasses 88 are distributed in the same way as cameras 8 and 9. The display function held by the smart glasses 88 is flat, like a normal display, so a portion of the wide-field image is displayed at the viewpoint indicated by the user. The VR goggles 89 have a display function (they may also have an additional camera with a normal field of view). The display function held by the smart glasses 88 projects a wide-field image at a viewpoint determined by the orientation of the user's head, so a portion of the wide-field image is displayed at a viewpoint corresponding to the orientation of the user's head. While viewing a wide-field image with the smart glasses 88 or VR goggles 89, the user can send an imaging request specifying the viewpoint information being viewed to the information processing system 50.

[0218] Next, the viewing status memory unit 5004 will be described with reference to Figure 23.

[0219] In the following description, it is assumed that user A at site A, user B at site B, user C at site C, and user D at site D are all in the same virtual room. Site D is assumed to have the same configuration as sites B and C. Furthermore, in the following description, it is assumed that distributed image data, including wide-field image data transmitted from the imaging device 10 located at site A, and normal-angle image data transmitted from the cameras or connected imaging devices 8 and 9 located in the respective communication terminals 30 at sites B, C, and D, is distributed to the communication terminals 30 located at each site.

[0220] Figure 23 shows an example of a viewing status memory unit. Figure 23(a) shows an example of the notable location information 5004a for each location included in the viewing status information.

[0221] The Feature Location Information 5004a is stored for each virtual room. The Feature Location Information 5004a includes the following information items: virtual room ID, timestamp, location address of each location, and the Feature Location of each location, each of which is associated with another. The value of the "Virtual Room ID" item is identification information to identify the virtual room. The value of the "Timestamp" item indicates the time when the distributed image data was distributed.

[0222] The value of the item "Location Address" is information that identifies each location. In the example in Figure 23(a), the value of the item "Location Address" indicates the IP address of the communication terminal 30 installed at each location. The value of the item "Location of Interest" indicates other locations of interest at the location identified by the value of the item "Location Address".

[0223] Figure 23(b) shows an example of designated viewpoint information 5004b for each location included in the viewing status information. The designated viewpoint information 5004b includes the virtual room ID, timestamp, and designated viewpoint information for the wide-field image at each location, and each of these is associated with another.

[0224] Next, the aggregated result storage unit 5005 will be described with reference to Figure 24. Figure 24 is a diagram showing an example of the aggregated result storage unit. Figure 24(a) is a diagram showing an example of the number of points of interest information 5005a included in the aggregated result information.

[0225] The attention location count information 5005a is stored for each virtual room. The attention location count information 5005a includes the virtual room ID, timestamp, and the number of attention locations for each location, and each of these is associated with another information item. The value of the "virtual room ID" item is identification information to identify the virtual room. The value of the "timestamp" item indicates the time when the distributed image data was distributed.

[0226] The value of the item "Number of Attentions for Site A" indicates the number of sites that are paying attention to Site A among the sites that have entered the virtual room identified by the virtual room ID. In other words, the value of the item "Number of Attentions for Site A" indicates the number of communication terminals 30 that are displaying enlarged wide-field images captured at Site A, among the communication terminals 30 that have a virtual room ID and IP address associated with them. The value of the item "Number of Attentions for Site B" indicates the number of sites that are paying attention to Site B among the sites that have entered the virtual room identified by the virtual room ID. In other words, the value of the item "Number of Attentions for Site B" indicates the number of communication terminals 30 that are displaying enlarged normal-angle images captured at Site B, among the communication terminals 30 that have a virtual room ID and IP address associated with them.

[0227] The value of the item "Number of Attentions for Site C" indicates the number of sites that are paying attention to Site C among the sites that have entered the virtual room identified by the virtual room ID. In other words, the value of the item "Number of Attentions for Site C" indicates the number of communication terminals 30 that are displaying enlarged normal-angle images captured at Site C, among the communication terminals 30 that have a virtual room ID and IP address associated with them. The value of the item "Number of Attentions for Site D" indicates the number of sites that are paying attention to Site B among the sites that have entered the virtual room identified by the virtual room ID. In other words, the value of the item "Number of Attentions for Site D" indicates the number of communication terminals 30 that are displaying enlarged normal-angle images captured at Site D, among the communication terminals 30 that have a virtual room ID and IP address associated with them.

[0228] In the example in Figure 24(a), at the timestamp "0:01:00", the number of locations focusing on location B among locations A to D is 1, and the number of locations focusing on location A, location C, and location D are each 0. Also, at the timestamp "0:01:02", the number of locations focusing on location B among locations A to D is 2, and the number of locations focusing on location A, location C, and location D are each 0.

[0229] Therefore, in this case, the most important location at timestamp "0:01:00" and the most important location at timestamp "0:01:02" are both considered to be location B. The information indicating the most important location identified by the location identification unit 63 is stored in the aggregate result storage unit 5005 as part of the aggregate result information. If the information indicating the most important location is not stored in the aggregate result storage unit 5005, the processing of the location identification unit 63 can be executed when playing back the distributed image data, as described later.

[0230] Figure 24(b) shows an example of the most important viewpoint information 5005b included in the aggregated results information. The most important viewpoint information 5005b includes the following information items: virtual room ID, timestamp, imaging device information, and viewpoint information, each of which is associated with another.

[0231] In the example in Figure 24(b), it can be seen that the most important viewpoint at timestamp "0:01:00" and the most important viewpoint at timestamp "0:01:02" have a radial radius of 15[r], a polar angle of 25[θ], and an azimuth angle of 35[φ], respectively.

[0232] Next, with reference to Figure 25, the operation of the communication system 1 of this embodiment will be described. Figure 25 is a first sequence diagram showing the operation of the communication system 1. The process from step S2501 to step S2515 in Figure 25 shows the operation when user A at site A, user B at site B, user C at site C, and user D at site D enter one virtual room, and user C at site C and user D at site D focus on site B.

[0233] In the information processing system 50, the communication terminal 30B at site B is made into a virtual room in the information processing system 50, where the communication terminals 30 at sites A to D are associated with their respective communication terminals 30 (steps S2501 to S2504).

[0234] Next, the communication terminal 30A at site A begins transmitting the wide-field image data captured by the imaging device 10 located at site A to the information processing system 50 (step S2505). In other words, the information processing system 50 begins distributing the distributed image data, including the wide-field image data captured by the imaging device 10 at site A, to each communication terminal 30 that has entered the virtual room.

[0235] Next, the communication terminal 30B at site B starts transmitting normal field-of-view image data to the information processing system 50 (step S2506). In other words, the information processing system 50 starts distributing distributed image data to each communication terminal 30 that has entered the virtual room, which includes wide-field-of-view image data captured by the imaging device 10 and normal field-of-view image data received from the communication terminal 30B.

[0236] In this embodiment, when image data is transmitted from each location, the transmitted image data is stored in association with a virtual room. Therefore, wide-field image data transmitted from location A is stored in the information processing system 50 in association with location A and the virtual room ID of the virtual room. Similarly, normal-angle image data transmitted from location B is stored in the information processing system 50 in association with location B and the virtual room ID of the virtual room. By storing image data for each location in this way, even when reviewing the distributed image data, it is possible to change the viewpoint, zoom in and out, and perform other operations on the wide-field and normal-angle images captured at each location.

[0237] Next, user C of communication terminal 30C focuses on location B (step S2507). Specifically, communication terminal 30C receives an operation from user C to enlarge the normal field-of-view image captured by communication terminal 30B in the distributed image displayed on communication terminal 30C, and transmits an instruction to enlarge the normal field-of-view image captured by location B to the information processing system 50. In response to this operation, the information processing system 50 updates the focus location information 5004a in the viewing status storage unit 5004.

[0238] Furthermore, user D of communication terminal 30D focuses on location B (step S2508). Specifically, communication terminal 30D receives an operation from user D to enlarge the normal field-of-view image captured by communication terminal 30B in the distributed image displayed on communication terminal 30D, and transmits an enlargement instruction for location B to information processing system 50. In response to this operation, information processing system 50 updates the focus location information 5004a in viewing status storage unit 5004.

[0239] Next, the information processing system 50 aggregates the attention location information 5004a stored in the viewing status storage unit 5004 using the viewing status aggregation unit 62 (step S2509). Subsequently, the information processing system 50 identifies the most attention location at this point using the attention location identification unit 63 (step S2510).

[0240] Here, among the users of locations A through D, location C and user C, and user D of location D are interested in location B. Therefore, the location of interest identification unit 63 designates location B as the location of greatest interest.

[0241] Next, the information processing system 50 notifies the communication terminals 30 at locations A to D of the identified location of most interest (steps S2511 to S2514). In other words, the information processing system 50 transmits information indicating the location of most interest identified by the location of interest identification unit 63 to the communication terminals 30 located at each of locations A to D.

[0242] In the example shown in Figure 25, the most important location is notified to all communication terminals 30 that the user is in the same virtual room, but this is not limited to this. The notification of the most important location may also be sent only to communication terminals 30 of locations that are not paying attention to the most important location.

[0243] The communication terminal 30A displays information indicating the most important location (step S2515). In this embodiment, by displaying information indicating the most important location on the communication terminal 30A, user A of the communication terminal 30A can be made aware that location B, which user A of the communication terminal 30A has not been paying attention to, has been designated as the most important location.

[0244] In the example shown in Figure 25, information indicating the most important location is displayed on the communication terminal 30A, allowing users in locations other than the most important location, who are not paying attention to the most important location, to become aware of it. This embodiment is not limited to this, and information indicating the most important location may be displayed on all communication terminals 30 in which users are in the same virtual room. Furthermore, in this embodiment, information indicating the most important location may be displayed on all communication terminals 30 located in locations other than the location designated as the most important location.

[0245] Next, we will explain the case where site A, which is capturing wide-field images, is the focus of attention. The process from step S2516 to step S2522 in Figure 25 shows the operation when user C at site C and user D at site D focus on site A.

[0246] User C of the communication terminal 30C focuses on location A (step S2516). Specifically, the communication terminal 30C receives an operation from user C to enlarge the wide-field image captured by the communication terminal 30A in the distributed image displayed on the communication terminal 30C, and transmits an instruction to enlarge the wide-field image to the information processing system 50. In response to this operation, the information processing system 50 updates the focus location information 5004a in the viewing status storage unit 5004.

[0247] User D of the communication terminal 30D focuses on location A (step S2517). Specifically, the communication terminal 30D receives an operation from user D to enlarge the wide-field image captured by the communication terminal 30A in the distributed image displayed on the communication terminal 30D, and transmits an instruction to enlarge the wide-field image to the information processing system 50. In response to this operation, the information processing system 50 updates the focus location information 5004a in the viewing status storage unit 5004.

[0248] Next, the information processing system 50 aggregates the attention location information 5004a stored in the viewing status storage unit 5004 using the viewing status aggregation unit 62 (step S2518). Subsequently, the information processing system 50 identifies the most attention location at this point using the attention location identification unit 63 (step S2519). Here, a wide-field image has been captured at location A, which has been identified as the most attention location. Therefore, the information processing system 50 determines the most attention location at this point using the attention viewpoint determination unit 64 (step S2520).

[0249] In the example shown in Figure 25, the viewpoint of interest is determined when the image captured at the point of interest is a wide-field image, but this is not limited to this. The information processing system 50 can also determine the viewpoint of interest separately from the point of interest and notify the communication terminals 30 at each location, even if the image captured at the point of interest is a normal-angle image.

[0250] Next, the information processing system 50 notifies the communication terminals 30 at locations A to D of the identified most important location and the most important viewpoint (steps S2521 to S2524). In other words, the information processing system 50 transmits information indicating the most important location identified by the location identification unit 63 and viewpoint information indicating the most important viewpoint determined by the viewpoint determination unit 64 to the communication terminals 30 located at each of locations A to D.

[0251] In the example shown in Figure 25, the most important location and most important viewpoint are notified to all communication terminals 30 that the user is in the same virtual room, but this is not limited to this. Notifications of the most important location and most important viewpoint may also be sent only to communication terminals 30 of locations that are not paying attention to the most important location.

[0252] Next, the communication terminal 30B displays information indicating the most important location and the most important viewpoint (step S2525).

[0253] In this embodiment, by displaying information indicating the most important location and the most important viewpoint on the communication terminal 30B, it is possible to make the communication terminal 30B user B aware that a user at another location is paying attention to the most important viewpoint in a wide-field image captured at location A, which the user B is not paying attention to.

[0254] In the example shown in Figure 25, the communication terminal 30D displays information indicating the most important location and the most important viewpoint, allowing users in locations other than the most important location who are not paying attention to the most important location among users who are in the same virtual room to understand the most important location and the most important viewpoint. This embodiment is not limited to this, and information indicating the most important location and the most important viewpoint may be displayed to all communication terminals 30 that are in the same virtual room. In addition, in this embodiment, information indicating the most important location and the most important viewpoint may be displayed to all communication terminals 30 located in locations other than the location designated as the most important location.

[0255] The following describes examples of the display of the communication terminal 30 with reference to Figures 26 to 28. Figure 26 is the first diagram showing an example of the display of the communication terminal.

[0256] The screen 3010 shown in Figure 26 is an example of a screen displayed on the communication terminal 30 at each location when users from locations A to D enter the same virtual room. Screen 3010 is displayed on the communication terminal 30 at each location, for example, between steps S2506 and S2507 in Figure 25.

[0257] Screen 3010 includes display areas 3011, 3012, 3013, and 3014. In screen 3010, the areas of display areas 3011, 3012, 3013, and 3014 are the same.

[0258] Display area 3011 displays a wide-field image captured at site A, and display area 3012 displays a normal-angle image captured at site B. Display area 3013 displays an image from site C, and display area 3014 displays an image from site D. When normal-angle image capture begins at site C, the normal-angle image captured at site C will be displayed in display area 3013. When normal-angle image capture begins at site D, the normal-angle image captured at site D will be displayed in display area 3014.

[0259] Figure 27 is a second diagram showing an example of a display on a communication terminal. The screen 3010C shown in Figure 27 is an example of the screen when an operation is performed to focus on the normal field-of-view image of base B on the screen 3010 shown in Figure 26, which is displayed on the communication terminal 30C. The operation to focus on the normal field-of-view image of base B is, for example, a selection operation to select the display area 3012 on screen 3010. In this embodiment, this selection operation causes the display area 3012 to be enlarged or only the display area 3012 to be displayed on screen 3010.

[0260] Screen 3010C is displayed on the communication terminal 30C, for example, after step S2507 in Figure 25. Screen 3010C includes display areas 3011a, 3012a, 3013a, and 3014a. Display area 3011a displays a wide-field image captured at site A, and display area 3012a displays a normal-angle image captured at site B. Display area 3013a displays a normal-angle image if one is captured at site C. Display area 3014a displays a normal-angle image if one is captured at site C.

[0261] In screen 3010C, the area of ​​display region 3012a is the largest, while the areas of display regions 3011a, 3013a, and 3014a are smaller compared to display region 3012a. Therefore, in screen 3010C, attention is focused on site B, where a normal-angle image is displayed in display region 3012a.

[0262] In the example shown in Figure 27, the display area of ​​the image captured at the location of interest is enlarged, but this is not the only example. For instance, suppose that the screen 3010 shown in Figure 26 is displayed on the communication terminal 30C at location C, and the user selects the normal-angle image captured at location B on screen 3010. In this case, the communication terminal 30C may not enlarge the display area 3012 where the normal-angle image captured at location B is displayed, but instead highlight the display area 3012 by changing the color of the frame image, for example.

[0263] Figure 28 is a third diagram showing an example of a display on a communication terminal. Screen 3010A shown in Figure 28 is an example of a screen displayed on communication terminal 30A when site B is designated as the site of most interest. Screen 3010A is, for example, an example of a screen displayed on communication terminal 30A in step S2515 of Figure 25.

[0264] Screen 3010A includes display areas 3011b, 3012b, 3013a, and 3014a. Display area 3011b displays a wide-field image captured at site A, and the area of ​​display area 3011b is larger than that of display areas 3012b, 3013a, and 3014a. Therefore, it can be seen that site A is not focusing on site B, but rather on the wide-field image captured at its own site.

[0265] Furthermore, in the display area 3011b, an image 3015 is displayed that shows the viewpoint specified in the wide-field image at site A.

[0266] In the display area 3012b, a normal angle-of-view image captured at base B, which is the most focused base, is displayed. Here, in the present embodiment, an image 3016 pointing to the display area 3012b is displayed. The image 3016 may be an image including a message such as "This base is the focus of attention".

[0267] On the screen 3010A, as described above, an image 3015 indicating the viewpoint in the wide-field image and an image 3016 indicating the most focused base are displayed. Therefore, according to the present embodiment, user A of the communication terminal 30A can be made to grasp the viewpoint that the user is focusing on in the wide-field image. Furthermore, according to the present embodiment, the user of the communication terminal 30A can be made to understand that another user who has entered the same virtual room is focusing on an image of a base different from the base that the user themself is focusing on.

[0268] Note that in the example of FIG. 28, the image 3016 is displayed as a method of notifying the user of the communication terminal 30A of the image of the most focused base, but the present invention is not limited thereto. It is only required that the screen 3010A can show that the most focused image is a normal angle-of-view image captured at base B, and the display mode may be various.

[0269] For example, on the screen 3010A, when base B is set as the most focused base, the display area 3012b can be automatically enlarged on the screen 3010A, and the display area 3011b can be reduced to the same area as the display areas 3013a and 3014a. By doing so, even without displaying the image 3016, user A of the communication terminal 30A can be made to understand that other users are focusing on the normal angle-of-view image of base B.

[0270] FIG. 29 is a fourth diagram showing a display example of a communication terminal. A screen 3010B shown in FIG. 29 is an example of a screen displayed on the communication terminal 30B when base A is set as the most focused base. The screen 3010B is, for example, an example of a screen displayed on the communication terminal 30B in step S2525 of FIG. 25.

[0271] Screen 3010B includes display areas 3011a, 3012a, 3013a, and 3014a. In screen 3010B, display area 3012a has the largest area, while the areas of display areas 3011a, 3013a, and 3014a are smaller compared to display area 3012a. Therefore, it can be seen that screen 3010B focuses on the normal field-of-view image captured at its own location, rather than the wide-field image captured at location A.

[0272] Furthermore, images 3017 and 3018 are displayed on screen 3010B. Image 3017 includes a message indicating that site A, which captured the wide-field image displayed in display area 3011a, is the site of greatest interest. Image 3018 shows the viewpoint of greatest interest in the wide-field image captured at site A, which is the site of greatest interest.

[0273] In this embodiment, by displaying images 3017 and 3018 in this way, even if user B at location B is not focusing on the most important location, user B can be made aware of the locations and viewpoints that other users in the same virtual room are focusing on.

[0274] Next, the operation of the communication system 1 when reviewing distributed image data after distribution in each communication terminal 30 of this embodiment will be described. In the following description, reviewing distributed image data after distribution in the communication terminal 30 may be expressed as "playing back the distributed image data."

[0275] In the information processing system 50 of this embodiment, there are two methods for playing back the distributed image data at the communication terminal 30. The first method involves transmitting the stored distributed image data and viewing status information associated with the distributed image data from the information processing system 50 to the communication terminal 30, and then the communication terminal 30 generates and plays back the playback image data that reflects the viewing status of the communication terminal 30 at each location based on the distributed image data and viewing status information.

[0276] The second method involves the information processing system 50 generating playback image data that reflects the viewing status information of the communication terminals 30 at each location, based on the stored distribution image data and viewing status information, and then transmitting the generated playback image data to the communication terminals 30.

[0277] The operation of the communication system 1 when the communication terminal 30 reviews the distributed image data after distribution will be described below with reference to Figure 30. Figure 30 is a second sequence diagram showing the operation of the communication system 1.

[0278] First, we will explain the operation when the communication terminal 30 reviews the distributed image data using the first method described above. The processes from step S3001 to step S3005 in Figure 30 show the process using the first method described above (first process).

[0279] In the communication system 1, the communication terminal 30 sends a request to the information processing system 50 to play back the stored distributed image data (step S3001). This play back request includes, for example, the IP address of the communication terminal 30 and the virtual room ID in which the communication terminal 30 was located.

[0280] Upon receiving this playback request, the information processing system 50 identifies the virtual room that the communication terminal 30 was in, and acquires the distributed image data associated with the virtual room and viewing status information (step S3002). Subsequently, the information processing system 50 transmits the acquired distributed image data and viewing status information to the communication terminal 30 (step S3003).

[0281] In this case, the information processing system 50 can either send the aggregated result information stored in the aggregated result storage unit 5005 to the communication terminal 30 instead of the viewing status information, or it can send both the viewing status information and the aggregated result information to the communication terminal 30.

[0282] When the communication terminal 30 acquires the distributed image data and viewing status information from the information processing system 50, the display control unit 33 generates playback image data that reflects the viewing status in the distributed image data (step S3004). The playback image data generated here is an image in which display areas are arranged to display images of each location, as shown in Figures 26 to 29. The playback image data that reflects the viewing status in the distributed image data is image data generated from the distributed image data and the viewing status.

[0283] Furthermore, the communication terminal 30 identifies the most important location for each timestamp from the distributed image data and viewing status information, and generates playback image data that includes information indicating the most important location. Also, if the most important location is location A, which transmitted the wide-field image data, the communication terminal 30 generates playback image data that includes the most important viewpoint.

[0284] Next, the communication terminal 30 displays the generated playback image data via the display control unit 33 (step S3005). In other words, the display control unit 33 of the communication terminal 30 displays image data that reflects the viewing status of each location in the distributed image data.

[0285] In this embodiment, viewing status information indicating the viewing status of the distributed image data at each location is transmitted to the communication terminal 30 along with the distributed image data distributed to each location. Therefore, in this embodiment, the communication terminal 30 can display playback image data that reflects the viewing status of the users of the communication terminal 30 at each location at the time of distribution of the distributed image data.

[0286] Next, we will explain the operation when the distributed image data is played back on the communication terminal 30 using the second method described above. The process from step S3006 to step S3010 in Figure 30 shows the process using the second method described above (second process).

[0287] The communication terminal 30 transmits a playback request for distributed image data to the information processing system 50 (step S3006). The information processing system 50 notes that this playback request includes, for example, the IP address of the communication terminal 30 and the virtual room ID of the virtual room that the communication terminal 30 had entered.

[0288] The information processing system 50 receives this playback request, identifies the virtual room that the communication terminal 30 had entered, and acquires the distributed image data associated with the virtual room and the viewing status information of the communication terminal 30 (step S3007). Subsequently, the information processing system 50 causes the screen generation unit 52 to generate playback image data reflecting the viewing status information based on the distributed image data and the viewing status information (step S3708).

[0289] Similar to step S3004, the playback image represented by the playback image data generated herein is an image in which display areas for displaying images captured at each of the bases are arranged, as shown in FIGS. 26 to 29.

[0290] Further, the information processing system 50 identifies the most notable base for each time stamp based on the distributed image data transmitted to each base and the viewing status information for each base, and generates playback image data including information indicating the most notable base (step S3008). In addition, when the most notable base is the base A that transmitted the wide-field image data, the information processing system 50 generates playback image data including the most notable viewpoint.

[0291] Subsequently, the information processing system 50 transmits the generated playback image data to the communication terminal 30 (step S3009). When the communication terminal 30 receives the playback image data from the information processing system 50, it plays back the playback image data (step S3010). Here, the playback image data is an example of a viewing result.

[0292] In Figure 30, in response to a request to play back the distributed image data, playback image data is generated using the distributed image data and viewing status information to reflect the viewing status at each location while the distributed image data is being distributed, and the playback image data is then played back. However, this is not the only way in which this is done. In this embodiment, in response to a request to acquire viewing status from the communication terminal 30, viewing status information indicating the viewing status at each location can also be displayed on the communication terminal 30. For example, the communication terminal 30 can display information as shown in Figures 23 and 24. Here, the viewing status information is an example of a viewing result.

[0293] This allows the communication terminal 30 to understand the viewing status of the distributed image data.

[0294] In this embodiment, viewing status information indicating the viewing status at each location while the distributed image data is being distributed to the communication terminal 30 at each location is stored in association with the distributed image data. Then, in this embodiment, when the distributed image data is played back after distribution, playback image data that reflects the viewing status at each location while the distributed image data was being distributed is generated using the distributed image data and the viewing status information.

[0295] Therefore, in this embodiment, the communication terminal 30 that requested playback of the distributed image data can display playback image data that reflects the viewing status at each location while the distributed image data is being distributed. For example, when playback of the distributed image data is requested at communication terminal 30A at location A, the communication terminal 30A will display playback image data that reflects the viewing status at locations other than location A, based on the viewing status information. Similarly, when playback of the distributed image data is requested at communication terminal 30B at location B, the playback image data that reflects the viewing status at locations other than location B, based on the viewing status information, will be displayed.

[0296] Therefore, according to this embodiment, a user viewing playback image data on a communication terminal 30 at a certain location can be made aware of the most important location and viewpoint, just as during the distribution of distributed image data. In other words, according to this embodiment, when a user reviews distributed image data at a later date, they can be made aware of which locations other users were focusing on and at what time. Furthermore, according to this embodiment, if the location that was being focused on is the location from which wide-field image data was captured, the user reviewing the distributed image data can be made aware of which viewpoints in the wide-field image other users were focusing on.

[0297] In this embodiment, an imaging device 10 for capturing wide-field images is provided at base A, but this is not limited to this. The communication system 1 of this embodiment does not necessarily include an imaging device 10 for capturing wide-field images. In that case, the image captured at base A will be an image captured by a communication terminal 30A, or an imaging device that can communicate with the communication terminal 30A and captures normal-angle images, etc.

[0298] Furthermore, the communication system 1 of this embodiment may be applied to telemedicine. The following describes the case in which the communication system 1 of this embodiment is applied to telemedicine.

[0299] <Examples of communication system applications in telemedicine> Figure 31 illustrates an example of remote communication where communication system 1 is applied to telemedicine. Note that the explanation of Figure 31 will primarily focus on the differences from Figure 1. While location A in Figure 31 is an operating room, the processing flow from (1) to (6) is the same as in Figure 1.

[0300] In Figure 31, the patient is placed on an operating table 355 and undergoes surgery performed by a medical professional such as a doctor. The medical professional (corresponding to the user) uses various surgical instruments 354, such as forceps and scalpels, to operate on the patient. The medical professional can also wear smart glasses 88 and transmit images of the medical professional's surgical field to a communication network N. In addition, various cameras, such as an operating room camera 351, a surgical field camera 352, and an endoscope 353, are arranged in the operating room as imaging devices similar to the imaging device 10. All of these imaging devices have the function of capturing images to generate wide-field images. All imaging devices and smart glasses 88 in the operating room can be associated with a virtual room.

[0301] A main unit 356 is located in the operating room to monitor the patient's vital signs and the operating status of medical equipment. The main unit 356 corresponds to the communication terminal 30 in this embodiment. In addition to the functions shown in Figure 1, the communication terminal 30 (main unit 356) in the operating room can also be equipped with the function of receiving images from the endoscope 353 and the surgical field camera 352. The communication terminal 30 can display the received images, including wide-field images, on the display 306, and can also transmit them to the information processing system 50 as images from the communication terminal 30's location.

[0302] The control panel 357 is an input interface that accepts various operations, allowing medical personnel to operate equipment in the operating room via the control panel 357. Furthermore, the endoscope 353, surgical camera 351, and surgical field camera 352 can communicate directly with the information processing system 50 without going through the communication terminal 30. In this way, multiple imaging devices 10 can be associated with the same virtual room, allowing users at remote locations to request the acquisition of wide-field images capturing various scenes at location A. For example, to image the inside of a patient's body, an imaging request can be sent to the imaging device corresponding to the endoscope 353, and to image the entire operating room, an imaging request can be sent to the imaging device corresponding to the surgical camera 351.

[0303] Furthermore, the communication terminal 30 can be equipped with the functionality of an electronic medical record system. The communication terminal 30 can also be equipped with the functionality to communicate with the electronic medical record system. The communication terminal 30 can also display information from the electronic medical record along with wide-field images on the display 306. The storage 90 can also function as an electronic medical record system. In such cases, the wide-field images (and associated viewpoint information) captured in response to an imaging request are associated with the patient's electronic medical record and saved by the association processing unit 53. The folders indicated by the storage location in the storage 90 can be categorized by patient or surgery. Additionally, information indicating the patient and surgery is stored in association with the virtual room information storage unit 5002. In this way, patient and surgery-related information can always be displayed on the communication terminal's viewing screen.

[0304] Figure 32 shows an example of a virtual room mapping screen (part 1) 360 for associating an imaging device with a virtual room in the case of telemedicine. The explanation of Figure 32 mainly describes the differences from Figure 17.

[0305] In the case of telemedicine, the virtual room mapping screen (part 1) 360 displays a list of virtual rooms 361 associated with, for example, remote surgeries or consultations. Medical cameras 362, including 360-degree cameras, are associated with each virtual room. Medical cameras 362 include endoscopes, surgical field cameras used for imaging the surgical field in the operating room, and cameras for capturing microscopic images.

[0306] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0307] For example, the configuration example shown in Figure 13 is divided according to its main function in order to facilitate understanding of the processing performed by the information processing system 50, the imaging device 10, and the communication terminal 30. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the information processing system 50, the imaging device 10, and the communication terminal 30 can be further divided into more processing units depending on the processing content. Furthermore, each processing unit can be divided to include even more processing.

[0308] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0309] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0310] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figures 21, 22, 25, and 30, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0311] <Note> <1> A communication system comprising multiple communication terminals and an information processing system that distributes distributed image data, including multiple image data captured at multiple locations, to the multiple communication terminals, A viewing status recording unit stores viewing status information received from each of the multiple communication terminals in a storage unit in association with the distributed image data, in response to operations performed on the multiple image data contained in the distributed image data at each of the multiple communication terminals. A communication system comprising: a display control unit that, after the distribution image data has been distributed, receives a request from the communication terminal to play the distribution image data, and displays the viewing result based on the viewing status information on the communication terminal that made the playback request.

[0312] <2> Includes an imaging device capable of capturing wide-field images with a wide field of view, The distributed image data includes wide-field image data showing the wide-field image captured by the imaging device, <1> The communication system described.

[0313] <3> The aforementioned viewing results include playback image data that reflects the viewing status indicated by the aforementioned viewing status information, The aforementioned viewing status information is, This includes information indicating the location where an image is of interest to each of the multiple communication terminals, The aforementioned regenerated image data is This includes information indicating the most noteworthy location among the aforementioned multiple locations, which is attracting the attention of the most communication terminals. <1> or <2> The communication system described. <4> The aforementioned viewing status information is, This includes viewpoint information indicating the viewpoint in the wide-field image for each of the plurality of communication terminals, which is specified for each of the communication terminals. The aforementioned regenerated image data is Based on viewpoint information indicating the viewpoint in the wide-field image for each communication terminal, the viewpoint of most interest, which is the most noteworthy viewpoint in the wide-field image, is determined. <1> ~ <3> A communication system as described in any one of the following items. <5> Based on the aggregated results of the aforementioned viewing status information, the system has a unit for identifying the most noteworthy location among the multiple locations that is attracting the attention of the most communication terminals. <1> ~ <4> A communication system as described in any one of the following items. <6> The system includes a focus point determination unit that determines the most noteworthy focus point in the wide-field image, using view information indicating the viewpoint in the wide-field image for each of the plurality of communication terminals, which is specified for each of the plurality of communication terminals. <1> ~ <5> A communication system as described in any one of the following items. <7> An information processing system that distributes distributed image data, which includes multiple image data captured at multiple locations where multiple communication terminals are located, to the multiple communication terminals, A viewing status recording unit stores viewing status information, which indicates the viewing status received from each of the multiple communication terminals, in association with the distributed image data, in response to operations performed on the multiple image data contained in the distributed image data at each of the multiple communication terminals. An information processing system comprising: a display control unit that, after the distribution image data has been distributed, receives a request from the communication terminal to play the distribution image data, and displays the viewing result based on the viewing status information on the communication terminal that made the playback request. <8> An information processing method by an information processing system that distributes distributed image data, which includes multiple image data captured at multiple locations where multiple communication terminals are located, to the multiple communication terminals, The aforementioned information processing system In response to operations performed on the multiple image data contained in the distributed image data at each of the multiple communication terminals, viewing status information indicating the viewing status received from each of the multiple communication terminals is stored in the storage unit in association with the distributed image data. An information processing method that, after the distribution image data has been distributed, receives a request from the communication terminal to play the distribution image data, and displays the viewing results based on the viewing status information on the communication terminal that made the playback request. <9> An information processing method by an information processing system that distributes distributed image data, which includes multiple image data captured at multiple locations where multiple communication terminals are located, to the multiple communication terminals, In the aforementioned information processing system, In response to operations performed on the multiple image data contained in the distributed image data at each of the multiple communication terminals, viewing status information indicating the viewing status received from each of the multiple communication terminals is stored in the storage unit in association with the distributed image data. A program that, after the aforementioned distributed image data has been distributed, receives a request from the communication terminal to play the distributed image data, and then executes a process to display the viewing results, based on the viewing status information, on the communication terminal that made the playback request. [Explanation of symbols]

[0314] 10 Imaging device 30 Communication terminals 50 Information Processing Systems [Prior art documents] [Patent Documents]

[0315] [Patent Document 1] Japanese Patent Publication No. 2022-151665

Claims

1. A communication system comprising multiple communication terminals and an information processing system that distributes distributed image data, including multiple image data captured at multiple locations, to the multiple communication terminals, A viewing status recording unit stores viewing status information received from each of the multiple communication terminals in a storage unit in association with the distributed image data, in response to operations performed on the multiple image data contained in the distributed image data at each of the multiple communication terminals. A communication system comprising: a display control unit that, after the distribution image data has been distributed, receives a request from the communication terminal to play the distribution image data, and displays the viewing result based on the viewing status information on the communication terminal that made the playback request.

2. Includes an imaging device capable of capturing wide-field images with a wide field of view, The communication system according to claim 1, wherein the distributed image data includes wide-field image data showing the wide-field image captured by the imaging device.

3. The aforementioned viewing results include playback image data that reflects the viewing status indicated by the aforementioned viewing status information, The aforementioned viewing status information is, This includes information indicating the location where an image is of interest to each of the multiple communication terminals, The aforementioned regenerated image data is The communication system according to claim 2, which includes information indicating the most noteworthy location among the multiple locations that has attracted the most attention from the communication terminals.

4. The aforementioned viewing status information is, This includes viewpoint information indicating the viewpoint in the wide-field image for each of the plurality of communication terminals, which is specified for each of the communication terminals. The aforementioned regenerated image data is The communication system according to claim 3, which includes the most noteworthy viewpoint in the wide-field image, determined based on viewpoint information indicating the viewpoint in the wide-field image for each communication terminal.

5. The communication system according to claim 3, further comprising a unit for identifying the most noteworthy location among the multiple locations that has attracted the most attention from the communication terminals, based on the aggregated results of the viewing status information.

6. The communication system according to claim 4, further comprising a focus viewpoint determination unit that determines the most noteworthy viewpoint in the wide-field image, using viewpoint information indicating a viewpoint in the wide-field image for each of the plurality of communication terminals specified in each of the plurality of communication terminals.

7. An information processing system that distributes distributed image data, which includes multiple image data captured at multiple locations where multiple communication terminals are located, to the multiple communication terminals, A viewing status recording unit stores viewing status information, which indicates the viewing status received from each of the multiple communication terminals, in association with the distributed image data, in response to operations performed on the multiple image data contained in the distributed image data at each of the multiple communication terminals. An information processing system comprising: a display control unit that, after the distribution image data has been distributed, receives a request from the communication terminal to play the distribution image data, and displays the viewing result based on the viewing status information on the communication terminal that made the playback request.

8. An information processing method by an information processing system that distributes distributed image data, which includes multiple image data captured at multiple locations where multiple communication terminals are located, to the multiple communication terminals, The aforementioned information processing system In response to operations performed on the multiple image data contained in the distributed image data at each of the multiple communication terminals, viewing status information indicating the viewing status received from each of the multiple communication terminals is stored in the storage unit in association with the distributed image data. An information processing method that, after the distribution image data has been distributed, receives a request from the communication terminal to play the distribution image data, and displays the viewing results based on the viewing status information on the communication terminal that made the playback request.

9. An information processing method by an information processing system that distributes distributed image data, which includes multiple image data captured at multiple locations where multiple communication terminals are located, to the multiple communication terminals, In the aforementioned information processing system, In response to operations performed on the multiple image data contained in the distributed image data at each of the multiple communication terminals, viewing status information indicating the viewing status received from each of the multiple communication terminals is stored in the storage unit in association with the distributed image data. A program that, after the distribution image data has been distributed, receives a request from the communication terminal to play the distribution image data, and executes a process to display the viewing results based on the viewing status information on the communication terminal that made the playback request.

Images