Information processing systems, communication systems, methods for determining points of interest, programs

Abstract

When a communication terminal is displaying wide-field images captured by multiple imaging devices, the communication terminal determines which points of interest to display. [Solution] The present invention provides an information processing system 50 for transmitting wide-field images received from multiple imaging devices to multiple communication terminals, comprising: a communication unit 51 that receives viewpoint information of the wide-field images and identification information of the imaging device that transmitted the wide-field images from the multiple communication terminals, the viewpoint information management unit 61 that records the viewpoint information of the wide-field images in association with the identification information of the imaging devices, and a point of interest determination unit 62 that determines one or more points of interest of the imaging devices based on the viewpoint information of the wide-field images associated with the identification information of the imaging devices, wherein the communication unit transmits the points of interest determined by the point of interest determination unit and the identification information of the imaging device that displays the wide-field images at the points of interest to the communication terminals.

Description

Technical Field

[0001] The present invention relates to an information processing system, a communication system, a method for determining a point of interest, and a program.

Background Art

[0002] There is known a communication system that transmits at least images and sounds in real time from one base point to one or more other base points, and enables remote communication using images and sounds between users located at remote locations. As such an image, there is known a wide-view image having a wide imaging range captured as an imaging range including locations that cannot be fully confirmed within a normal angle of view, for example, a 360-degree image (also referred to as an all-sky image, an omnidirectional image, or a full-surround image) in which a 360-degree full circumference is captured. A user can operate a communication terminal to view a partial area of the wide-view image from an arbitrary virtual viewpoint by changing the virtual viewpoint for a partial image of the wide-view image displayed on the display screen of the communication terminal.

[0003] There is known a technique for sharing the viewpoints that other users are focusing on among users at different base points when viewing one image at multiple base points (see, for example, Patent Document 1). Patent Document 1 discloses a technique in which the results of detecting the viewpoints (points of interest) of each conversation participant are exchanged via a communication line, and specific symbols indicating the points of interest of each conversation participant are displayed on a program screen.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with conventional technology, when there are multiple imaging devices transmitting wide-field images, the communication terminal does not determine the point of focus for displaying the wide-field image. If users at each location are displaying wide-field images from different locations, there is a risk that the subject of discussion will not be clearly defined. Furthermore, since the entire wide-field image does not fit within the predetermined area image displayed by the communication terminal, it is difficult to determine whether the viewpoint of a user at another location is different, or whether they are displaying a wide-field image from a different location, making it even more difficult for users to display a predetermined area image from the same viewpoint.

[0005] This invention provides a technology for determining the points of interest to be displayed on a communication terminal when the communication terminal is displaying a wide-field image captured by multiple imaging devices. [Means for solving the problem]

[0006] In view of the above issues, the present invention provides an information processing system for transmitting wide-field images received from multiple imaging devices to multiple communication terminals, comprising: a communication unit that receives viewpoint information of the wide-field images and identification information of the imaging device that transmitted the wide-field images from the multiple communication terminals, the viewpoint information of the wide-field images received from the user; a viewpoint information management unit that records the viewpoint information of the wide-field images in association with the identification information of the imaging devices; and a point of interest determination unit that determines one or more points of interest of the imaging devices based on the viewpoint information of the wide-field images associated with the identification information of the imaging devices, wherein the communication unit transmits the points of interest determined by the point of interest determination unit and identification information of the imaging device that displays the wide-field images at the points of interest to the communication terminals. [Effects of the Invention]

[0007] This invention makes it possible to determine the points of interest to be displayed on a communication terminal when a wide-field image captured by multiple imaging devices is being displayed on the communication terminal. [Brief explanation of the drawing]

[0008] [Figure 1]This figure illustrates an example of remote communication using wide-field imaging. [Figure 2] This is an example of a schematic diagram of a communication system configuration. [Figure 3] This is an example of a hardware configuration diagram for an imaging device. [Figure 4] This is an example of a hardware configuration diagram for a communication terminal and an information processing system. [Figure 5] (a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a top view of the imaging device. [Figure 6] This is a diagram illustrating the use of the imaging device. [Figure 7] (a) is a hemispherical image (front) captured by the imaging device, (b) is a hemispherical image (back) captured by the imaging device, and (c) is an image represented using equirectangular projection. [Figure 8] (a) A conceptual diagram showing the state of covering the sphere with an equirectangular projection image, and (b) A diagram showing a full-sphere image. [Figure 9] This diagram shows the positions of a virtual camera and a predetermined region when a 360-degree image is treated as a three-dimensional sphere. [Figure 10] (a) is a stereoscopic perspective view of Figure 5, and (b) is a diagram showing the state in which an image of a predetermined area is displayed on the display. [Figure 11] This figure shows the relationship between information from a predetermined region and an image of a predetermined region T. [Figure 12] This is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. [Figure 13] This is an example of a functional configuration diagram for a communication system. [Figure 14] This is a conceptual diagram showing an example of image management information stored in the image management information storage unit. [Figure 15] This is a conceptual diagram showing an example of virtual room information stored in the virtual room information storage unit. [Figure 16] This is a conceptual diagram showing an example list of viewpoint information stored in the viewpoint information memory unit. [Figure 17](a) shows an example of an entrance screen. (b) is a diagram showing an example of an image viewing screen displayed on a communication terminal when a user enters a virtual room. [Figure 18] It is an example of a sequence diagram explaining the process of a user (or a communication terminal) entering a virtual room. [Figure 19] It is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room. [Figure 20] It is a diagram showing an example of a virtual room association screen (part 2). [Figure 21] It is a diagram showing an example of a virtual room association screen (part 3). [Figure 22] It is a diagram showing an example of a wide - view image transmission start / stop dialog displayed on a communication terminal. [Figure 23] It is an example of a sequence diagram showing the procedure for a user to register an imaging device in a virtual room. [Figure 24] It is an example of a sequence diagram explaining the flow of sharing a wide - view image. [Figure 25] It is a diagram showing an example of a setting screen for performing settings related to the control of automatically displaying a point of interest. [Figure 26] It is an example of a sequence diagram explaining the process by which an information processing system records viewpoint information. [Figure 27] It is an example of a sequence diagram explaining the process of determining a point of interest based on viewpoint information. [Figure 28] It is an example of a sequence diagram explaining the process of determining a point of interest based on a moving area. [[ID=;]] [Figure 29] It is an example of a flowchart diagram explaining the algorithm by which a point - of - interest determination unit determines popular viewpoint information as the point of interest. [[ID=;]] [Figure 30] It is an example of a flowchart diagram explaining the algorithm by which a point - of - interest determination unit determines popular viewpoint information as the point of interest. [Figure 31] It is an example of a flowchart diagram explaining the algorithm by which a point - of - interest determination unit determines an area with changes as the point of interest. [Figure 32] This diagram illustrates object detection and tracking. [Figure 33] This flowchart illustrates an example of an algorithm used by the focus point determination unit to identify areas of change as focus points. [Figure 34] This diagram illustrates the method for determining popular points of interest, along with an example of a wide-field image. [Figure 35] This figure shows an example of an image viewing screen displayed by a communication terminal during a meeting. [Figure 36] This diagram shows an example image viewing screen where points of interest are displayed. [Figure 37] This diagram illustrates an example of remote communication where a communication system is applied to telemedicine. [Figure 38] This figure shows an example of a virtual room mapping screen (part 1) used to associate imaging equipment with a virtual room in the case of telemedicine. [Modes for carrying out the invention]

[0009] Below, as an example of an embodiment for carrying out the present invention, an information processing system and a method for determining points of interest performed in the information processing system will be described.

[0010] <An example of remote communication> Figure 1 illustrates an example of remote communication using wide-field imaging. In Figure 1, three locations A to C communicate via an information processing system 50. The number of locations is just an example; there could be two locations, four or more locations, etc. Remote communication refers to communicating with someone who is physically far away, using IT tools to communicate through images and audio.

[0011] Location A is, for example, a construction site. Locations B and C can be anywhere, such as an office, as long as wide-field images can be transmitted. Location A is equipped with an imaging device 10 that can capture wide-angle images, such as a 360-degree spherical image, or wide-angle images with a horizontal range of 180 to 360 degrees. Hereafter, such wide-angle images will simply be referred to as wide-field images. Locations A to C are equipped with various communication terminals 30A to 30C that can view wide-field images. Hereafter, any of the communication terminals 30A to 30C will be referred to as "communication terminal 30".

[0012] At the construction site, various construction activities are being carried out by workers at different locations. The entire construction site is captured using wide-field imaging, and users at each location A to C can arbitrarily change their viewpoint to monitor specific construction or work they wish to focus on. The viewpoint refers to the central position or range displayed on the displays of communication terminals 30A to 30C within the entire wide-field image.

[0013] The imaging device 10 can be mounted on a tripod 86 or on an arm 85 via a gimbal 87. A relay device (in Figure 1, the communication terminal 30A also serves as the relay device) is installed at the construction site, and the communication terminal 30A transmits the wide-field images received from the imaging device 10 via wired or wireless connection to the information processing system 50. The communication terminal 30A can also serve as a terminal for viewing the wide-field images. A camera 9 is connected to the communication terminal 30A (it may also be built-in), and images of a normal field of view (or 360-degree images) captured by the camera 9 can also be transmitted to the information processing system 50. In addition, user A (for example, a worker) can wear smart glasses 88, and images of a normal field of view (or 360-degree images) captured by the smart glasses 88 may be transmitted to the information processing system 50. Smart glasses 88 are information terminals that display information acquired via the internet on a display while maintaining a field of view. Smart glasses 88 may be placed at any location.

[0014] At site B, a PC (Personal Computer) or smartphone is provided as an example of a communication terminal 30B. Furthermore, any device capable of communicating with the information processing system 50 via the communication network N may be used as a communication terminal 30B, and other devices such as tablet terminals, PDAs (Personal Digital Assistants), electronic whiteboards, or projectors may also be used. A camera may be built into or connected to the communication terminal 30B.

[0015] At base C, a PC, smartphone, VR goggles 89, etc., are placed as examples of communication terminals 30C. In Figure 1, a camera 8 is built into or connected to the communication terminal 30C. VR goggles 89 are information terminals that display an artificial world on a computer or a 360-degree image according to the direction of movement of the head or body. The camera 8 can be either wide-angle or standard-angle. Furthermore, the communication terminal 30C can be any device that can communicate with the information processing system 50 via the communication network N, and may be a display device such as a tablet terminal, PDA, electronic whiteboard, or projector. VR goggles 89 may be placed at any base.

[0016] In this embodiment, the imaging device 10 and the communication terminal 30 are managed in a communication group called a virtual room. The imaging device 10 is associated with the virtual room, and the communication terminal 30 (the user operating the communication terminal 30) can enter this virtual room to receive the wide-field image transmitted by the imaging device 10 and view the wide-field image. Smart glasses 88 and VR goggles 89 can also be associated with the virtual room. Cameras 8 and 9 enter the virtual room together with the communication terminal 30.

[0017] Users A to C at locations A to C can arbitrarily change their viewpoint on the wide-field image. Therefore, since each user A to C viewing the wide-field image in real time may be seeing a different viewpoint, there is a risk of communication difficulties. To address this, in this embodiment, for example, the viewpoint of communication terminal 30A at location A is transmitted to communication terminals 30B and 30C at locations B and C. The outline of viewpoint sharing will be explained below. For the sake of explanation, it will be assumed that the viewpoint of a user at location B is shared by users at locations A and C.

[0018] (1) Communication terminals 30A to 30C share the wide-field image captured by the imaging device 10. When user B requests the capture of a wide-field image while viewing from an arbitrary viewpoint on communication terminal 30B, communication terminal 30B transmits the viewpoint information and the capture request to the information processing system 50.

[0019] (2) In response to an imaging request, the information processing system 50 specifies viewpoint information and requests the imaging device 10 to take an image (either a still image or a video).

[0020] (3) The imaging device 10 takes an image in response to an imaging request and saves the wide-field image and viewpoint information to a URL (an example of storage destination information; in Figure 1, the imaging device 10 is stored in storage 90) notified by the information processing system 50. The wide-field image saved in storage 90 can be downloaded and displayed by any communication terminal 30.

[0021] (4) The information processing system 50 sends a URL to the communication terminal 30B.

[0022] (5) The information processing system 50 also automatically or at the request of user B sends the URL to the communication terminals 30A and 30C that are in the same virtual room.

[0023] (6) The communication terminals 30A and 30C connect to a URL to receive viewpoint information and a wide-field image, and set the viewpoint of the wide-field image identified by the viewpoint information to align with the center of the image field and display it. However, it is not necessary to align the viewpoint perfectly with the center; the viewpoint may be set to be included in the area near the center of the image field and displayed.

[0024] The same applies when users at locations B and C share the perspective of users at location A, and when users at locations A and B share the perspective of users at location C.

[0025] As described above, in this embodiment, even when wide-field images are distributed, viewpoint information is shared at each location, making it easier for users to communicate.

[0026] In (3), the imaging device 10 can transmit the wide-field image itself to the information processing system 50, and in (4), the information processing system 50 can transmit the wide-field image to the communication terminals 30A to 30C.

[0027] While Figure 1 illustrates an example where the imaging device 10 is deployed at a construction site, this embodiment can also be applied to VR education, event streaming, remote customer service, telemedicine, and the like. In VR education, the imaging device 10 is deployed at a site location such as a laboratory, allowing students to view blackboards, equipment, samples, experimental results, etc., from a remote location by arbitrarily changing their viewpoint. In event streaming, the imaging device 10 is deployed at the event venue, allowing event participants, such as spectators, to view the venue online from a remote location by arbitrarily changing their viewpoint. The venue includes images of performers, contestants, presenters, products and exhibits explained at the event, images of materials explained at the event, and images of the venue's condition. The event venue can be indoors or outdoors and includes venues for sports, concerts, plays, etc. In remote customer service, for example, when applied to customer service at a travel agency, the imaging device 10 is deployed at the travel destination, allowing customers to review their itinerary from a remote location by arbitrarily changing their viewpoint. In telemedicine, the imaging device 10 is placed in a medical setting such as an operating room, and doctors, students, medical equipment personnel, etc., can remotely change their viewpoint and view the actions of doctors and nurses performing medical procedures in the medical setting, the placement of equipment, the patient's condition, vital signs, etc.

[0028] The locations where images are captured are not limited to these; any space where users (viewers) at the viewing location have a need to understand the situation at a remote location, such as schools, factories, warehouses, construction sites, server rooms, or stores, is acceptable.

[0029] <About Terminology> A tenant is a company or organization that has contracted with a service provider (in this embodiment, an information processing system) to receive image distribution services. While a user is typically affiliated with a tenant, they may also subscribe to the service individually. In addition to users, tenants may also include imaging devices, virtual rooms, and other similar entities.

[0030] A base of operations refers to a place that serves as the foundation for activities. In this embodiment, a conference room is used as an example of a base of operations. A conference room is a room set up primarily for the purpose of holding meetings. Meetings are also called gatherings, meetings, discussions, assemblies, and other similar terms.

[0031] A device refers to an apparatus other than a general-purpose communication terminal 30 such as a PC or smartphone, and is an imaging device or a wide-field image viewing device. In this embodiment, examples include an imaging device 10, smart glasses 88, and VR goggles 89.

[0032] Viewpoint information refers to parameter information that specifies which predetermined region of a wide-field image to display on the screen. In this embodiment, the radial, polar, and azimuth angles corresponding to the center of the wide-field image displayed on the screen are described as examples of viewpoint information, but it may also be specified by other parameter information such as the coordinates of diagonal vertices.

[0033] A wide-field image is an image with a wide field of view, captured over a wide imaging range, including a 360-degree image (also called a full-sphere image, omnidirectional image, or all-around image) that captures the entire 360 ​​degrees.

[0034] Remote communication refers to communicating with someone who is physically far away, using IT tools to convey information through images and audio. Communication can take various forms, including customer service, meetings, gatherings, discussions, study groups, classes, seminars, and presentations. It does not necessarily have to be two-way communication. Therefore, a virtual room can also be called a virtual conference room.

[0035] Images with a normal field of view are not wide-field images, but in this embodiment, they will be described as non-wide-field images (planar images).

[0036] A communication group is a collection of users from whom wide-field images are shared (distributed). In a normal space, a communication group is described as a virtual room, meaning that when each user enters the same room, each user can share the wide-field image. "Virtual" means that it is realized through information processing via a network.

[0037] An image is a visual representation of an event or phenomenon captured on a medium. A video is an image displayed on a display device.

[0038] A point of interest is the viewpoint that should be focused on in a wide-field image. This viewpoint can be set using various algorithms. The viewpoint may also be referred to as the line of sight, gaze point, or eye line.

[0039] <Example of a communication system configuration> Figure 2 shows an example of a schematic diagram of the communication system 1. The communication system 1 is a system that transmits and receives wide-field images and normal-angle images captured by the imaging device 10 bidirectionally between multiple locations. By displaying images distributed from one location at other locations, users at other locations can view the images. As an example of a wide-field image, a 360-degree spherical image captured by the imaging device 10 is distributed. The communication system 1 can, for example, allow a wide-field image captured at a predetermined location to be viewed at other locations in remote locations.

[0040] As shown in Figure 2, in the communication system 1, the imaging device 10, communication terminal 30A, and information processing system 50 located at base A, and communication terminals 30B and 30C located at each of the multiple bases (bases B and C), are connected in a way that enables communication.

[0041] If the imaging device 10 can be directly connected to the communication network N, a communication terminal 30A acting as a relay device (e.g., a router) is not necessary. In this case, the communication terminal 30A is connected to the communication network N without the imaging device 10. However, if the communication terminal 30A is located at site A, the communication terminal 30A also functions as a relay device, allowing user A to view wide-field images in the same way as communication terminals 30B and 30C. Furthermore, imaging devices 10 may be located at other sites besides site A, or multiple imaging devices 10 may be located at site A.

[0042] Each communication terminal 30 and information processing system 50 can communicate via the communication network N. The communication network N is constructed using the Internet, mobile communication networks, LANs (Local Area Networks), etc. The communication network N may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0043] The imaging device 10 is a special digital camera capable of capturing a full-sphere image by capturing a subject, landscape, etc., to obtain two hemispherical images as the basis. The wide-field image obtained by the imaging device 10 may be a video, a still image, or both. Furthermore, the captured image may be a video containing sound along with the image.

[0044] The communication terminal 30 is a computer such as a PC used by users at each location. The communication terminal 30 displays images captured at its own location, wide-field images (still images or videos) distributed from other locations, and images with a normal field of view. For example, the communication terminal 30 acquires wide-field images captured by the imaging device 10 via the communication network N. The communication terminal 30 also has image processing software such as OpenGL ES installed, and can display images based on viewpoint information that identifies a portion of the wide-field image. Note that OpenGL ES is just one example of image processing software, and other software may be used. Furthermore, even if image processing software is not installed, image processing may be performed by software received from an external source, or image display may be performed by receiving the results of image processing performed by external software. In other words, the communication terminal 30 can display a predetermined portion of the wide-field image.

[0045] The communication terminal 30 can arbitrarily change the viewpoint relative to the display range of a wide-field image in response to user operation. The communication terminal 30 can change the field of view (predetermined area) based on viewpoint information corresponding to the changed viewpoint by moving the virtual viewpoint in response to user operation input (including key input, drag, scrolling, etc.) to a touch panel, directional buttons, mouse, keyboard, touchpad, etc. Furthermore, if the communication terminal 30 is a communication terminal worn by a user, such as VR goggles, the communication terminal 30 may detect the change in posture information of the communication terminal 30 in response to changes in the movements of the user wearing it, and change the field of view (predetermined area) based on viewpoint information corresponding to the changed viewpoint by moving the virtual viewpoint in response to the detected posture information.

[0046] The communication terminal 30A distributes wide-field images acquired from the imaging device 10 via a wired cable such as a USB (Universal Serial Bus) cable to communication terminals 30 at other locations via the information processing system 50. The connection between the imaging device 10 and the communication terminal 30A may be a wireless connection using short-range wireless communication or the like, rather than a wired connection using a wired cable. Multiple communication terminals 30A may be located at location A.

[0047] In some cases, a user at location A may wear smart glasses 88, and the smart glasses 88 may connect to the communication network N. In this case, the images captured by the smart glasses 88 are transmitted to the information processing system 50 via the communication network N, and the information processing system 50 can distribute them to the communication terminals 30 at each location.

[0048] Communication terminal 30B is located at site B where user B is located, and communication terminal 30C is located at site C where user C is located. Multiple communication terminals 30B and 30C may be located at sites B and C. Furthermore, communication terminals 30B and 30C may be carried by user B and user C respectively.

[0049] Furthermore, the communication terminals 30A to 30C at locations A to C may be either terminals with built-in cameras or terminals that can accommodate external cameras. Communication terminals 30A to 30C can distribute images captured by their own cameras 8, 9, etc., to other locations. In addition, any devices may be placed at locations A to C.

[0050] The arrangement of each terminal and device (communication terminal 30 and imaging device) and the user shown in Figure 2 is an example, and other examples may be used. Furthermore, the communication terminal 30 is not limited to a PC, but may be, for example, a tablet terminal, smartphone, wearable device, projector, electronic whiteboard (a whiteboard with electronic blackboard functionality that enables mutual communication), or autonomous robot. The communication terminal 30 can be any computer running a dedicated application for a web browser or image distribution service.

[0051] Furthermore, if the imaging device 10 has a display, it may be configured to display images distributed from other locations.

[0052] The information processing system 50 has one or more information processing devices. The information processing system 50 manages and controls communication between the imaging devices 10 and communication terminals 30 at each location, and manages the wide-field images that are transmitted and received. The information processing system 50 provides a platform that allows users to utilize the functions necessary to provide an image distribution service that distributes wide-field images. This platform may be made available to service providers, such as individuals or companies, who wish to provide image distribution services, by contract. Hereinafter, in order to distinguish them from tenants that receive image distribution services, service providers who provide image distribution services to users using the platform will be referred to as platform subscribers.

[0053] Therefore, the information processing system 50 may, as a platform, expose an API (Application Programming Interface), allowing platform subscribers to use this API to provide various image distribution services. Platform subscribers only need to develop software such as applications that handle screens displayed by the communication terminal 30 and API calls, and do not need to develop functions provided by APIs such as image distribution from scratch.

[0054] The information processing system 50 may be constructed using a single computer, or it may be constructed using multiple computers, each part (function or means) of which is arbitrarily assigned. Furthermore, all or part of the functions of the information processing system 50 may be server computers located in a cloud environment, or server computers located in an on-premises environment.

[0055] Storage 90 is a storage device separate from the information processing system 50. Storage 90 may be external storage (which may be storage located in the cloud or storage located on-premises) or storage included in the information processing system 50.

[0056] <Example Hardware Configuration> Next, the hardware configuration of each device or terminal in the image communication system according to this embodiment will be described using Figures 3 and 4. Note that the hardware configuration shown in Figures 3 and 4 may have components added or removed as needed.

[0057] <<Hardware configuration of the imaging device>> First, the hardware configuration of the imaging device 10 will be explained using Figure 3. Figure 3 is a diagram showing an example of the hardware configuration of the imaging device 10. In the following, the imaging device 10 will be described as a 360-degree (omnidirectional) imaging device using two image sensors, but there may be more than two image sensors. Furthermore, it is not necessarily required to be a device dedicated to omnidirectional imaging; an omnidirectional imaging unit can be attached to a regular digital camera or smartphone, etc., to provide essentially the same functionality as the imaging device 10.

[0058] As shown in Figure 3, the imaging device 10 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output interface 116, a short-range communication circuit 117, an antenna 117a for the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network interface 121.

[0059] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lens 102 unless otherwise specified) each having a field of view of 180° or more for forming a hemispherical image, and two image sensors 103a and 103b provided corresponding to each lens. The image sensors 103a and 103b have an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the lenses 102a and 102b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers for setting various commands or parameters necessary for the operation of the image sensors. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example; it may have only one, or three or more.

[0060] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, operation unit 115, input / output I / F 116, short-range communication circuit 117, electronic compass 118, gyro sensor 119, acceleration sensor 120, and network I / F 121, etc.

[0061] The image processing unit 104 receives image data output from image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create the data for the equirectangular projection image described later.

[0062] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 acting as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0063] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 10, there may also be functions to display a preview or video on a display (for example, the display of an external terminal such as a smartphone that communicates with the imaging device 10 via a short-range communication circuit 117). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0064] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0065] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress of processing by the image processing unit 104 and data of completed equirectangular projection images.

[0066] The control unit 115 is a collective term for various operation buttons, a power switch, a shutter button, and a touch panel that combines display and operation functions. The user inputs various imaging modes, imaging conditions, etc., by operating the control unit 115.

[0067] The input / output interface (I / F) 116 is a general term for interface circuits (such as USB I / F) to external media such as SD cards or personal computers. The I / F 116 can be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded to external media via the I / F 116, or transmitted to an external terminal (device) via the I / F 116 as needed.

[0068] The short-range communication circuit 117 communicates with an external terminal (device) via an antenna 117a provided on the imaging device 10 using a short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi. The short-range communication circuit 117 can transmit equirectangular projection image data to the external terminal (device).

[0069] The electronic compass 118 calculates the orientation of the imaging device 10 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) according to Exif and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data. The gyro sensor 119 is a sensor that detects changes in angle (Roll angle, Pitch angle, Yaw angle) associated with the movement of the imaging device 10. The changes in angle are an example of related information (metadata) according to Exif and are used for image processing such as image correction of captured images. Furthermore, the acceleration sensor 120 is a sensor that detects acceleration in three axes. The imaging device 10 calculates its own orientation (angle relative to the direction of gravity) based on the acceleration detected by the acceleration sensor 120. By providing the acceleration sensor 120, the accuracy of image correction of the imaging device 10 is improved.

[0070] Network I / F121 is an interface for data communication using a communication network N, such as the Internet, via a router or similar device.

[0071] <<Hardware configuration of the communication terminal>> Figure 4 shows an example of the hardware configuration of the communication terminal 30 and the information processing system 50. First, the communication terminal 30 will be described. The hardware configuration of the communication terminal 30 is indicated by codes in the 300 series. The communication terminal 30 is built by a computer and, as shown in Figure 4, is equipped with a CPU 301, ROM 302, RAM 303, HD (Hard Disk) 304, HDD controller 305, display 306, external device connection I / F 308, network I / F 309, bus line 310, keyboard 311, pointing device 312, DVD-RW (Digital Versatile Disk Rewritable) drive 314, media I / F 316, sound input / output I / F 317, microphone 318, speaker 319, short-range communication circuit 320, and camera 321.

[0072] Of these components, the CPU 301 controls the overall operation of the communication terminal 30. The ROM 302 stores programs used to drive the CPU 301, such as IPLs. The RAM 303 is used as the work area for the CPU 301. The HD 304 stores various data, such as programs. The HDD controller 305 controls the reading or writing of various data to the HD 304 according to the control of the CPU 301. The display 306 displays various information such as cursors, menus, windows, characters, or images. The display 306 is an example of a display unit. The display 306 may also be a touch panel display equipped with input means. The external device connection I / F 308 is an interface for connecting various external devices. In this case, external devices include, for example, USB memory or printers. The network I / F 309 is an interface for data communication using the communication network N. The bus line 310 is an address bus or data bus, etc., for electrically connecting each component, such as the CPU 301 shown in Figure 4.

[0073] The keyboard 311 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting or executing various instructions, selecting processing targets, or moving a cursor, etc. Note that the input means may not be limited to the keyboard 311 and the pointing device 312, but may also be a touch panel or an audio input device, etc. The DVD-RW drive 314 controls the reading or writing of various data to the DVD-RW 313, which is an example of a removable recording medium. Note that the DVD-RW 313 may be a DVD-R or a Blu-ray® Disc, etc. The media I / F 316 controls the reading or writing (storage) of data to the recording medium 315, such as flash memory. The microphone 318 is a type of built-in sound collection means for inputting sound. The sound input / output I / F 317 is a circuit that processes the input and output of sound signals between the microphone 318 and the speaker 319 according to the control of the CPU 301. The short-range communication circuit 320 is a communication circuit for communicating with an external terminal (device) using short-range wireless communication technology such as NFC, Bluetooth (registered trademark), or Wi-Fi. The camera 321 is a type of built-in imaging means that captures an image of a subject and obtains image data. Note that the microphone 318, speaker 319, and camera 321 may be external devices rather than being built into the communication terminal 30.

[0074] <<Hardware configuration of the information processing system>> As shown in Figure 4, each hardware component of the information processing system 50 is indicated by a 500-series code in parentheses. The information processing system 50 is built using a computer and has a configuration similar to that of the communication terminal 30, as shown in Figure 4; therefore, a description of each hardware component is omitted.

[0075] Furthermore, each of the above programs may be distributed as an installable or executable file recorded on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc (registered trademark), SD card, USB memory, etc. The recording media can also be provided domestically or internationally as a program product. For example, the communication terminal 30 realizes the image display method according to the present invention when the program according to the present invention is executed.

[0076] <Regarding wide-field images and viewpoint information> The following section explains how to generate wide-field images (spherical images) using Figures 5 to 12.

[0077] First, the external appearance of the imaging device 10 will be described using Figure 5. The imaging device 10 is a digital camera for obtaining captured images that will form the basis of a 360° spherical image. Figure 5(a) is a left side view of the imaging device, Figure 5(b) is a front view of the imaging device, and Figure 5(c) is a top view of the imaging device. This external view is merely one example of the imaging device 10, and other external appearances are also possible.

[0078] As shown in Figure 5(a), the imaging device 10 is small enough to be held in one hand, but this shape is merely an example, and other shapes are also possible. Also, as shown in Figures 5(a), 5(b), and 5(c), the upper part of the imaging device 10 is equipped with an image sensor 103a on the front side and an image sensor 103b on the back side. These image sensors 103a and 103b are used in conjunction with optical components (for example, lenses 102a and 102b described later) capable of capturing hemispherical images (angle of view of 180° or more). Also, as shown in Figure 5(b), an operating section 115 such as a shutter button is provided on the side of the imaging device 10 opposite to the front side. As mentioned above, there may be only one image sensor, or there may be three or more.

[0079] Next, the usage of the imaging device 10 will be explained using Figure 6. Figure 6 is an illustrative diagram of the imaging device in use. As shown in Figure 6, the imaging device 10 is used, for example, to image a subject around the imaging device 10. In this case, two hemispherical images can be obtained by imaging the subject around the imaging device 10 using the image sensors 103a and 103b shown in Figure 5, respectively.

[0080] Next, using Figures 7 and 8, we will outline the process from the image captured by the imaging device 10 to the creation of a full-sphere image. Figure 7(a) shows the hemispherical image (front side) captured by the imaging device, Figure 7(b) shows the hemispherical image (back side) captured by the imaging device, and Figure 7(c) shows the image represented by equirectangular projection (hereinafter referred to as "equistratic projection image"). Figure 8(a) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 8(b) shows the full-sphere image.

[0081] As shown in Figure 7(a), the image obtained by the image sensor 103a is a curved hemispherical image (front side) by the lens 102a described later. Similarly, as shown in Figure 7(b), the image obtained by the image sensor 103b is a curved hemispherical image (rear side) by the lens 102b described later. The imaging device 10 then combines the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) to create an equirectangular projection image EC as shown in Figure 7(c).

[0082] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to overlay an equirectangular projection image EC so that it covers the sphere, as shown in Figure 8(a), and creates a full-sphere image (full-sphere panoramic image) CE as shown in Figure 8(b). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. OpenGL ES is merely one example of software that performs image processing, and the full-sphere image CE may be created using other software. Also, the full-sphere image CE may be a still image or a video.

[0083] As described above, the 360-degree spherical image CE is an image pasted to cover a sphere, which can cause discomfort to the human eye. Therefore, the imaging device 10 can display a predetermined area T of the 360-degree spherical image CE (hereinafter referred to as the "predetermined area image") as a planar image with less curvature, thereby providing a display that does not cause discomfort to the human eye. This will be explained with reference to Figures 9 to 10.

[0084] Figure 9 shows the positions of the virtual camera and a predetermined region when the 360-degree image is treated as a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of the user viewing the 360-degree image CE, which is displayed as a three-dimensional sphere. Figure 10(a) is a stereoscopic perspective view of Figure 9, and Figure 10(b) shows the predetermined region image as it appears on a display. Figure 10(a) represents the 360-degree image CE shown in Figure 9 as a three-dimensional sphere CS. If the 360-degree image CE generated in this way is a sphere CS, then, as shown in Figure 9, the virtual camera IC is located inside the 360-degree image CE. The predetermined region T in the 360-degree image CE is the imaging area of ​​the virtual camera IC and is identified by predetermined region information indicating the imaging direction and field of view of the virtual camera IC in the three-dimensional virtual space including the 360-degree image CE. Furthermore, zooming in on the predetermined region T can also be represented by moving the virtual camera IC closer to or further away from the 360-degree image CE. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. Therefore, the predetermined region T can be determined by the field of view α and the distance f from the virtual camera IC to the 360-degree spherical image CE (see Figure 11).

[0085] Then, the predetermined region image Q shown in Figure 10(a) is displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in Figure 10(b). The image shown in Figure 10(b) is a predetermined region image represented by the initially set (default) predetermined region information. The following explanation will use the imaging direction (ea, aa) and field of view (α) of the virtual camera IC. Note that the predetermined region T may be represented by the position coordinates (X, Y, Z) of the imaging area of ​​the virtual camera IC, which is the predetermined region T, rather than by the field of view α and distance f.

[0086] Next, we will explain the relationship between the predetermined region information and the image of the predetermined region T using Figure 11. Figure 11 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 11, "ea" is the elevation angle, "aa" is the azimuth angle, and "α" is the field of view (Angle). That is, the orientation of the virtual camera IC is changed so that the point of fixation of the virtual camera IC, indicated by the imaging direction (ea, aa), becomes the center point CP(x, y) of the predetermined region T, which is the imaging area of ​​the virtual camera IC. As shown in Figure 11, the center point CP(x, y) when the diagonal field of view of the predetermined region T, represented by the field of view α of the virtual camera IC, is α becomes the parameter ((x, y)) of the predetermined region information. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between any vertex of a given region T and the center point CP(x, y) (2L is the diagonal). In Figure 11, the following trigonometric function generally holds:

[0087]

number

[0088] Figure 12 shows the relationship described in Figure 11 as a point in a three-dimensional Euclidean space using spherical coordinates. Here, the position coordinates of the center point CP shown in Figure 11, expressed in spherical polar coordinates, are (r, θ, φ). (r, θ, φ) are the radial, polar angle, and azimuth angle, respectively. The radial r is the distance from the origin of the three-dimensional virtual space containing the full-sphere image to the center point CP, and is therefore equal to the distance f shown in Figure 11. Figure 12 is a diagram illustrating these relationships. Hereafter, the position coordinates (r, θ, φ) of the virtual camera IC will be used as an example of viewpoint information for explanation. The viewpoint information only needs to be parameter information that can identify a predetermined region T (predetermined region image Q) that is displayed as an image of the imaging area of ​​the virtual camera IC on the predetermined display shown in Figure 10, as described above, and includes the coordinates of the diagonal vertices of the predetermined region T. Alternatively, the information indicating the field of view α of the virtual camera IC and the information indicating the center point CP(x, y) described in Figure 11 may also be considered as viewpoint information. Furthermore, viewpoint information includes not only position coordinate information in spherical coordinates, but also position coordinate information in Cartesian coordinates and coordinate difference values ​​from the initially set (default) predetermined region information. Also, viewpoint information may be information other than coordinate information, such as angles and distances, as shown in Figure 11. In addition, although Figures 11 and 12 use the center point of the predetermined region T as the reference, the predetermined region T may also be identified by parameter information based on any of the vertices of the predetermined region T. Note that the above explanation of viewpoint information was given using the example of a 360-degree spherical image as the wide-field image, but in the case of other wide-field images, the viewpoint information is the information that identifies the predetermined region T in that wide-field image.

[0089] <About the features> Next, the functional configuration of the communication system 1 according to this embodiment will be described using Figure 13. Figure 13 is a diagram showing an example of the functional configuration of the communication system 1 according to this embodiment. In Figure 13, the terminals, devices, and servers shown in Figure 1 that are related to the processing or operation described later are shown.

[0090] <<Functional Configuration of Imaging Device>> First, the functional configuration of the imaging device 10 will be described with reference to Figure 13. The imaging device 10 includes a communication unit 11, a reception unit 12, an imaging processing unit 13, an analysis unit 14, a registration request unit 15, a connection unit 16, a storage processing unit 17, an image transmission control unit 18, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 111 that follow a program deployed on the SRAM 113 or DRAM 114. The imaging device 10 also has a storage unit 1000 constructed from a ROM 112, etc., as shown in Figure 3.

[0091] The communication unit 11 has the function of connecting to a communication network N using wireless communication means such as Wi-Fi and sending and receiving various data or information with other devices. In this embodiment, the connection unit 16 mainly transmits the wide-field image acquired by the imaging processing unit 13 to the information processing system 50, but it is also possible for the communication unit 11 to transmit the wide-field image to the information processing system 50.

[0092] The reception unit 12 is a function that receives user input for the imaging device 10. The reception unit 12 receives input such as power on / off, shutter button on / off (start or stop transmission of wide-field images), etc.

[0093] The imaging processing unit 13 captures images of subjects, landscapes, etc., and acquires the captured images. The captured images acquired by the imaging processing unit 13 may be videos or still images (or both), and may include sound along with the images. The imaging processing unit 13 also captures images of, for example, a two-dimensional code displayed on the display 306 of the communication terminal 30.

[0094] The analysis unit 14 analyzes the two-dimensional code captured and acquired by the imaging processing unit 13 and extracts the information contained in the two-dimensional code (URL for registering the imaging device with the tenant, temporary ID and password).

[0095] The registration request unit 15 uses the information contained in the two-dimensional code read by the analysis unit 14 to send a request to the information processing system 50 to register the imaging device 10 as a tenant of the information processing system 50.

[0096] The connection section 16 is implemented, for example, by a short-range communication circuit 117, and has the function of receiving power supply from the communication terminal 30A and performing data communication.

[0097] The storage processing unit 17 processes the wide-field images captured in response to imaging requests from any location and saves them to a URL (e.g., storage 90) notified by the information processing system 50.

[0098] The image transmission control unit 18 controls the transmission of wide-field images to the information processing system 50. For example, the image transmission control unit 18 transmits captured images acquired by the imaging processing unit 13 to the information processing system 50 periodically or in response to user operation if they are still images, or at a predetermined FPS (Flame Per Second) if they are videos. The image transmission control unit 18 also switches between the communication unit 11 and the connection unit 16.

[0099] The storage / reading unit 19 has the function of storing various data in the storage unit 1000 or reading various data from the storage unit 1000. The storage unit 1000 also stores the image capture data acquired by the image processing unit 13, the image capture device ID, etc. The image capture data stored in the storage unit 1000 may be deleted after a predetermined time has elapsed since it was acquired by the image processing unit 13, or it may be deleted when data has been transmitted to the information processing system 50.

[0100] Furthermore, the imaging device 10 has an application (also called a plug-in) installed to support the communication system 1. This application is not necessary for commercially available imaging devices 10, but it is used when associating the imaging device 10 with a virtual room and when receiving control from external sources. Some of the functions shown in Figure 13 (for example, the registration request unit 15) are provided by this application.

[0101] <<Communication terminal function configuration>> Next, the functional configuration of the communication terminal 30 will be explained using Figure 13. The communication terminal 30 has a communication unit 31, a reception unit 32, a display control unit 33, an imaging unit 34, a storage / reading unit 35, and a connection unit 36. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 301 following a program (which may be a web browser or a dedicated application) deployed on the RAM 303. The communication terminal 30 also has a storage unit 3000 constructed from the ROM 302 or recording medium 315 shown in Figure 4.

[0102] The communication unit 31 is implemented, for example, by a network interface 309, and is connected to a communication network N, performing functions such as sending and receiving various data or information with other devices.

[0103] The reception unit 32 has the function of receiving various selections or operation inputs to the communication terminal 30. The display control unit 33 has the function of displaying wide-field images, normal-angle images, and various screens on the display 306 of the communication terminal 30. The display control unit 33 displays, for example, a two-dimensional code transmitted from the information processing system 50 on the display 306. The two-dimensional code may be, for example, QR code (registered trademark), DataMatrix (DataCode), MaxiCode, or PDF417. The two-dimensional code may also be a barcode.

[0104] The connection section 36 is implemented, for example, by a short-range communication circuit 320, and has the function of supplying power to the imaging device 10 and performing data communication.

[0105] The storage / reading unit 35 is executed by instructions from the CPU 301 shown in Figure 4, and has the function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. The storage unit 3000 has an image management information storage unit 3001. The image management information storage unit 3001 will be explained in the description of the information processing system 50.

[0106] <<Functional Configuration of the Information Processing System>> Next, the functional configuration of the information processing system 50 will be described. The information processing system 50 includes a communication unit 51, a screen generation unit 52, an association processing unit 53, an image distribution unit 54, an authentication unit 55, a communication group management unit 56, a communication control unit 57, a connection management unit 58, a storage / reading unit 59, an API management unit 60, a viewpoint information management unit 61, and a point of interest determination unit 62. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with a program deployed on the RAM 503. The information processing system also has a storage unit 5000 constructed from a ROM 502, HD 504, or recording medium 515 as shown in Figure 4.

[0107] The communication unit 51 has the function of sending and receiving various data or information with other devices via the communication network N.

[0108] The screen generation unit 52 generates screen information to be displayed by the communication terminal 30. When the communication terminal 30 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc. When the communication terminal 30 runs a native application, the screen information is held by the communication terminal 30, and the information to be displayed is transmitted in XML, etc. The screen generation unit 52 generates screen information on which wide-field images, etc., distributed by the image distribution unit 54 are placed.

[0109] The association processing unit 53 controls the sharing of viewpoint information for wide-field images. When the association processing unit 53 receives an imaging request along with viewpoint information from the communication terminal 30, it requests imaging from the imaging device 10 and processes the acquisition of a wide-field image and the associated viewpoint information. Furthermore, it stores the associated wide-field image and viewpoint information in the image management information storage unit 5001. The association processing unit 53 also sends a URL to the communication terminal 30 as information indicating the storage location where the associated wide-field image and viewpoint information are stored. Note that the information processing system 50 does not need to receive viewpoint information and imaging requests from the communication terminal 30 simultaneously; it may receive them separately and then process the association. Also, the URL is just one example of information indicating a storage location, and other formats such as URIs may be used.

[0110] The image distribution unit 54 distributes wide-field images transmitted by the imaging device 10, which is associated with the same virtual room, to the communication terminal 30 operated by the user who is in the virtual room. Images of the normal field of view captured by the camera of the communication terminal 30 or connected cameras 8 and 9 are also distributed in the same manner.

[0111] The authentication unit 55 is a function that authenticates the requester based on the authentication request received by the communication unit 51. For example, the authentication unit 55 authenticates the user by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, facial recognition, biometric information such as fingerprints or voiceprints, etc. The authentication unit 55 may also authenticate using an external authentication system or an authentication method such as OAuth.

[0112] The communication group management unit 56 manages the entry of communication terminals 30 and users into the virtual room, as well as the mapping of devices. When authentication by the authentication unit 55 is successful, the communication group management unit 56 registers the user ID and the IP address of the communication terminal 30 in the virtual room information storage unit 5002, and maps the imaging device 10 to the virtual room.

[0113] The communication control unit 57 manages the start, establishment, and termination of communication with the imaging device 10 associated with each virtual room. The communication control unit 57 also manages the start, establishment, and termination of communication for distributing wide-field images and audio in response to the communication terminal 30 entering or leaving the virtual room.

[0114] The connection management unit manages the communications (connections) established between the communication terminal 30 and the imaging device 10 and the information processing system 50, associating them with virtual rooms.

[0115] The API management unit 60 manages the APIs used by platform subscribers when providing wide-field image distribution services. When using APIs, platform subscribers only need to develop software to call the APIs separately. The developed software may run on a server or on a client such as a communication terminal. Any function provided by the information processing system 50, such as the image distribution unit 54, association processing unit 53, and communication control unit 57, can be provided as an API. It is also possible to provide functions added to the information processing system 50 later as APIs. Whether or not to provide an API is determined by the communication terminal operated by the platform provider accessing the information processing system 50 and accepting the API disclosure setting, allowing the API management unit 60 to control the API based on the disclosure setting. The API management unit 60 may also perform authentication processing to verify whether the requesting software requesting the API call is software developed by a legitimate platform subscriber. Authentication processing can be performed by comparing the information stored in the platform subscriber information storage unit with the information transmitted from the requesting software. As a specific example of the authentication process, the information processing system 50 receives an application ID, which has been pre-issued by the API management unit 60, from the requesting software developed by the platform subscriber. If the API management unit 60 determines that the application ID is stored in the platform subscriber information storage unit, the API management unit 60 performs a control to permit the provision of the API as legitimate software. On the other hand, if it cannot determine that the software is legitimate, the API management unit 60 performs a control to deny the provision of the API. Note that the application ID is just one example of authentication information used to determine legitimacy, and the API management unit 60 may also verify the legitimacy of the requester using authentication information such as an access token, ticket, security key, password, or PIN code that has been pre-issued by the API management unit 60 of the information processing system or an external system.In this embodiment, the form in which the functions provided by the information processing system 50 are used as an API is not described, but the processing flow is the same except that the software such as an application developed by the platform subscriber uses the functions provided by the information processing system 50 via a decision made by the API management unit 60.

[0116] The viewpoint information management unit 61 stores viewpoint information transmitted from the communication terminal 30 in the viewpoint information storage unit 5003, associating it with the user ID and the imaging device ID. The viewpoint information management unit 61 may store only one viewpoint information for one user and one imaging device 10 (storing only the latest viewpoint information), or it may store all past viewpoint information specified by one user for each imaging device 10.

[0117] The point of focus determination unit 62 determines the point of focus based on an algorithm set by the user. This point of focus is transmitted to the communication terminal 30, and the user can automatically display the point of focus on each of the multiple imaging devices 10 without having to change the viewpoint. The point of focus is the polar angle and azimuth angle, and may also include the radial movement. The user can set an algorithm according to the application of the communication system 1.

[0118] The memory / read unit 59 has the function of storing various data in the memory unit 5000 or reading various data from the memory unit 5000. The memory unit 5000 also includes an image management information storage unit 5001, a virtual room information storage unit 5002, and a viewpoint information storage unit 5003.

[0119] "Image Management Information Storage Unit 5001" Figure 14(a) is a conceptual diagram showing the image management information stored in the image management information storage unit 5001. The image management information storage unit 5001 stores the image management information as shown in Figure 14. The image management information is information for managing wide-field images captured in response to an imaging request, and when a user sends an imaging request from the communication terminal 30, one record of image management information is generated. Each item of the image management information will be explained below.

[0120] • The data ID is identification information used to identify a wide-field image. The data ID is assigned by the information processing system 50. ID is an abbreviation of Identification and means identifier or identification information. An ID is a name, code, string, number, or a combination of one or more of these used to uniquely distinguish a particular object from multiple objects.

[0121] The data name is the name of the wide-field image set by the user of the communication terminal 30. The data name can be set by the user, but it may also be set automatically.

[0122] The imaging date and time information is information used to identify the imaging date and time of the wide-field image, such as the date and time when the user entered an imaging request into the communication terminal 30, and the date and time when the imaging device 10 captured the wide-field image. The imaging date and time information may be replaced with the timestamp of the wide-field image.

[0123] The imager information is the identification information (which may be the username) of the user who entered the imaging request into the communication terminal 30. Since the user enters the imaging request into the communication terminal 30 while inside the virtual room, the user registered in the imager information is identified by authentication to the information processing system 50 or the virtual room. The imager information is transmitted to the information processing system 50 along with the imaging request.

[0124] The imaging device information is the identification information (imaging device ID) of the imaging device 10 that captured the wide-field image. The imaging device ID is assigned by the information processing system 50 and shared with the imaging device 10, but information unique to the imaging device 10, such as the MAC address or serial number, may also be used. The imaging device ID is transmitted to the information processing system 50 along with the wide-field image.

[0125] The imager's predetermined area information is viewpoint information consisting of radial radius, polar angle, and azimuth angle. The viewpoint information indicates the center coordinate of the wide-field image displayed by the communication terminal 30. The viewpoint information is transmitted from the communication terminal 30 requesting the image capture. In addition to radial radius, polar angle, and azimuth angle, the viewpoint information may also include information specifying the width and height of the display range. Alternatively, the viewpoint information may consist only of the width and height of the display range.

[0126] The virtual room ID used during imaging is the identification information of the virtual room to which the imaging device 10 is associated.

[0127] • The data storage location information includes the URL or file path where the wide-field images are stored. The wide-field images may also be videos. In the case of videos, radial motion, polar angle, and azimuth angles, such as the imager's designated area information, are stored in correspondence with the elapsed time of the meeting (video recording time).

[0128] Figure 14(b) is also a conceptual diagram showing image management information. In Figure 14(b), wide-field images with the same virtual room ID at the time of acquisition are stored. Image management information may be classified by virtual room.

[0129] "Virtual Room Information Storage Unit 5002" Figure 15 is a conceptual diagram showing the virtual room information stored in the virtual room information storage unit 5002. The virtual room information storage unit 5002 stores virtual room information as shown in Figure 15. Virtual room information is information about a virtual room and is maintained for each virtual room. The items contained in the virtual room information will be explained below. Note that virtual rooms are registered with tenants.

[0130] The virtual room ID is identification information that identifies the virtual room. In this embodiment, virtual rooms can be created at the user's discretion.

[0131] The virtual room name is a name used by users to identify the virtual room, and can be set by the user at will.

[0132] The imaging device information is the identification information (imaging device ID) of the imaging device 10 associated with the virtual room. The user currently in the room is the user ID of the user currently in the virtual room. This user is capable of viewing wide-field images. The method of entering the room will be described later. In addition, the user ID is associated with the IP address of the communication terminal 30 operated by that user.

[0133] "Perspective Information Storage Unit 5003" Figure 16 is a conceptual diagram showing the list of viewpoint information stored in the viewpoint information storage unit 5003. The viewpoint information storage unit 5003 stores a list of viewpoint information as shown in Figure 16. • The viewing device ID is the user ID of the user who manually changed the viewpoint of the wide-field image. The imaging device ID is the identification information of the imaging device 10 that captured the wide-field image, which is a wide-field image taken by the user with a changed viewpoint. The radial radius, polar angle, and azimuth angle are predetermined area information when the user displays a wide-field image on the communication terminal 30. The polar angle and azimuth angle, or the polar angle, azimuth angle, and radial radius, constitute viewpoint information. The save time is the time when the user changed their viewpoint (the time when the information processing system 50 received the viewpoint information). The weight indicates how much importance is given to viewpoint information when determining the point of focus. The viewpoint information in Figure 16 is recorded when the user changes their viewpoint, so there may be a time lag between the timing of determining the point of focus and the recording of viewpoint information. Since the wide-field image delivered to the communication terminal 30 is a video, the subjects captured in the predetermined area of ​​the image may change moment by moment. For this reason, the viewpoint information used to determine the point of focus is also given higher importance the shorter the time elapsed since the viewpoint information was recorded. The viewpoint information management unit 61 sets a larger weight the shorter the difference between the current time and the recording time when determining the point of focus. For example, the viewpoint information management unit 61 determines the weight as follows, depending on the difference between the current time and the recording time. Weight 5 if less than 1 minute Weight 4 if within 1-3 minutes Weight 3 if within 3-5 minutes Weight 2 if within 5-10 minutes If the value is 10 or more, the weight is 1 or 0. In Figure 16, only one viewpoint information is recorded for each combination of user and imaging device 10, but viewpoint information may be recorded additionally each time the user changes their viewpoint. Even in this case, the older viewpoint information has less weight and therefore has less influence on determining the point of interest, so there is no problem. On the other hand, if the information processing system 50 records a wide-field image as a video, it becomes possible to determine the point of interest at the time of playback when the video is played back.

[0134] <Entering a virtual room using a communication terminal> Next, referring to Figures 17 and 18, the process of user B entering the virtual room will be explained. It is assumed that the imaging device 10 has already been associated with the virtual room, and that the communication terminal 30A has transmitted wide-field images and normal-angle images to the information processing system 50 (the association of the imaging device 10 with the virtual room will be explained in Figure 18 and later). Furthermore, in the following explanation, there will be no particular distinction made between user B entering the virtual room and the communication terminal 30B operated by user B entering the virtual room.

[0135] Figure 17 shows an example of the screen displayed by the communication terminal 30B when user B enters a virtual room. Figure 17(a) is an example of the entry screen 200. To clarify, prior to the display of the entry screen 200, user B is logged into the information processing system 50. By logging in, the tenant to which user B belongs is identified. Virtual rooms are associated with tenants. User B displays a list of virtual rooms associated with the tenant on the communication terminal 30B (see Figure 19) and selects a virtual room to enter from the list. Figure 17(a) is the entry screen 200 for the virtual room selected by user B in this way.

[0136] Alternatively, the creator of the virtual room may request the information processing system 50 to issue a URL corresponding to the virtual room and send this URL to user B via email or other means. When user B clicks the URL displayed on the communication terminal 30B, the communication terminal 30B displays the room entry screen 200 shown in Figure 17(a).

[0137] The entry screen 200 has a virtual room name 201, a participant name input field 202, and an entry button 203. The virtual room name 201 is the same as the one stored in the virtual room information storage unit 5002. The participant name input field 202 may be a nickname or other designation for user B. Since user B's username is identified when user B logs in, this username may be displayed automatically. The entry button 203 is a button that user B presses to request entry into the virtual room.

[0138] Furthermore, authentication for entering the virtual room may be required separately from the tenant login process.

[0139] Figure 17(b) shows the image viewing screen 210 displayed by the communication terminal 30B when user B enters the virtual room. In the image viewing screen 210 of Figure 17(b), the imaging device 10 has already started distributing wide-field images, and the communication terminal 30A has already started distributing images with a normal field of view. Therefore, the image viewing screen 210 has a first image section 211 and a second image section 212. The first image section 211 displays the wide-field image, and the second image section 212 displays the image with a normal field of view. If there are three or more locations transmitting images, the image viewing screen 210 is divided according to the number of transmitting locations.

[0140] The first image field 211 displays a wide-field image mark 213. The screen generation unit 52 of the information processing system 50 sets the wide-field image mark 213 when it determines that the image to be displayed in the first image field 211 is a wide-field image. The communication terminal 30B may also determine and display it. User B can see the wide-field image mark 213 and understand that the viewpoint can be changed. The first image field 211 also displays the device name 214 (transmitted from the imaging device 10 along with the wide-field image). The device name 214 was set by user A.

[0141] The second image field 212 displays the participant name 215. The participant name 215 is the participant name of a user who has already entered the room (in this case, user A has already entered, so it is "AAA" that user A entered in the participant name input field 202).

[0142] Figure 18 is a sequence diagram illustrating the process by which user B (or communication terminal 30B) enters a virtual room.

[0143] S1: First, user B at site B performs an operation to display the virtual room list screen. When the reception unit 32 receives the operation to display the list screen, the display control unit 33 of the communication terminal 30B displays the selection screen on the display 306.

[0144] S2: When user B selects a virtual room selection button, the reception unit 32 of the communication terminal 30B accepts the selection of a virtual room. The display control unit 33 of the communication terminal 30B displays the room entry screen 200 on the display 306.

[0145] S3: User B enters the required information and presses the enter button 203. Upon receiving the press from the reception unit 32, the communication unit 31 of the communication terminal 30B sends an entry request to the information processing system 50. This entry request includes the virtual room ID indicating the virtual room selected in step S2, the user ID of the logged-in user B, and the IP address of the requesting terminal, the communication terminal 30B.

[0146] S4: As a result, the communication unit 51 of the information processing system 50 receives the entry request. The communication group management unit 56 registers the logged-in user ID and IP address in the virtual room information identified by the virtual room ID in the virtual room information storage unit 5002.

[0147] S5: Then, the communication unit 51 of the information processing system 50 transmits a message to the communication terminal 30B indicating that the room has been entered. As a result, the communication unit 31 of the communication terminal 30B receives the message that the room has been entered.

[0148] <Mapping of imaging equipment to rooms> Next, the mapping of the imaging device 10 to a room will be explained with reference to Figures 19 to 23. Note that this explanation assumes that user A at site A performs the mapping of the imaging device 10 to a room, but this can also be done by a system administrator, tenant administrator, etc.

[0149] Figure 19 shows an example of a virtual room mapping screen (part 1) 260 for associating the imaging device 10 with a virtual room. The screen configuration can be the same for VR goggles 89 and smart glasses 88. The virtual room mapping screen (part 1) 260 has a virtual room list 261. The virtual room list 261 displays individual virtual room fields 262 to 264 based on the virtual rooms created in the tenant. Each individual virtual room field 262 to 264 has a link issuance button 265, an entry button 266, a settings button 267, and a virtual room name 268. The link issuance button 265 is a button for issuing a link to the virtual room (URL for invitation) and a passcode. The entry button 266 is a button for user A to enter the virtual room. The settings button 267 is a button for associating the imaging device 10 with the virtual room. The virtual room name 268 is the same as that stored in the virtual room information storage unit 5002. Therefore, user A presses the settings button 267. Pressing the settings button 267 causes the communication terminal 30A to display the virtual room mapping screen (part 2) 270.

[0150] Additionally, if a device is already associated with a virtual room, the device name 269 will be displayed in the individual virtual room section (individual virtual room section 264 in the diagram).

[0151] Figure 20 shows an example of the virtual room mapping screen (part 2) 270. Note that the virtual room mapping screen (part 2) 270 is displayed as a pop-up on the virtual room mapping screen (part 1) 260. The screen transition from the virtual room mapping screen (part 1) 260 to the virtual room mapping screen (part 2) 270 does not go through the information processing system 50, but a screen transition that does go through the information processing system 50 is also possible.

[0152] The virtual room mapping screen (part 2) 270 has the name 271 of the imaging device 10 currently (already) mapped to the virtual room (not yet registered, so not shown in the diagram), a connect button 272, and a storage button 273. The connect button 272 is a button that displays a list of devices registered to the tenant. The storage button 273 is a button that displays a list of storage 90 where the imaging device 10 mapped to the virtual room saves wide-field images. When the connect button 272 is pressed, the communication terminal 30A displays the virtual room mapping screen (part 3).

[0153] The communication terminal 30A transmits the virtual room ID to the information processing system 50 and obtains the name of the device registered in the tenant where the virtual room is created (including the device ID, etc.), and the name of the device associated with the virtual room (including the device ID, etc.).

[0154] Figure 21 shows an example of the virtual room mapping screen (part 3) 280. The virtual room mapping screen (part 3) 280 has the name 281 of the imaging device 10 currently (already) mapped to the virtual room, a list of addable devices 282, and a save button 283. User A selects the device they want to map to the virtual room from the list of addable devices 282 and presses the save button 283. This maps the device to the virtual room (the imaging device ID is registered in the virtual room information storage unit 5002).

[0155] <Processing to initiate transmission of wide-field images to the imaging device> With the above steps, devices such as the imaging device 10 are now associated with the virtual room, but user A needs to initiate image transmission to the device.

[0156] For the VR goggles 89 and smart glasses 88, user A turns image transmission on and off by operating the device itself. This is because currently, there is no dedicated application running on the communication system 1 for the VR goggles 89 and smart glasses 88. If a dedicated application were running on the communication system 1 for the VR goggles 89 and smart glasses 88, user A would be able to remotely turn image transmission on and off.

[0157] In the case of the imaging device 10, if the application is enabled, user A can enter the virtual room and turn the transmission of wide-field images on or off from the menu.

[0158] Figure 22 shows an example of a wide-field image transmission start / stop dialog 290 displayed by the communication terminal 30A. The wide-field image transmission start / stop dialog 290 is displayed as a pop-up on the image viewing screen 210. Assume that user A has entered a virtual room associated with the imaging device 10 by operating the communication terminal 30A. The wide-field image transmission start / stop dialog 290 displays the name 292 of the imaging device 10 associated with this virtual room. A toggle button 291 is displayed near the name 292, and user A can operate the toggle button 291 to turn on or off the transmission of wide-field images by the imaging device 10. Note that the method of setting on or off using the toggle button is just one example, and it is sufficient if it can be set according to user input. For example, the user may set on or off the transmission of wide-field images by selecting radio buttons or predetermined icons, or by operating a menu. Alternatively, the transmission of wide-field images may start automatically after the imaging device 10 enters the room, without requiring user operation. Alternatively, certain conditions such as the date and time, the number of users who entered the room, or whether a specific user participated may be predetermined, and the transmission of wide-field images may begin when it is determined that these conditions have been met.

[0159] The communication terminal 30A transmits the state of the toggle button 291 to the information processing system 50. The information processing system 50 transmits a transmission start request or a transmission stop request to the imaging device 10 according to the state of the toggle button 291.

[0160] Figure 22(a) shows the toggle button 291 in the off state. Therefore, the wide-field image is not displayed in Figure 22(a). On the other hand, in Figure 22(a), when the communication terminal 30A enters the room, the image of the normal field of view captured by the camera 9 of the communication terminal 30A has already been shared and is displayed on the image viewing screen 210.

[0161] Figure 22(b) shows the toggle button 291 in the ON state. When the toggle button 291 is turned ON, the information processing system 50 sends an ON request to the imaging device 10, causing the imaging device 10 to start transmitting the wide-field image. As a result, two images are shared in one virtual room, and the image viewing screen 210 is divided into two.

[0162] <<Procedure for registering the imaging device in the virtual room>> Next, referring to Figure 23, the procedure for registering the imaging device 10 to the virtual room, as described in the screen transition section, will be explained. Figure 23 is an example of a sequence diagram showing the procedure for user A to register the imaging device 10 to the virtual room.

[0163] S11: First, user A connects the communication terminal 30A to the information processing system 50, enters authentication information (user ID, password, etc.), and requests to log in to the tenant to which they belong. The reception unit 32 of the communication terminal 30A accepts the operation.

[0164] S12: The communication unit 31 of the communication terminal 30A sends a login request to the information processing system 50, specifying the authentication information. The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 55 performs authentication. Here, we assume that authentication was successful.

[0165] S13: In response to user operation, the screen generation unit 52 of the information processing system 50 generates a device registration screen 220, and the communication unit 51 transmits the screen information of the device registration screen 220 to the communication terminal 30A.

[0166] S14: The communication unit 31 of the communication terminal 30A receives screen information from the device registration screen 220, and the display control unit 33 displays the device registration screen 220. User A selects the type of device (in this case, the imaging device 10 is selected), and then enters the name and description of the imaging device 10. The reception unit 32 accepts the input.

[0167] S15: The communication unit 31 of the communication terminal 30A sends a request for a two-dimensional code to the information processing system 50, specifying the name and description entered by user A.

[0168] S16: The communication unit 51 of the information processing system 50 receives a request for a two-dimensional code. The communication group management unit 56 generates a URL (connection destination for registration) associated with the name and description, and generates a two-dimensional code containing the URL, temporary ID and password. The communication unit 51 of the information processing system 50 transmits the two-dimensional code to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the two-dimensional code, and the display control unit 33 displays the two-dimensional code.

[0169] S17: Next, user A operates the imaging device 10 that they want to associate with the virtual room to capture a two-dimensional code. The reception unit 12 of the imaging device 10 receives the operation.

[0170] S18: The imaging processing unit 13 of the imaging device 10 generates image data by performing imaging processing including a two-dimensional code, and the analysis unit 14 analyzes the image data to extract a URL, a temporary ID, and a password. As a result, the registration request unit 15 connects to the URL via the connection unit 16, specifies the temporary ID and password, and sends a registration request from the imaging device 10 to the information processing system 50.

[0171] S19: The communication unit 51 of the information processing system 50 receives a temporary ID and password, and the authentication unit 55 determines whether they match the temporary ID and password associated with the connected URL. Here, we assume they match.

[0172] S20: The communication group management unit 56 of the information processing system 50 receives a request to register the imaging device 10, so it generates an imaging device ID and registers it in the tenant when user A logs in. The imaging device ID is associated with a name and description.

[0173] S21: The communication unit 51 of the information processing system 50 transmits the imaging device ID to the imaging device 10. The connection unit 16 of the imaging device 10 receives the imaging device ID and stores it in the storage unit 1000.

[0174] S22: The communication terminal 30A is notified that registration is complete, and user A begins associating the imaging device 10 with a virtual room. User A displays the virtual room association screen (part 1) 260 on the communication terminal 30A and selects the virtual room to which the imaging device 10 registered with the tenant should be associated. The reception unit 32 of the communication terminal 30A accepts the selection.

[0175] S23: Next, user A displays the virtual room mapping screen (part 2) 270 on the communication terminal 30A and presses Add Device. The reception unit 32 of the communication terminal 30A receives the press.

[0176] S24: The communication unit 31 of the communication terminal 30A requests the information processing system 50 for devices registered with the tenant and devices associated with the virtual room ID selected in step S23.

[0177] S25: The communication unit 51 of the information processing system 50 receives requests from devices registered with the tenant and devices associated with the virtual room ID, and the screen generation unit 52 generates a virtual room mapping screen (part 3) 280 that includes the device IDs of the devices registered with the tenant and devices associated with the virtual room ID. The communication unit 51 of the information processing system 50 transmits the screen information of the virtual room mapping screen (part 3) 280 to the communication terminal 30A.

[0178] S26: The communication unit 31 of the communication terminal 30A receives screen information from the virtual room mapping screen (part 3) 280, and the display control unit 33 displays the virtual room mapping screen (part 3) 280. User A selects the imaging device 10 to associate with the virtual room. The reception unit 32 of the communication terminal 30A accepts the selection, and the imaging device ID is identified.

[0179] S27: The communication unit 31 of the communication terminal 30A sends an association request to the information processing system 50, specifying the virtual room ID selected in step S23 and the imaging device ID selected in S27.

[0180] S28: The communication unit 51 of the information processing system 50 receives a mapping request, and the communication group management unit 56 registers the imaging device 10 in the virtual room. That is, the communication group management unit 56 registers the imaging device ID in the virtual room information storage unit 5002.

[0181] S29: Since the imaging device ID has been associated with the virtual room, the communication unit 51 of the information processing system 50 transmits the virtual room ID, name, and description to the imaging device 10. The information processing system 50 may use push notifications or transmit the information by utilizing polling by the imaging device 10. The connection unit 16 of the imaging device 10 receives the virtual room ID, name, and description and stores them in the storage unit 1000. As a result, when the imaging device 10 transmits a wide-field image, it can attach the imaging device ID, virtual room ID, name, description, etc.

[0182] S30: Communication terminal 30A is notified that the mapping is complete, and user A turns on the toggle button 291 of the imaging device 10 associated with the virtual room on the image viewing screen 210. The reception unit 32 of communication terminal 30A receives the ON signal.

[0183] S31: The communication unit 31 of the communication terminal 30A sends a request to the information processing system 50 to start transmitting wide-field images, specifying the imaging device ID. Alternatively, user A may directly start transmitting wide-field images by operating a button on the imaging device 10. In addition, user A may also cause the communication unit 31 of the communication terminal 30A to send a request to stop transmission to the information processing system 50.

[0184] S32: The communication unit 51 of the information processing system 50 receives a transmission start request and requests the imaging device 10, identified by the imaging device ID, to start transmission. The information processing system 50 may use push notifications or the imaging device 10 may use polling. The connection unit 16 of the imaging device 10 receives the transmission start request and the imaging processing unit 13 starts imaging. The image transmission control unit 18 repeatedly transmits wide-field images via the connection unit 16 at a fixed FPS or an FPS corresponding to the bandwidth. Therefore, the communication terminal 30 that has entered the virtual room can display the status of site A on the image viewing screen 210 in real time.

[0185] <Distribution of wide-field images, etc.> Referring to Figure 24, the process of sharing wide-field images and normal-angle images will be explained. Figure 24 is an example of a sequence diagram illustrating the process of sharing wide-field images. In Figure 24, communication terminals 30A and 30B have already entered the virtual room. Communication terminal 30A has a normal-angle camera 9, which is shared with communication terminal 30B. Instead of the camera 9 of communication terminal 30A, images captured by smart glasses 88 associated with the virtual room may also be shared.

[0186] S41-S43: The user enters a virtual room to view wide-field images, similar to steps S1-S5 in Figure 18.

[0187] S44: The imaging unit 34 of the communication terminal 30A repeatedly captures images, and the communication unit 31 specifies the virtual room ID in which the user is present and repeatedly transmits the images and audio to the information processing system 50.

[0188] S45, S46: When the communication unit 51 of the information processing system 50 receives images and audio, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A and 30B that are in the virtual room from the virtual room information storage unit 5002 and transmits the images and audio via the communication unit 51. In Figure 24, the communication unit 31 of the communication terminal 30A receives and displays an image with a normal field of view from the information processing system 50, but it may also display an image with a normal field of view captured by the imaging unit 34 without receiving it.

[0189] S47: Next, in response to the request to start transmission by the toggle button 291 being turned on by the imaging device 10, the imaging processing unit 13 repeatedly captures wide-field images, and the image transmission control unit 18 repeatedly transmits the wide-field images and audio to the information processing system 50 via the connection unit 16, specifying the virtual room ID, imaging device ID, name, and description.

[0190] S48, S49: When the communication unit 51 of the information processing system 50 receives a wide-field image and audio, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A and 30B that are in the virtual room from the virtual room information storage unit 5002 and transmits the wide-field image and audio via the communication unit 51. Preferably, the communication unit 51 transmits the imaging device ID and name so that it is clear which location the wide-field image is from.

[0191] S50: Next, communication terminal 30C, equipped with camera 9, entered the new virtual room.

[0192] S51: The communication unit 31 of the communication terminal 30C transmits images and audio of a normal field of view to the information processing system 50.

[0193] S52-S54: The communication unit 51 of the information processing system 50 receives images and audio of the normal field of view from the communication terminal 30C, obtains the IP addresses of the communication terminals 30A-30C that are in the virtual room from the virtual room information storage unit 5002, and the image distribution unit 54 transmits images and audio of the normal field of view.

[0194] S55: In addition, the communication unit 51 of the information processing system 50 transmits wide-field images and audio to the communication terminal 30C that has entered the same virtual room.

[0195] In this way, users A and B who enter the same virtual room can share the wide-field image captured by the imaging device 10 associated with the virtual room. Note that the transmission order of each image shown in Figure 24 is just an example; the wide-field image may be shared first, or the image with a normal field of view may be shared first.

[0196] Let's add some details about the smart glasses 88 and VR goggles 89. The smart glasses 88 have a camera and display function with a normal field of view. Images with a normal field of view captured by the camera held by the smart glasses 88 are distributed in the same way as cameras 8 and 9. The display function held by the smart glasses 88 is flat, like a normal display, so a portion of the wide-field image is displayed at the viewpoint indicated by the user. The VR goggles 89 have a display function (they may also have a camera with a normal field of view). The display function held by the smart glasses 88 projects a wide-field image at a viewpoint determined by the orientation of the user's head, so a portion of the wide-field image is displayed at a viewpoint corresponding to the orientation of the user's head. While viewing a wide-field image with the smart glasses 88 or VR goggles 89, the user can send an imaging request specifying the viewpoint information being viewed to the information processing system 50.

[0197] <Settings regarding points of interest> Figure 25 shows a settings screen 600 for configuring the control to automatically display points of interest. The settings screen 600 has an algorithm selection field 601 and an automatic switching selection field 602. In the algorithm selection field 601, the user can select the algorithm that determines the points of interest. In Figure 25, "Display areas with changes" 603, used for monitoring purposes, and "Display popular areas" 604, used for video streaming, are displayed as options.

[0198] In the automatic switching selection field 602, the user can choose whether or not to switch to the point of interest at regular intervals. If "Yes" 605 is selected, the point of interest will switch at regular intervals without user instruction, reducing the amount of work required. If "No" 606 is selected, the point of interest will not switch until instructed by the user, allowing the user to view from their own perspective and switch to the point of interest automatically when necessary.

[0199] In the setup method shown in Figure 25, the automatic switching selection field 602 is common regardless of the algorithm. However, the user could choose whether or not to switch to the point of interest at regular intervals for each algorithm. For example, it would be possible to use the system in a way that allows for automatic switching to the point of interest at regular intervals for monitoring purposes, but not for streaming purposes.

[0200] Furthermore, if "Yes" 605 is selected in the automatic switching selection field 602, the user may set a certain period of time.

[0201] Furthermore, the settings on the settings screen 600 may be set for the virtual room or for each individual user. If set for the virtual room, the same settings will be applied to all users who log in to the virtual room. If set for each individual user, the information processing system 50 will store the settings in association with the user ID and determine the point of focus using an algorithm set for each user, based on whether the user is set to switch to a point of focus at regular intervals.

[0202] The algorithm can be chosen based on the intended use of the communication system 1. For example, for applications where movement is desired, such as surveillance, an algorithm that focuses on areas of movement is used. For applications such as video streaming, where there are many viewers but each viewer should have the same perspective, an algorithm that focuses on popular viewpoints is used.

[0203] <Highlights> Next, referring to Figure 26 and other figures, we will explain the process by which each communication terminal 30 displays the points of interest determined by the information processing system 50. Figure 26 is a sequence diagram illustrating the process by which the information processing system 50 records viewpoint information.

[0204] S61-S65: As explained in Figure 24, communication terminals 30A-30C receive wide-field images and normal-angle images via the information processing system 50. Also, in Figure 26, three imaging devices 10A, 10B, and 10C are associated with a virtual room, and communication terminals 30A-30C can display wide-field images A-C captured by imaging devices 10A-10C.

[0205] S66: During a meeting (while participating in a virtual room), user A can specify a viewpoint for wide-field images A to C captured by imaging devices 10A to 10C at any time. User A rotates wide-field images A to C to change the predetermined area image displayed by the communication terminal 30A. As a result, the reception unit 32 receives the viewpoint, and the display control unit 33 displays the predetermined area image corresponding to the viewpoint on the display.

[0206] S67: When the reception unit 32 receives a change in viewpoint, the communication unit 31 of the communication terminal 30A transmits viewpoint information to the information processing system 50. The viewpoint information includes the user ID of user A operating the communication terminal 30A, and the imaging device ID of the imaging device 10 that captured the wide-field image with the changed viewpoint.

[0207] S68, S69: Similarly, user B can specify the viewpoint of the wide-field images A to C captured by the imaging devices 10A to 10C. Steps S66 to S69 may be repeated.

[0208] S70a: The viewpoint information management unit 61 stores viewpoint information, the imaging device ID of the communication terminal 30, the user ID of the user who changed the viewpoint, and the save time in the viewpoint information storage unit 5003. The viewpoint information management unit 61 may also periodically calculate the difference between the current time and the save time, and assign a larger weight the shorter the difference. Alternatively, the viewpoint information management unit 61 may calculate the difference between the current time and the save time when determining the point of interest, and assign a larger weight the shorter the difference.

[0209] S70b: When saving only one viewpoint information for each combination of user and imaging device 10, the viewpoint information management unit 61 updates (overwrites) the viewpoint information associated with the user ID of the user who changed the viewpoint and the imaging device ID of the communication terminal 30 each time the user changes the viewpoint. If, even if the combination of user and imaging device 10 is the same, viewpoint information is added and saved each time the user changes the viewpoint, the viewpoint information management unit 61 should simply add the viewpoint information associated with the user ID and imaging device ID.

[0210] <Determining the points of focus> Figure 27 is a sequence diagram illustrating the process of determining a point of interest based on viewpoint information. The process in Figure 27 is performed in parallel with that in Figure 26. That is, during live streaming of a wide-field image, the user can display the wide-field image with a point of interest as their viewpoint. A point of interest is determined when one or more communication terminals 30 are displaying wide-field images captured by multiple imaging devices 10. In the explanation of Figure 27, it is assumed that each user has an algorithm selection field 601 and an automatic switching selection field 602 set. Also, the user has selected "Show popular areas" 604 in the algorithm selection field 601.

[0211] If user A sets the automatic switching selection field 602 to "Yes", steps S71 to S75 are executed. If the automatic switching selection field 602 is set to "No", steps S76 to S82 are executed.

[0212] S71: The point of focus determination unit 62 obtains a list of viewpoint information for all communication terminals 30 from the viewpoint information storage unit 5003 at regular intervals.

[0213] S72: The point of interest determination unit 62 determines the point of interest using an algorithm corresponding to the "display popular areas" 604. Details will be described later.

[0214] S73: The communication unit 51 of the information processing system 50 notifies the communication terminal 30A of the imaging device ID of the imaging device 10 that is capturing the wide-field image with the point of interest determined. This is so that the communication terminal 30A can inform the user of the wide-field image with the point of interest that has been automatically switched. The communication unit 31 of the communication terminal 30A receives this notification and the display control unit 33 highlights the wide-field image identified by the imaging device ID (see Figure 36). Since the point of interest is determined for each imaging device 10, the communication unit 51 notifies the communication terminal 30A of the imaging device ID for each of the imaging devices 10A to 10C.

[0215] S74: The communication unit 51 of the information processing system 50 also transmits the point of interest and a request to change the viewpoint to the point of interest to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the point of interest and the viewpoint change request, and the display control unit 33 aligns the point of interest with the center of the wide-view image field, which will be described later.

[0216] S75: The display control unit 33 changes the predetermined area image displayed in the wide-field image section based on the radius movement.

[0217] Thus, even when multiple imaging devices 10 are distributing wide-field images during a meeting, the communication system 1 of this embodiment can automatically display the points of interest determined by the point of interest determination unit 62 without requiring the user to have any trouble.

[0218] Next, we will explain the case where the automatic switching selection field 602 is set to "No".

[0219] S76: User A performs an operation on the communication terminal 30A to automatically display points of interest. The reception unit 32 of the communication terminal 30A accepts the operation.

[0220] S77: The communication unit 31 of the communication terminal 30A transmits a request for points of interest to the information processing system 50.

[0221] S78-S82: The subsequent processing can be the same as S71-S75.

[0222] Thus, even when multiple imaging devices 10 are distributing wide-field images in a conference, the communication system 1 of this embodiment can automatically display popular points of interest determined by the point of interest determination unit 62.

[0223] Figure 28 is a sequence diagram illustrating the process of determining points of interest based on areas of movement. The explanation of Figure 28 primarily focuses on explaining the differences from Figure 27. In the explanation of Figure 28, the user has selected "Show areas of change" 603 in the algorithm selection field 601.

[0224] S85a, S85b: The imaging devices 10A to 10C repeatedly transmit wide-field images A to C to the information processing system 50.

[0225] S86: When the communication unit 51 of the information processing system 50 receives wide-field images A to C, the "display areas with changes" 603 is selected, so at regular intervals, the point of interest determination unit 62 analyzes wide-field images A to C and determines points of interest. Details will be described later.

[0226] Steps S87 to S89 below can be the same as in Figure 27.

[0227] S90: If the automatic switching selection field 602 is set to "No", user A performs an operation on the communication terminal 30A to automatically display the points of interest. The reception unit 32 of the communication terminal 30A accepts the operation.

[0228] S91: The communication unit 31 of the communication terminal 30A transmits a request for points of interest to the information processing system 50.

[0229] S92-S96: The subsequent processing can be the same as S85-S89.

[0230] Thus, even when multiple imaging devices 10 are distributing wide-field images in a conference, the communication system 1 of this embodiment can automatically display the points of interest determined by the points of interest determination unit 62 when the user performs an operation to display the points of interest.

[0231] <An algorithm for determining popular viewpoint information as a point of focus> Next, the algorithm for determining the point of interest will be explained with reference to Figure 29, etc. Figure 29 is a flowchart illustrating the algorithm by which the point of interest determination unit 62 determines popular viewpoint information as the point of interest. Note that Figure 29 illustrates the case where the communication terminal 30A has a dedicated wide-field image field for each of the three imaging devices 10A to 10C. Therefore, one point of interest is determined for each imaging device 10.

[0232] First, the point of interest determination unit 62 classifies the viewpoint information from the acquired list of viewpoint information according to the imaging device ID (S101). The following processing is performed for each imaging device ID.

[0233] Next, the point of interest determination unit 62 groups viewpoint information that can be considered to have the same viewpoint (S102). Viewpoint information that can be considered to have the same viewpoint is viewpoint information in which the difference between the polar angle and the azimuth angle is within a threshold. As a method for grouping viewpoint information that can be considered to have the same viewpoint, the point of interest determination unit 62 uses the polar angle and azimuth angle as vector elements and groups the viewpoint information into K groups using a clustering method such as the k-means method. Each group contains one or more viewpoint pieces of information.

[0234] The point of interest determination unit 62 counts the number of viewpoint information within a group and identifies the group with the highest number (S103). When counting, the point of interest determination unit 62 assigns the weight associated with the viewpoint information to the viewpoint information and counts the number. For example, if there are two viewpoint information in a group and the weights of the viewpoint information are 1 and 2, the count will be "3" by adding 1 and 2. If there is one viewpoint information in a group and the weight of this viewpoint information is 4, the count will be "4" instead of "1". In this case, the group with only one viewpoint information is determined to be the highest. By doing this, the point of interest determination unit 62 can more easily reflect viewpoint information stored at a time close to the current time as a point of interest.

[0235] The point of interest determination unit 62 determines the polar angle and azimuth angle of the point of interest by taking into account the average of the polar angle and azimuth angle of the viewpoint information included in the group with the highest count, which has the highest weight count (S104).

[0236] The point of interest determination unit 62 determines the radius to be applied to the point of interest based on the radius of the group with the highest weighted count (S105). The point of interest determination unit 62 may determine the largest radius, the average radius, the smallest radius, etc., from among all the radius of interest that the viewpoint information of the highest-ranking group has, to be applied to the point of interest.

[0237] The point of interest determination unit 62 determines whether a point of interest has been determined for all imaging device IDs (S106).

[0238] If the decision in step S106 is Yes, the process in Figure 29 ends; if it is No, the process proceeds to step S102.

[0239] In this way, users can automatically view wide-angle images of each location from popular viewpoints.

[0240] In Figure 29, a point of interest was determined for each imaging device 10, but it is possible that multiple points of interest may arise within the same imaging device 10. Therefore, the point of interest determination unit 62 may determine points of interest not for each imaging device 10, but from popular viewpoint information obtained from wide-field images of multiple imaging devices 10.

[0241] Figure 30 is a flowchart illustrating the algorithm by which the point of interest determination unit 62 determines popular viewpoint information as a point of interest. Note that in Figure 30, even if the communication terminal 30A has multiple wide-field image fields, it is not fixed which wide-field image from the imaging device 10 is displayed in each wide-field image field.

[0242] First, the point of interest determination unit 62 classifies the viewpoint information from the acquired list of viewpoint information according to the imaging device ID (S201).

[0243] Next, the point of interest determination unit 62 groups viewpoint information that has the same imaging device ID and can be considered to have the same viewpoint (S202). The grouping method can be the same as in Figure 29. However, in the flow in Figure 30, processing is not performed for each imaging device 10, so if there are three imaging devices 10, 3 × K groups will be created. If K is set to 1, a point of interest can be obtained for each imaging device 10, similar to the processing in Figure 29.

[0244] The point of interest determination unit 62 counts the number of viewpoint information within a group, taking weights into consideration, and identifies the top N groups (S203).

[0245] The point of interest determination unit 62 determines the polar angle and azimuth angle of the point of interest for each of the N groups by taking the average of the polar angle and azimuth angle of the viewpoint information included in the group (S204).

[0246] The point of interest determination unit 62 determines the radius to be applied to the point of interest for each of the N groups based on the radius of the group (S205). The point of interest determination unit 62 may determine the largest radius, the average radius, the smallest radius, etc., from among all the radius of interest that the viewpoint information of the group has, to be applied to the point of interest.

[0247] This processing allows the communication terminal 30 to display each of the multiple points of interest in the wide-field image from the same imaging device 10.

[0248] <An algorithm for determining areas of focus where changes are occurring> Next, we will explain the algorithm for determining the area of ​​change as the point of focus. The area of ​​change is the area occupied by a person or object when that person or object has moved from its previous location.

[0249] Figure 31 is a flowchart illustrating the algorithm by which the point of interest determination unit 62 determines a region with change as a point of interest. In Figure 31, the case where the communication terminal 30A has dedicated wide-field image fields for the three imaging devices 10A to 10C is explained. Therefore, one point of interest is determined for each imaging device 10.

[0250] First, the point of focus determination unit 62 determines an arbitrary imaging device 10 to focus on and acquires the current wide-field image distributed by this imaging device 10 (S301).

[0251] Next, the focus determination unit 62 detects objects from the wide-field image (S302). The objects are people or things that are to be of interest in the location where the imaging device 10 is placed (see Figure 32). Objects are pre-defined, for example, people, animals, machines, devices, buildings, doors, etc. Existing methods such as YOLO (Region CNN), Fast R-CNN (Region CNN), and SSD (Single Shot MultiBox Detector) may be used as object detection methods.

[0252] The point of interest determination unit 62 uses wide-field images captured by the imaging device 10 at different imaging times to track the same object as the detected object in the wide-field images (S303). Existing methods such as Kalman filters or optical flow may be used for object tracking.

[0253] For example, the Kalman filter tracks the position of an object by repeating the following two steps. (1) Prediction step (2) Update Steps The point of focus determination unit 62 detects the position of an object for each wide-field image (frame) captured at different imaging times, then uses a Kalman filter to predict and update the object's position and velocity. This allows for smooth tracking of the object's movement.

[0254] (1) Prediction step In the prediction step, the position of the object in the next frame is predicted from the information of the previous frame. The focus point determination unit 62 predicts the position where the object will exist in the next frame based on the previous state (position, speed, etc.) and the change in the state (acceleration, moving distance, etc.). For this prediction, a motion model of the object (for example, the premise that the object is moving at a constant speed and the time difference between frames) is used. Since there is uncertainty in the predicted position, the uncertainty of the predicted position is also calculated using the error covariance matrix.

[0255] (2) Update step In the update step, the prediction is corrected based on the actual observation data (object detection result) to obtain a more accurate position of the object. The focus point determination unit 62 acquires the position of the object detected in the current frame and compares it with the position predicted in the previous frame. Based on the deviation (residual) between the prediction and the observation, the position of the object is corrected. This correction amount is called the Kalman gain. The Kalman gain is a weight that determines which of the observed value and the predicted value to trust. When the Kalman gain is large, the observed value is emphasized, and when it is small, the predicted value is emphasized. As a result of the correction, the focus point determination unit 62 acquires the updated position and speed (state vector) in the next frame. Also, the error covariance matrix is updated, and the uncertainty in the next prediction step is reset.

[0256] By repeating prediction and update in each frame, the Kalman filter can continuously track the position of the object in the video. Even with noisy observed values (for example, the detected position fluctuates slightly), the position of the object can be smoothly tracked.

[0257] The point of interest determination unit 62 determines the center or centroid of the object with the largest amount of movement per unit time as the point of interest (S304). The unit time may be determined based on the assumed movement speed of the object. For example, it could be around 1 second to 1 minute. The point of interest determination unit 62 may also use the area of ​​the object in addition to the amount of movement, and determine the object with the largest product of the object's area and amount of movement as the point of interest. Furthermore, the point of interest determination unit 62 determines the radius of movement so that the entire object is displayed in the wide-field image area.

[0258] The point of interest determination unit 62 determines whether a point of interest has been determined for all imaging devices 10 (S305).

[0259] If the decision in step S305 is Yes, the process in Figure 31 ends; if it is No, the process proceeds to step S301.

[0260] Furthermore, since object detection may violate privacy and security regulations, it is preferable that object detection can be turned off by the user.

[0261] Furthermore, the above method for determining points of interest is just one example; the point of interest determination unit 62 may also compare wide-field images taken at different times on a pixel or pixel block basis to identify areas where changes have occurred.

[0262] In this way, the information processing system 50 can determine objects with significant movement as points of interest. By setting a fixed time interval, for example, to 1 to several seconds, the communication terminal 30 can be used to constantly monitor the moving object (always displaying the object in the center of the predetermined area image).

[0263] Figure 32 illustrates object detection and tracking. Wide-field images 623 and 624 are images captured by the same imaging device 10, but at different times. Wide-field images 623 and 624 are, for example, frames at time t and t+1. The frames for detecting movement are not limited to consecutive frames; movement may be detected in n (>2) frames later. In wide-field images 623 and 624, people 620 and 621 are detected as objects. The point of interest determination unit 62 can calculate the movement amount 622 of the people by tracking person 620 detected in wide-field image 623 with wide-field image 624 to detect person 621. The dotted rectangle represents the area of ​​the object.

[0264] In Figure 31, a point of interest was determined for each imaging device 10, but it is possible that multiple points of interest may arise within the same imaging device 10. Therefore, instead of determining a point of interest for each imaging device 10, viewpoint information with significant changes may be selected from wide-field images of multiple imaging devices 10.

[0265] Figure 33 is a flowchart illustrating the algorithm by which the point of interest determination unit 62 determines a region with change as a point of interest. Note that in Figure 33, even if the communication terminal 30A has multiple wide-field image fields, it is not fixed which wide-field image from the imaging device 10 is displayed in each wide-field image field.

[0266] First, the point of focus determination unit 62 acquires the current wide-field image from all imaging devices 10 that are distributing wide-field images in the virtual room during the meeting (S401).

[0267] Next, the point of focus determination unit 62 detects objects from each wide-field image (S402). The object detection method can be the same as in Figure 31.

[0268] The point of focus determination unit 62 tracks objects detected from the wide-field image for each imaging device 10 (S403).

[0269] The point of interest determination unit 62 determines the top N imaging devices 10 that are imaging an object with a large amount of movement per unit time. The point of interest determination unit 62 also determines the center or centroid of the object in the wide-field image captured by this imaging device 10 as the point of interest (S404). The point of interest determination unit 62 also determines the radius of movement so that the entire object is displayed in the wide-field image area.

[0270] This processing allows the communication terminal 30 to display each of the multiple points of interest in the wide-field image from a single imaging device 10.

[0271] <Examples of determining points of focus> Figure 34 illustrates the method for determining popular points of interest, using a wide-field image as an example. Imaging devices 10A to 10C, associated with a virtual room, transmit wide-field images. In Figure 34, it is assumed that points of interest are determined for each imaging device 10. Three users are viewing the wide-field image, and the viewpoint information manually displayed by each user is as shown in Figure 34. Figure 34(a) shows the predetermined region images displayed by each user in the wide-field image from imaging device 10A. Figure 34(b) shows the predetermined region images displayed by each user in the wide-field image from imaging device 10B. Figure 34(c) shows the predetermined region images displayed by each user in the wide-field image from imaging device 10C. For simplicity, it is assumed that users A to C changed their viewpoint information at approximately the same time (they have the same weight).

[0272] In the wide-field image of the imaging device 10A, user A displays a predetermined region image 631, user B displays a predetermined region image 632, and user C displays a predetermined region image 633. In this case, the point of interest determination unit 62 groups the viewpoint information of the predetermined region images 631 and 632 into group A, and the viewpoint information of the predetermined region image 633 into group B. Therefore, the point of interest is determined based on the viewpoint information of the predetermined region images 631 and 632, which have the largest number of viewpoint information points in their group.

[0273] In the wide-field image of the imaging device 10B, user A displays a predetermined region image 634, user B displays a predetermined region image 635, and user C displays a predetermined region image 636. In this case, the point of interest determination unit 62 groups the viewpoint information of the predetermined region images 634 and 636 into group C, and the viewpoint information of the predetermined region image 635 into group D. Therefore, the point of interest is determined based on the viewpoint information of the predetermined region images 634 and 636, which have the largest number of viewpoint information points in their group.

[0274] In the wide-field image of the imaging device 10C, user A displays a predetermined region image 637, user B displays a predetermined region image 638, and user C displays a predetermined region image 639. In this case, the point of interest determination unit 62 groups the viewpoint information of the predetermined region images 637 and 639 into group E (although the wide-field images themselves are different, the viewpoints are almost the same), and groups the viewpoint information of the predetermined region image 638 into group F. Therefore, the point of interest is determined based on the viewpoint information of the predetermined region images 637 and 639, which have the largest number of viewpoint information entries in their respective groups.

[0275] When multiple points of interest are determined from a single imaging device 10, the point of interest determination unit 62 extracts N points from groups A to F that have a large number of viewpoint information points. In the example in Figure 34, there are three groups with a maximum of two viewpoint information points, so if N=3, the number of points of interest determined will be the same as when one point of interest is determined for each imaging device 10.

[0276] FIG. 35 is an example of an image viewing screen 650 displayed by the communication terminal 30A during a meeting. The image viewing screen 650 has three wide-view image columns 651 to 653 and three face image display columns 654 to 656. Wide-view images transmitted by the imaging devices 10A to 10C are displayed in the wide-view image columns 651 to 653. Note that since the imaging device ID and name are displayed in the wide-view image columns 651 to 653, the user can determine which imaging device 10 captured the wide-view image. The face image display columns 654 to 656 display the faces of the users operating the communication terminals 30A to 30C. Note that document images may be displayed in the face image display columns 654 to 656. The user of the communication terminal 30A can change the viewpoint by dragging or swiping the wide-view image columns 651 to 653.

[0277] FIG. 36 is an example of the image viewing screen 650 in which the attention points are displayed. When the automatic switching selection column 602 is set to "Yes", a predetermined area image that displays the wide-view image at the attention point is automatically displayed in the wide-view image columns 651 to 653 at regular intervals. Also, when the automatic switching selection column 602 is set to "No", the user presses the attention point display buttons 657 to 659, and a predetermined area image that displays the wide-view image at the attention point is automatically displayed. Since the attention point display buttons 657 to 659 are provided for each of the wide-view image columns 651 to 653, the user can select only the imaging device 10 for which the attention point is to be displayed. Note that the display control unit 33 may display only one of the attention point display buttons 657 to 659, and pressing the attention point display button may display the attention points in all of the three wide-view image columns 651 to 653.

[0278] In FIG. 36, for example, the attention point of the wide-view image column 651 is determined based on the viewpoint information of the predetermined area images 631 and 632 in FIG. 34. The attention point of the wide-view image column 652 is determined based on the viewpoint information of the predetermined area images 634 and 636. The attention point of the wide-view image column 653 is determined based on the viewpoint information of the predetermined area images 637 and 639.

[0279] When wide-field image sections 651-653 display images of a predetermined region based on a point of interest, these sections are highlighted by frames 661-663. Frames 661-663 indicate that the predetermined region images of the wide-field images displayed in wide-field image sections 651-653 are displayed based on a point of interest. The highlighting can also be achieved through frame blinking, graying out, brightness reduction, or textual explanations. When the user of the communication terminal 30A changes their viewpoint by dragging or swiping wide-field image sections 651-653, frames 661-663 are also removed. Therefore, the user can understand that a point of interest is automatically displayed in wide-field image sections 651-653.

[0280] In this way, even if multiple imaging devices 10 are distributing wide-field images in a conference, the user can view the predetermined area of ​​interest automatically displayed by the point of interest determination unit 62 without any difficulty.

[0281] <Examples of communication system applications in telemedicine> Figure 37 illustrates an example of remote communication where communication system 1 is applied to telemedicine. The explanation of Figure 37 primarily focuses on the differences from Figure 1. While base A in Figure 37 is an operating room, the processing flow from (1) to (6) is the same as in Figure 1. In Figure 37, the patient is placed on an operating table 355 and undergoes surgery performed by a medical professional such as a doctor. The medical professional (corresponding to the user) uses various surgical instruments 354, such as forceps and scalpels, to operate on the patient. The medical professional can also wear smart glasses 88, which can transmit images of the surgical field to the communication network N. Various cameras, including an operating room camera 351, a surgical field camera 352, and an endoscope 353, are also installed in the operating room. All cameras and smart glasses 88 in the operating room are associated with a virtual room.

[0282] A main unit 356 is located in the operating room to monitor the patient's vital signs and the operating status of medical equipment. The main unit 356 corresponds to the communication terminal 30 in this embodiment. In addition to the functions shown in Figure 1, the communication terminal 30 (main unit 356) in the operating room may also have the function of receiving images from the endoscope 353 and the surgical field camera 352. The communication terminal 30 can display the received images on the display 306 and transmit them to the information processing system 50 as images from the communication terminal 30's location. The operation panel 357 is an input interface that accepts various operations, and may allow medical personnel to operate the equipment in the operating room via the operation panel 357. Alternatively, the endoscope 353, surgical field camera 351, and surgical field camera 352 may communicate directly with the information processing system 50 without going through the communication terminal 30.

[0283] Furthermore, the communication terminal 30 may have the functionality of an electronic medical record system, or it may have the functionality to communicate with an electronic medical record system. The communication terminal 30 may display electronic medical record information on the display 306.

[0284] Figure 38 shows an example of a virtual room mapping screen (part 1) 360 for associating an imaging device with a virtual room in the case of telemedicine. The explanation of Figure 38 mainly describes the differences from Figure 19.

[0285] In the case of telemedicine, the virtual room mapping screen (part 1) 360 displays a list of virtual rooms 361 associated with, for example, remote surgeries or consultations. Medical cameras 362, including 360-degree cameras, are associated with each virtual room. Medical cameras 362 include endoscopes, surgical field cameras used for imaging the surgical field in the operating room, and cameras for capturing microscopic images.

[0286] <Main effects> As described above, even when multiple imaging devices 10 are distributing wide-field images in a conference, the communication system 1 of this embodiment can automatically display the points of interest determined by the point of interest determination unit 62 without requiring the user to have any trouble.

[0287] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0288] For example, the configuration example shown in Figure 13 is divided according to its main function in order to facilitate understanding of the processing performed by the information processing system 50, the imaging device 10, and the communication terminal 30. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the information processing system 50, the imaging device 10, and the communication terminal 30 can be further divided into more processing units depending on the processing content. Furthermore, each processing unit can be divided to include even more processing.

[0289] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0290] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0291] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figures 23, 24, 33, and 34, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0292] <Nature> [Aspect 1] An information processing system that transmits wide-field images received from multiple imaging devices to multiple communication terminals, A communication unit that receives viewpoint information of the wide-field image received from the user by the communication terminal, and identification information of the imaging device that transmitted the wide-field image from the multiple communication terminals, A viewpoint information management unit records viewpoint information of the wide-field image in association with identification information of the imaging device, The system includes a point of interest determination unit that determines one or more points of interest of the imaging device based on viewpoint information of the wide-field image associated with the identification information of the imaging device, The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. Information processing system. [Aspect 2] The point of interest determination unit groups viewpoint information associated with the identification information of the imaging device, and determines a point of interest for each imaging device based on one or more of the grouped viewpoint information. The information processing system described in Embodiment 1. [Aspect 3] The aforementioned point of interest determination unit counts the number of viewpoint information within a group for each imaging device and identifies the highest-ranking group. Based on the one or more viewpoint pieces included in the top-level group, the point of interest is determined. The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. The information processing system described in Embodiment 2. [Aspect 4] The aforementioned point of interest determination unit counts the number of viewpoint information within a group and identifies the top N groups. For each of the N groups, a point of interest is determined based on one or more viewpoint pieces of information included in the group. The communication unit transmits to the communication terminal, for each imaging device, the point of interest determined by the point of interest determination unit, and identification information of the imaging device that displays a wide-field image at the point of interest. The information processing system described in Embodiment 2. [Aspect 5] The viewpoint information management unit records the time at which the communication unit received the viewpoint information of the wide-field image and the identification information of the imaging device. The point of interest determination unit assigns a weight to the viewpoint information of the wide-field image, which increases as the difference between the current time and the aforementioned time decreases, and then counts the number of viewpoint information within the group. The information processing system according to embodiment 3 or 4. [Aspect 6] The aforementioned point of interest determination unit detects predetermined objects from each wide-field image transmitted by the plurality of imaging devices, In wide-field images captured by the same imaging device at different times, the same object detected is tracked. For each imaging device, a point of interest is determined that includes the object with the largest amount of movement per unit time. The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. An information processing system as described in any one of the descriptions in 1 to 5. [Aspect 7] The aforementioned point of interest determination unit detects predetermined objects from each wide-field image transmitted by the plurality of imaging devices, In wide-field images captured by the same imaging device at different times, the same object detected is tracked. Determine the top N objects with the largest amount of movement per unit time, and determine the point of interest that includes the determined objects. The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. An information processing system as described in any one of the descriptions in 1 to 5. [Aspect 8] The system can accept settings to determine whether the aforementioned points of interest are determined at regular intervals or when a request to determine the aforementioned points of interest is received. If a setting is accepted to determine the aforementioned points of interest at regular intervals, The aforementioned point of interest determination unit determines the point of interest at regular intervals, The communication unit transmits to the communication terminal, at regular intervals, the determined point of interest and identification information of the imaging device that displays a wide-field image at the point of interest for each of the multiple imaging devices. An information processing system as described in any one of the descriptions 1 to 7. [Aspect 9] When a setting is received to determine the aforementioned points of interest, The point of interest determination unit determines the point of interest for the imaging device requested by the communication terminal from among the plurality of imaging devices. The communication unit transmits to the communication terminal that sent the request to determine the point of interest the identification information of the imaging device requested by the communication terminal and the determined point of interest. The information processing system described in Embodiment 8. [Aspect 10] The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. The communication terminal displays that a predetermined region image of the wide-field image, identified by the identification information of the imaging device, is being displayed based on the point of interest. An information processing system as described in any one of the descriptions 1 to 9. [Explanation of symbols]

[0293] 1. Communication System 10 Imaging device 30 Communication terminals 50 Information Processing Systems [Prior art documents] [Patent Documents]

[0294] [Patent Document 1] Japanese Patent Publication No. 2007-300452

Claims

1. An information processing system that transmits wide-field images received from multiple imaging devices to multiple communication terminals, A communication unit that receives viewpoint information of the wide-field image received from the user by the communication terminal, and identification information of the imaging device that transmitted the wide-field image from the multiple communication terminals, A viewpoint information management unit records viewpoint information of the wide-field image in association with identification information of the imaging device, It includes a point of interest determination unit that determines one or more points of interest of the imaging device based on viewpoint information of the wide-field image associated with the identification information of the imaging device, The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. Information processing system.

2. The point of interest determination unit groups viewpoint information associated with the identification information of the imaging device, and determines a point of interest for each imaging device based on one or more of the grouped viewpoint information. The information processing system according to claim 1.

3. The aforementioned point of interest determination unit counts the number of viewpoint information within a group for each imaging device and identifies the highest-ranking group. Based on the one or more viewpoint information included in the top-level group, the point of interest is determined. The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. The information processing system according to claim 2.

4. The aforementioned point of interest determination unit counts the number of viewpoint information within a group and identifies the top N groups. For each of the N groups, a point of interest is determined based on one or more viewpoint pieces of information included in the group. The communication unit transmits to the communication terminal, for each imaging device, the point of interest determined by the point of interest determination unit, and identification information of the imaging device that displays a wide-field image at the point of interest. The information processing system according to claim 2.

5. The viewpoint information management unit records the time at which the communication unit received the viewpoint information of the wide-field image and the identification information of the imaging device. The point of interest determination unit assigns a weight to the viewpoint information of the wide-field image, which increases as the difference between the current time and the aforementioned time decreases, and then counts the number of viewpoint information within the group. The information processing system according to claim 3 or 4.

6. The aforementioned point of interest determination unit detects predetermined objects from each wide-field image transmitted by the plurality of imaging devices, In wide-field images captured by the same imaging device at different times, the same object detected is tracked. For each imaging device, a point of interest is determined that includes the object with the largest amount of movement per unit time. The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. The information processing system according to claim 1.

7. The aforementioned point of interest determination unit detects predetermined objects from each wide-field image transmitted by the plurality of imaging devices, In wide-field images captured by the same imaging device at different times, the same object detected is tracked. Determine the top N objects with the largest amount of movement per unit time, and determine the point of interest that includes the determined objects. The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. The information processing system according to claim 1.

8. The system can accept settings to determine whether the aforementioned points of interest are determined at regular intervals or when a request to determine the aforementioned points of interest is received. If a setting is accepted to determine the aforementioned points of interest at regular intervals, The aforementioned point of interest determination unit determines the point of interest at regular intervals, The communication unit transmits to the communication terminal, at regular intervals, the determined point of interest and identification information of the imaging device that displays a wide-field image at the point of interest for each of the multiple imaging devices. The information processing system according to claim 1.

9. When a setting is received to determine the aforementioned points of interest, The point of interest determination unit determines the point of interest for the imaging device requested by the communication terminal from among the plurality of imaging devices. The communication unit transmits to the communication terminal that sent the request to determine the point of interest the identification information of the imaging device requested by the communication terminal and the determined point of interest. The information processing system according to claim 8.

10. The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. The communication terminal displays that a predetermined region image of the wide-field image, identified by the identification information of the imaging device, is being displayed based on the point of interest. The information processing system according to claim 1.

11. A communication system comprising: an information processing system that receives wide-field images from multiple imaging devices; and multiple communication terminals that receive the wide-field images from the information processing system. The aforementioned information processing system is A communication unit that receives viewpoint information of the wide-field image received from the user by the communication terminal, and identification information of the imaging device that transmitted the wide-field image from the multiple communication terminals, A viewpoint information management unit records viewpoint information of the wide-field image in association with identification information of the imaging device, It includes a point of interest determination unit that determines one or more points of interest of the imaging device based on viewpoint information of the wide-field image associated with the identification information of the imaging device, The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. The aforementioned communication terminal is At the point of interest received, the wide-field image of the imaging device, identified by the identification information of the imaging device, is displayed. Communication system.

12. A method for determining a point of interest performed by a communication system having an information processing system that receives wide-field images from multiple imaging devices, and multiple communication terminals that receive the wide-field images from the information processing system, The communication unit processes the reception of viewpoint information of the wide-field image received by the communication terminal from the user, and identification information of the imaging device that transmitted the wide-field image, from the multiple communication terminals. The viewpoint information management unit performs a process of associating the viewpoint information of the wide-field image with the identification information of the imaging device and recording it, The point of interest determination unit performs a process to determine one or more points of interest of the imaging device based on viewpoint information of the wide-field image associated with the identification information of the imaging device, The communication unit performs the process of transmitting to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. A method for determining points of interest.

13. An information processing system that transmits wide-field images received from multiple imaging devices to multiple communication terminals, A communication unit that receives viewpoint information of the wide-field image received from the user by the communication terminal, and identification information of the imaging device that transmitted the wide-field image from the multiple communication terminals, A viewpoint information management unit records viewpoint information of the wide-field image in association with identification information of the imaging device, It functions as a point of interest determination unit that determines one or more points of interest of the imaging device based on the viewpoint information of the wide-field image associated with the identification information of the imaging device, The communication unit transmits to the communication terminal the point of interest determined by the point of interest determination unit and identification information of the imaging device that displays a wide-field image at the point of interest. program.

Images