Information processing system, communication system, image sharing method, and program

Abstract

To provide a communication system capable of associating a wide-field image captured by an imaging device in response to an imaging request from a communication terminal with viewpoint information in the communication terminal.SOLUTION: The present invention is a communication system including an imaging device capable of capturing wide-field images having a wide range of viewing angles, and an information processing system that transmits a video including a first wide-field image captured by the imaging device to a plurality of communication terminals. The information processing system includes a communication unit that transmits an imaging request to the imaging device when a request is received from the communication terminal. The imaging device includes an imaging unit that captures a second wide-field image in response to the imaging request. The information processing system further includes an association processing unit that associates the second wide-field image with viewpoint information specifying a predetermined area when the wide-field image designated on the communication terminal is displayed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, a communication system, an image sharing method, and a program. [Background technology]

[0002] There is known a communication system that transmits images and audio from one location to one or more other locations in real time, and enables remote communication between users in remote locations using images and audio. As such images, there are known wide-field-of-view images with a wide viewing angle captured in a wide imaging range including, for example, a 360-degree image (also called a celestial sphere image, omnidirectional image, or panoramic image) in which the entire 360-degree surroundings are captured as an imaging range that includes areas that cannot be confirmed with a normal angle of view, and a user can view a part of the wide-field-of-view image from any virtual viewpoint by changing the virtual viewpoint of a part of the wide-field-of-view image displayed on the display screen of the communication terminal by operating the communication terminal.

[0003] These wide-field images may contain subjects that should not be shown, and techniques have been devised to prevent users from seeing such subjects (see, for example, Patent Document 1). Patent Document 1 discloses a technique for cutting out a more suitable range from a wide-field image without requiring complicated operations. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technologies, only a portion of the wide-field image is cut out, so for example, when performing remote communication such as collaboration between a remote location and an on-site location, if a communication terminal displays a specific area when the wide-field image is viewed from a specific virtual viewpoint, when it is desired to save or share the wide-field image captured by an imaging device, it is not possible to identify from which viewpoint it was viewed.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide an efficient communication system that enables a wide-field-of-view image captured by an imaging device to be associated with viewpoint information in a communication terminal. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a communication system including an imaging device capable of capturing a wide-field-of-view image having a wide range of viewing angles, and an information processing system that transmits a video including a first wide-field-of-view image captured by the imaging device to a plurality of communication terminals, The information processing system has a communication unit that sends an imaging request to the imaging device when a request is received from the communication terminal, and the imaging device has an imaging unit that captures a second wide-field image in response to the imaging request, and the information processing system further has an association processing unit that associates the second wide-field image with viewpoint information that identifies a specified area when displaying the wide-field image specified in the communication terminal. Effect of the Invention

[0007] It is possible to provide a communication system that can associate a wide-field-of-view image captured by an imaging device with viewpoint information in a communication terminal. [Brief description of the drawings]

[0008] [Figure 1] 1A and 1B are diagrams illustrating an example of remote communication using a wide-field-of-view image. [Diagram 2] FIG. 1 is a schematic diagram illustrating an example of a configuration of a communication system. [Diagram 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal and an information processing system. [Diagram 5] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. [Figure 6] FIG. 1 is a conceptual diagram of an imaging device in use. [Figure 7](a) is a hemispherical image (front) captured by an imaging device, (b) is a hemispherical image (back) captured by an imaging device, and (c) is an image represented by equirectangular projection. [Figure 8] (a) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (b) is a diagram showing a spherical image. [Figure 9] FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional solid sphere. [Figure 10] FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on a display. [Figure 11] 13 is a diagram showing the relationship between predetermined region information and an image of a predetermined region T. FIG. [Figure 12] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 13] FIG. 1 is a diagram illustrating an example of a functional configuration of a communication system. [Figure 14] 4 is a conceptual diagram showing image management information stored in an image management information storage unit. FIG. [Figure 15] 1 is a conceptual diagram showing virtual room information stored in a virtual room information storage unit and tenant information stored in a tenant information storage unit. FIG. [Figure 16] 1A is a diagram showing an example of an entry screen, and FIG. 1B is a diagram showing an example of an image viewing screen displayed by a communication terminal when a user enters a virtual room. [Figure 17] 11 is an example of a sequence diagram illustrating a process in which a user (or a communication terminal) enters a virtual room. [Figure 18] FIG. 13 is a diagram illustrating an example of a device registration screen displayed on the communication terminal. [Figure 19] 1A is a diagram showing an example of an imaging device registration dialogue, and FIG. 1B is a diagram showing an example of a two-dimensional code screen. [Figure 20] FIG. 13 is a diagram showing an example of a VR goggles registration screen that is displayed when a VR goggles registration button is pressed. [Figure 21]FIG. 13 is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room. [Figure 22] FIG. 13 is a diagram showing an example of a virtual room association screen (part 2). [Figure 23] FIG. 13 is a diagram showing an example of a virtual room association screen (part 3). [Figure 24] 13 is a diagram showing an example of a wide-field image transmission start / stop dialogue displayed by the communication terminal. FIG. [Diagram 25] 11 is an example of a sequence diagram showing a procedure in which a user registers an imaging device in a virtual room. [Figure 26] FIG. 13 is an example of a sequence diagram illustrating a flow in which a wide-field image is shared. [Figure 27] FIG. 13 is a diagram showing an example of an image viewing screen displayed by the communication terminal. [Figure 28] FIG. 13 is a diagram showing an example of an image viewing screen on the communication terminal on which a download button is displayed in response to pressing of an image capture button. [Figure 29] 1A is an example of a diagram showing an image viewing screen displayed before a user presses a download button, and FIG. 1B is an example of a diagram showing an image viewing screen displayed after a user presses a download button. [Diagram 30] FIG. 13 is a diagram illustrating an example of an image viewing screen on which thumbnail images are displayed. [Diagram 31] FIG. 13 is a diagram showing an example of an image viewing screen in the case where there are three image columns. [Diagram 32] 11 is an example of a sequence diagram illustrating a process in which a user requests an imaging device to capture an image from a communication terminal when a wide-field-of-view image is shared. FIG. [Diagram 33] FIG. 13 is a diagram showing a modified example of a sequence diagram illustrating a process in which a user requests an imaging device to capture an image from a communication terminal when a wide-field-of-view image is shared. [Diagram 34] FIG. 1 is a diagram illustrating an example of remote communication in which the communication system is applied to remote medical care. [Diagram 35]FIG. 13 is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room in the case of remote medical care. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An information processing system and an image sharing method performed by the information processing system will be described below as an example of an embodiment of the present invention.

[0010] <An example of remote communication> Fig. 1 is a diagram for explaining an example of remote communication using a wide-field-of-view image. In Fig. 1, three locations A to C are communicating via an information processing system 50. The number of locations is merely an example, and may be two locations, or four or more locations.

[0011] As an example, the site A is a construction site. Sites B and C can be anywhere, for example, an office, or other place where wide-field images can be communicated. At site A, an imaging device 10 is disposed that can capture a wide-field image, such as a celestial sphere image, or a wide-angle image of the surroundings, such as 180 to 360 degrees in the vertical or horizontal directions. Hereinafter, such a wide-angle image is simply referred to as a wide-field image. At sites A to C, various communication terminals 30A to 30C that view the wide-field images are disposed. Hereinafter, any one of the communication terminals 30A to 30C is referred to as a "communication terminal 30."

[0012] At the construction site, various construction works are being carried out by workers at each location, and while capturing the entire image using a wide-field-of-view image, users at each of the bases A to C can arbitrarily change the virtual viewpoint to check any construction or work that they want to focus on. The viewpoint is the center position or range of a specific area displayed on a display screen of a display or the like within the entire wide-field-of-view image.

[0013] The imaging device 10 is attached to a tripod 86 or an arm 85 via a gimbal 87. A relay device (in FIG. 1, the communication terminal 30A also serves as the relay device) is installed at the construction site, and the communication terminal 30A transmits the wide-field-of-view image received from the imaging device 10 by wire or wirelessly to the information processing system 50. The communication terminal 30A can also be a terminal for viewing the wide-field-of-view image. A camera 9 is connected to the communication terminal 30A (may be built-in), and an image with a normal angle of view (which may be capable of capturing a celestial sphere image) captured by the camera 9 can also be transmitted to the information processing system 50. In addition, a user A (which may be a worker, for example) can wear smart glasses 88, and an image with a normal angle of view (which may be capable of capturing a celestial sphere image) captured by the smart glasses 88 may be transmitted to the information processing system 50. The smart glasses 88 are an information terminal that displays information acquired via the Internet on a display while ensuring a field of view. The smart glasses 88 may be placed at any base.

[0014] At site B, a PC (Personal Computer), a smartphone, or the like is disposed as an example of the communication terminal 30B. The communication terminal 30B may be any device capable of communicating with the information processing system 50, and may also be a tablet terminal, a PDA (Personal Digital Assistant), an electronic whiteboard, a projector, or the like. A camera may be built into or connected to the communication terminal 30B.

[0015] At the site C, a PC, a smartphone, a VR goggles 89, etc. are placed as examples of the communication terminal 30C, and in FIG. 1, a camera 8 is built in or connected to the communication terminal 30C. The VR goggles 89 are information terminals that display an artificial world or a celestial sphere image on a computer according to the direction of movement of the neck or body. The camera 8 may be a wide-angle or normal-angle camera. In addition, the communication terminal 30C may be any device that can communicate with the information processing system 50, and may be a tablet terminal, a PDA, an electronic whiteboard, a projector, etc. The VR goggles 89 may be placed at any site.

[0016] In this embodiment, the imaging device 10 and the communication terminal 30 are managed in a communication group called a virtual room. The imaging device 10 is associated with the virtual room, and the communication terminal 30 (the user who operates the communication terminal 30) enters the virtual room to receive the wide-field image transmitted by the imaging device 10, allowing the user to view the wide-field image. Smart glasses 88 and VR goggles 89 can also be associated with the virtual room. The cameras 8 and 9 enter the virtual room together with the communication terminal 30.

[0017] Users A to C at locations A to C can arbitrarily change the viewpoint of the wide-field-of-view image. For this reason, there is a possibility that each of users A to C viewing the wide-field-of-view image in real time is looking at a different viewpoint, which may make communication difficult. Therefore, in this embodiment, information on a virtual viewpoint set in a communication terminal 30 at a location can be shared with communication terminals 30 at other locations. An overview of sharing will be described. In the following, for the sake of explanation, it is assumed that the viewpoint specified by the user at location B is shared with users at locations A and C.

[0018] (1) The communication terminals 30A to 30C share a wide-field-of-view image (an example of a first wide-field-of-view image) captured by the imaging device 10. When a user B requests imaging of a wide-field-of-view image while viewing the image from an arbitrary viewpoint on the communication terminal 30B, the communication terminal 30B (an example of a first communication terminal) transmits viewpoint information and an imaging request to the information processing system 50.

[0019] (2) In response to an image capture request, the information processing system 50 requests the image capture device 10 to capture an image (either a still image or a moving image) by specifying viewpoint information.

[0020] (3) The imaging device 10 captures an image in response to an imaging request, and stores the wide-field image (an example of a second wide-field image) and viewpoint information in association with a URL (an example of storage destination information; in FIG. 1, the storage location on the storage 90 is shown) notified by the information processing system 50. The wide-field image stored in the storage 90 can be downloaded and displayed by any communication terminal 30.

[0021] (4) The information processing system 50 transmits the URL to the communication terminal 30B.

[0022] (5) Furthermore, the information processing system 50, automatically or in response to a request from user B, transmits the URL to the communication terminals 30A and 30C (an example of a second communication terminal) that are currently in the same virtual room.

[0023] (6) The communication terminals 30A and 30C connect to the URL to receive the viewpoint information and the wide-field image, and display the wide-field image identified by the viewpoint information by setting the viewpoint to match the center of the image field. Note that the viewpoint does not need to be perfectly matched to the center, and the viewpoint may be set to be included in a range near the center of the image field.

[0024] The same applies when the viewpoint of a user at location A is shared with users at locations B and C, and when the viewpoint of a user at location C is shared with users at locations A and B.

[0025] As described above, in the communication system 1 of this embodiment, even when a wide-field-of-view image is distributed, viewpoint information is shared without the need to instruct each location to move the viewpoint relative to the wide-field-of-view image captured so that the specific area that was being focused on is displayed, making it easy for users to communicate their intentions.

[0026] It is also possible that the imaging device 10 transmits the wide-field image itself to the information processing system 50 in (3), and the information processing system 50 transmits the wide-field image to the communication terminals 30A to 30C in (4).

[0027] In FIG. 1, an example in which the imaging device 10 is placed at a construction site has been described, but the present embodiment can also be applied to VR (Virtual Reality) education, event distribution, remote customer service, remote medical care, and the like. In VR education, the imaging device 10 is placed at a base at the site, such as a laboratory, and students can freely change the viewpoint from a remote base to view a blackboard, equipment, samples, experiment results, and the like. In event distribution, the imaging device 10 is placed at the location where the event is held at the site, and event participants such as spectators can freely change the viewpoint online from a remote base to view the state of the event site. The state of the event site includes images of subjects such as performers, contestants, presenters, products and exhibits explained at the event, images of materials explained at the event, images of the state of the event site, and the like. The event site may be indoors or outdoors, and includes venues for sports, concerts, plays, and the like. In remote customer service, for example, when applied to customer service at a travel agency, the imaging device 10 is placed at the travel destination at the site, and customers can freely change the viewpoint from a remote base to consider their itinerary. In remote medical care, the imaging device 10 is placed in a medical facility such as an operating room, and doctors, students, medical equipment personnel, etc. can change the viewpoint at will from a remote location and view the actions of doctors and nurses performing medical procedures at the medical facility, the placement of instruments, the patient's condition, vital signs, etc.

[0028] The locations where images are captured are not limited to these, and can be any space where there is a need for a user (viewer) at the viewing location to understand the situation at a remote location, such as a school, factory, warehouse, construction site, server room, or store.

[0029] <Terminology> A tenant refers to a group of users linked to a contract unit when signing a contract to receive an image distribution service from a service provider (an information processing system in this embodiment), and is a company, organization, individual, etc. that has signed the contract. Therefore, a tenant can also be referred to as a user group. As an example, a user belongs to a tenant, but a user may subscribe to the service individually. In addition to users, imaging devices, virtual rooms, etc. are registered in a tenant (user group).

[0030] A base is a place where activities are based. In this embodiment, a conference room is used as an example of a base. A conference room is a room that is set up mainly for use in meetings. A conference is also called a meeting, a conference, a gathering, a get-together, a get-together, or the like.

[0031] The device refers to a device other than a general-purpose communication terminal 30 such as a PC or a smartphone, and is an imaging device or a viewing device for a wide-field image. In this embodiment, examples of the device include the imaging device 10, smart glasses 88, and VR goggles 89.

[0032] The viewpoint information is parameter information that specifies which predetermined area of ​​the wide-field-of-view image is to be displayed on the display screen of the display. In this embodiment, the radius vector, polar angle, and azimuth angle corresponding to the center of the wide-field-of-view image displayed on the display screen of the display are described as examples of viewpoint information, but the viewpoint information may be specified by other parameter information such as the coordinates of the diagonal vertices.

[0033] A wide-field-of-view image means an image having a wide viewing angle that is wider than the display range that can be displayed at one time on the display screen (area where the wide-field-of-view image is displayed) of a display in a specified display method. A wide-field-of-view image has a display range of up to 360 degrees (or 180 degrees) in the vertical direction and 360 degrees in the horizontal direction, but even if it is less than 360 degrees up and down or less than 360 degrees left and right, it is included in the wide-field-of-view image as long as it has a wide viewing angle that is wider than the display range that can be displayed at one time on the display. For example, an image having a display range wider than the range that a human can visually recognize at one time is also included in the wide-field-of-view image. In addition, an image having a display range of 160 degrees or more in the vertical and horizontal directions is also included in the wide-field-of-view image. Note that even if an image can be displayed at one time on the display screen of a display depending on the display method, it is included in the wide-field-of-view image if it has a wide viewing angle by switching or changing to a specified display method. In this embodiment, an equirectangular omnidirectional image is described as an example of a wide-field image, but the wide-field image also includes an omnidirectional image, a hemispherical image, a 3D panoramic image, a 2D panoramic image, and a VR (Virtual Reality) image. The wide-field image may be, for example, an image in a cube mapping format or a dome master format. The omnidirectional image may be in a format other than the equirectangular format.

[0034] An image captured at a normal angle of view is not a wide-field image, but in this embodiment, it will be described as an image that is not a wide-field image (planar image).

[0035] A communication group is a group of users to whom wide-field images are shared (distributed). In a normal space, a communication group is described as a virtual room, in the sense that wide-field images can be shared by each user when they enter the same room. Here, "virtual" means that it is realized by information processing via a network.

[0036] Users at each location communicate remotely with each other. Remote communication is a meeting held at a remote location. A meeting is when people get together for consultation or discussion. Meetings can take various forms, such as customer service, conferences, meetings, consultations, study sessions, classes, seminars, and presentations. Communication does not necessarily have to be two-way. Therefore, a virtual room may be called a virtual conference room.

[0037] <Example of communication system configuration> FIG. 2 is an example of a schematic diagram of the configuration of the communication system 1. FIG. 1 is an example in which the communication system 1 of FIG. 2 is applied to remote communication with a site. The communication system 1 is a system for bidirectionally transmitting and receiving wide-field images and normal-angle images captured by an imaging device 10 between multiple bases, and is a system in which an image distributed from a certain base is displayed at another base, so that a user at the other base can view the image. Note that, as an example of a wide-field image, a celestial sphere image captured by the imaging device 10 is distributed. For example, the communication system 1 allows a wide-field image captured at a specific base to be viewed at another base in a remote location.

[0038] As shown in FIG. 2, in the communication system 1, an imaging device 10, a communication terminal 30A, an information processing system 50, and communication terminals 30B and 30C, each of which is located at a plurality of bases (bases B and C), are communicatively connected to each other.

[0039] If the imaging device 10 has a communication function that allows it to directly connect to the communication network N, the communication terminal 30A is not necessary as a relay device (e.g., a router). In this case, the communication terminal 30A is connected to the communication network N without the imaging device 10. However, if the communication terminal 30A is placed at the site A, the communication terminal 30A also functions as a relay device, and user A can view the wide-field image in the same way as the communication terminals 30B and 30C. Note that an imaging device 10 may be placed at a site other than the site A, and multiple imaging devices 10 may be placed at the site A.

[0040] Each communication terminal 30 and the information processing system 50 can communicate with each other via a communication network N. The communication network N is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), etc. Note that the communication network N may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0041] As described below, the imaging device 10 is a digital camera capable of capturing a celestial sphere image by capturing an image of a subject, scenery, or the like and obtaining two original hemispherical images. The wide-field image obtained by the imaging device 10 may be a moving image or a still image, or may be both a moving image and a still image. The captured image may also be a video including sound together with the image.

[0042] The communication terminal 30 is a computer such as a PC used by a user at each base. The communication terminal 30 displays an image captured at the own base, a wide-field image (still image or video) distributed from another base, and an image with a normal angle of view. The communication terminal 30 acquires a wide-field image captured by the imaging device 10, for example, via a communication network N. In addition, software for performing image processing such as OpenGL ES is installed in the communication terminal 30, and an image display based on viewpoint information that specifies a partial area of ​​the wide-field image can be performed. Note that OpenGL ES is an example of software for performing image processing, and other software may be used. In addition, even if software for performing image processing is not installed, image processing may be performed by software received from the outside, or image display may be performed by receiving the result of image processing performed by external software. In other words, the communication terminal 30 can display a predetermined partial area of ​​the wide-field image.

[0043] The communication terminal 30 can arbitrarily change the viewpoint relative to the display range of the wide-field image in response to user operation. The communication terminal 30 can change and display the field of view range (predetermined area) based on viewpoint information corresponding to the viewpoint after the movement by moving the virtual viewpoint in response to user operation input (including key input, drag, scroll, etc.) on a touch panel, directional buttons, mouse, keyboard, touch pad, etc. Furthermore, in the case where the communication terminal 30 is a communication terminal worn by a user such as VR goggles, for example, the communication terminal 30 may detect posture information of the communication terminal 30 changed in response to changes in the movement of the user wearing the goggles, and move the virtual viewpoint in response to the detected posture information, thereby changing and displaying the field of view range (predetermined area) based on viewpoint information corresponding to the viewpoint after the movement.

[0044] The communication terminal 30A distributes the wide-field image acquired from the imaging device 10 via a wired cable such as a USB (Universal Serial Bus) cable to the communication terminal 30 at another site via the information processing system 50. The connection between the imaging device 10 and the communication terminal 30A may be a wireless connection using short-range wireless communication or the like instead of a wired connection using a wired cable. A plurality of communication terminals 30A may be arranged at site A.

[0045] In some cases, a user at site A wears the smart glasses 88, and the smart glasses 88 connect to the communication network. In this case, an image captured by the smart glasses 88 is transmitted to the information processing system 50 via the communication network, and the information processing system 50 can distribute the image to the communication terminal 30 at each site.

[0046] The communication terminal 30B is located at a location B where a user B is present, and the communication terminal 30C is located at a location C where a user C is present. A plurality of communication terminals 30B and 30C may be located at the locations B and C. The communication terminal 30B and the communication terminal 30C may be carried by the user B and the user C, respectively.

[0047] The communication terminals 30A to 30C at the locations A to C can have built-in or external cameras 8, 9 as an example of an imaging unit, and the communication terminals 30A to 30C can distribute images of their own locations captured by their own cameras 8, 9 to other locations. Any device may be installed at the locations A to C.

[0048] The arrangement of each terminal and device (communication terminal 30 and imaging device) and user shown in Fig. 2 is one example, and other examples may be used. Furthermore, the communication terminal 30 is not limited to a PC, and may be, for example, a tablet terminal, a smartphone, a PDA, a wearable device (including smart glasses and VR goggles), a PJ (Projector), an electronic whiteboard (a whiteboard with an electronic whiteboard function capable of mutual communication), or an autonomous robot. The communication terminal 30 may be any computer on which a web browser or an application dedicated to an image distribution service runs.

[0049] Furthermore, if the imaging device 10 has a display, it may be configured to display images distributed from other locations.

[0050] The information processing system 50 has one or more information processing devices. The information processing system 50 manages and controls the communication between the imaging device 10 and the communication terminal 30 between each base, and manages the wide-field images transmitted and received. The information processing system 50 provides a platform that can utilize functions required to provide an image distribution service that distributes wide-field images. This platform may be made available to service providers such as individuals and companies that wish to provide image distribution services by contract. Hereinafter, a service provider that provides an image distribution service to users using a contracted platform is referred to as a platform contractor, in order to distinguish it from a tenant that receives the image distribution service.

[0051] For this reason, the information processing system 50 may publish an API (Application Programming Interface) as a platform, and the platform subscriber may use this API to provide various image distribution services. The platform subscriber only needs to develop software such as applications that mainly perform screens displayed by the communication terminal 30 and calls the API, and does not need to develop functions provided by the API, such as image distribution, from scratch.

[0052] The information processing system 50 may be constructed by a single computer, or may be constructed by multiple computers to which each unit (function or means) is divided and arbitrarily assigned. In addition, all or part of the functions of the information processing system 50 may be a server computer existing in a cloud environment, or may be a server computer existing in an on-premise environment.

[0053] The storage 90 is a storage device that stores data such as captured wide-field images. The storage 90 may be an external storage separate from the information processing system 50 (which may be a storage arranged on a cloud or a storage arranged on-premise), or may be a storage included in the information processing system 50.

[0054] <Hardware configuration example> Next, the hardware configuration of each device or terminal included in the image communication system according to this embodiment will be described with reference to Figures 3 and 4. Note that components may be added or deleted from the hardware configurations shown in Figures 3 and 4 as necessary.

[0055] <<Hardware configuration of the imaging device>> First, the hardware configuration of the imaging device 10 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the hardware configuration of the imaging device 10. In the following, the imaging device 10 is assumed to be an omnidirectional (all-sphere) imaging device using two imaging elements, but the imaging element may be one, or two or more. In addition, the imaging device does not necessarily have to be a device dedicated to omnidirectional imaging, and an omnidirectional imaging unit may be attached to a normal digital camera, smartphone, or the like to have substantially the same functions as the imaging device 10.

[0056] As shown in FIG. 3, the imaging device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0057] Among these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter, when there is no need to distinguish between them, they will be referred to as lenses 102) each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to the lenses. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates a horizontal or vertical synchronization signal or a pixel clock for the image sensor, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set. Note that the configuration in which the imaging unit 101 includes two wide-angle lenses is merely an example, and the imaging unit 101 may include only one wide-angle lens, or may include three or more wide-angle lenses.

[0058] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the input / output I / F 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network I / F 121 are also connected to the bus 110.

[0059] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs a predetermined processing on each image data, and then synthesizes the image data to create data for an equirectangular projection image (an example of a wide-field image) described below.

[0060] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0061] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function corresponding to video display on a display (for example, a display of an external terminal such as a smartphone that performs short-distance communication with the imaging device 10 using the short-distance communication circuit 117). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / min).

[0062] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via an I / F bus and performs a predetermined process on the sound data.

[0063] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104, and data of a processed equirectangular projection image.

[0064] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, etc. A user operates the operation unit 115 to input various imaging modes, imaging conditions, etc.

[0065] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as necessary.

[0066] The short-range communication circuit 117 communicates with an external terminal (device) by a short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the imaging device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0067] The electronic compass 118 calculates the direction of the imaging device 10 from the earth's magnetism and outputs direction information. This direction information is an example of related information (metadata) according to Exif and is used for image processing such as image correction of the captured image. The related information also includes each data of the image capture date and time and the data capacity of the image data. The gyro sensor 119 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) accompanying the movement of the imaging device 10. The angle changes are an example of related information (metadata) according to Exif and are used for image processing such as image correction of the captured image. The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The imaging device 10 calculates the attitude (angle with respect to the direction of gravity) of its own device (imaging device 10) based on the acceleration detected by the acceleration sensor 120. By providing the acceleration sensor 120, the imaging device 10 improves the accuracy of image correction.

[0068] The network I / F 121 is an interface for performing data communication using a communication network N such as the Internet via a router or the like.

[0069] Furthermore, the hardware configuration of the imaging device 10 is not limited to that shown here, and may be anything that can realize the functional configuration of the imaging device 10. Furthermore, at least a part of the above hardware configuration may exist on a network.

[0070] <<Hardware configuration of communication terminal>> 4 is a diagram showing an example of the hardware configuration of the communication terminal 30 and the information processing system 50. First, the communication terminal 30 will be described. The hardware configuration of the communication terminal 30 is indicated by a reference number in the 300 series. The communication terminal 30 is constructed by a computer, and includes a CPU 301, a ROM 302, a RAM 303, a HDD (Hard Disk Drive) 304, a HDD controller 305, a display 306, an external device connection I / F 308, a network I / F 309, a bus line 310, a keyboard 311, a pointing device 312, a DVD-RW (Digital Versatile Disk Rewritable) drive 314, a media I / F 316, an audio input / output I / F 317, a microphone 318, a speaker 319, a short-distance communication circuit 320, and a camera 321, as shown in FIG.

[0071] Among these, the CPU 301 controls the operation of the entire communication terminal 30. The ROM 302 stores programs such as an IPL used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HDD 304 stores various data such as programs and data. The HDD controller 305 controls reading or writing of various data from the HDD 304 according to the control of the CPU 301. The display 306 displays various information such as a cursor, a menu, a window, a character, or an image. The display 306 is an example of a display unit. The display 306 may be a touch panel display equipped with an input means. The external device connection I / F 308 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory or a printer. The network I / F 309 is an interface for data communication using the communication network N. The bus line 310 is an address bus or a data bus for electrically connecting each component such as the CPU 301 shown in FIG. 4. The HDD 304 and the HDD controller 305 are examples of storage that stores programs, data, and the like, and may be an SSD (Solid State Drive) and an SSD controller, respectively.

[0072] The keyboard 311 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. The input means may be not only the keyboard 311 and the pointing device 312, but also a touch panel, a voice input device, etc. The DVD-RW drive 314 controls reading or writing of various data from the DVD-RW 313 as an example of a removable recording medium. The DVD-RW 313 may be a DVD-R or a Blu-ray (registered trademark) Disc, etc. The media I / F 316 controls reading or writing (storing) of data from the recording medium 315 such as a flash memory. The microphone 318 is a type of built-in sound collecting means for inputting sound. The sound input / output I / F 317 is a circuit for processing input / output of sound signals between the microphone 318 and the speaker 319 under the control of the CPU 301. The short-range communication circuit 320 is a communication circuit for communicating with an external terminal (device) by short-range wireless communication technology such as NFC, Bluetooth (registered trademark), or Wi-Fi. The camera 321 is a type of built-in imaging means that captures an image of a subject and obtains image data. The microphone 318, the speaker 319, and the camera 321 may be external devices rather than built into the communication terminal 30.

[0073] Furthermore, the hardware configuration of the communication terminal 30 is not limited to that shown here, and may be anything that can realize the functional configuration of the communication terminal 30. Furthermore, at least a part of the above hardware configuration may exist on the network.

[0074] <<Hardware configuration of information processing system>> As shown in Fig. 4, each hardware configuration of the information processing system 50 is indicated by a reference number in parentheses in the 500 range. The information processing system 50 is constructed by a computer, and as shown in Fig. 4, has the same configuration as the communication terminal 30, so a description of each hardware configuration will be omitted.

[0075] Furthermore, the hardware configuration of the information processing system 50 is not limited to that shown here, and may be anything that can realize the functional configuration of the information processing system 50. Furthermore, at least a part of the above hardware configuration may exist on a network.

[0076] Each of the above programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the communication terminal 30 realizes the image display method according to the present invention by executing the program according to the present invention.

[0077] <Wide-field images and viewpoint information> A method for generating a wide-field image (panoramic image) will be described below with reference to FIGS.

[0078] First, the appearance of the imaging device 10 will be described with reference to Fig. 5. The imaging device 10 is a digital camera for obtaining captured images that are the basis of a spherical (360°) image. Fig. 5(a) is a left side view of the imaging device, Fig. 5(b) is a front view of the imaging device, and Fig. 5(c) is a plan view of the imaging device. This external view is merely one example of the imaging device 10, and other external views may be used.

[0079] As shown in FIG. 5(a), the imaging device 10 is sized so that a person can hold it in one hand, but this shape is merely an example and may be other shapes. Also, as shown in FIG. 5(a), FIG. 5(b), and FIG. 5(c), an imaging element 103a is provided on the front side (front side) and an imaging element 103b is provided on the back side (rear side) of the upper part of the imaging device 10. These imaging elements (image sensors) 103a and 103b are used in conjunction with optical members (for example, lenses 102a and 102b described later) capable of capturing hemispherical images (angle of view of 180° or more). Also, as shown in FIG. 5(b), an operation unit 115 such as a shutter button is provided on the surface opposite to the front side of the imaging device 10. As described above, the imaging device 10 may have only one imaging element, or may have three or more imaging elements.

[0080] Next, the usage of the imaging device 10 will be described with reference to Fig. 6. Fig. 6 is an image diagram of the imaging device in use. As shown in Fig. 6, the imaging device 10 is used, for example, to capture an image of a subject around the imaging device 10. In this case, two hemispherical images can be obtained by capturing images of the subject around the imaging device 10 using the imaging element 103a and the imaging element 103b shown in Fig. 5, respectively.

[0081] Next, an outline of the process of creating a celestial sphere image from an image captured by the imaging device 10 will be described with reference to Fig. 7 and Fig. 8. Fig. 7(a) is a diagram showing a hemispherical image (front side) captured by the imaging device, Fig. 7(b) is a diagram showing a hemispherical image (rear side) captured by the imaging device, and Fig. 7(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 8(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 8(b) is a diagram showing a celestial sphere image.

[0082] As shown in Fig. 7(a), the image obtained by the imaging element 103a becomes a hemispherical image (front side) curved by the lens 102a described below. Also, as shown in Fig. 7(b), the image obtained by the imaging element 103b becomes a hemispherical image (rear side) curved by the lens 102b described below. Then, the imaging device 10 synthesizes the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees to generate an equirectangular projection image EC as shown in Fig. 7(c).

[0083] The imaging device 10 uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to paste an equirectangular projection image EC so as to cover the sphere as shown in FIG. 8(a) and generate a celestial sphere image (a celestial sphere panoramic image) CE as shown in FIG. 8(b). In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. OpenGL ES is merely an example of software that executes image processing, and the celestial sphere image CE may be generated by other software. In addition, the celestial sphere image CE may be a still image or a video. Note that, although an example in which the imaging device 10 generates a celestial sphere image has been described here, the information processing system 50 or the communication terminal 30 may execute the same image processing or a part of the image processing steps.

[0084] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people when they see it. Therefore, in the imaging device 10 or the communication terminal 30, a predetermined region T (hereinafter, referred to as a "predetermined region image") of the spherical image CE is displayed as a planar image with little curvature, so that the image can be displayed without giving a sense of incongruity to people. This will be described with reference to Figs. 9 and 10.

[0085] FIG. 9 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of a virtual viewpoint of a user who views the omnidirectional image CE displayed as a three-dimensional sphere. FIG. 10(a) is a three-dimensional perspective view of FIG. 9, and FIG. 10(b) is a diagram showing a predetermined area image when displayed on a display. FIG. 10(a) shows the omnidirectional image CE shown in FIG. 9 as a three-dimensional sphere CS. If the omnidirectional image CE generated in this way is a sphere CS, as shown in FIG. 9, the virtual camera IC is located inside the omnidirectional image CE. The predetermined area T in the omnidirectional image CE is an imaging area of ​​the virtual camera IC, and is specified by predetermined area information indicating the imaging direction and the angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. In addition, the zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the omnidirectional image CE. The predetermined area image Q is an image of a predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 11).

[0086] Then, the predetermined area image Q shown in Fig. 10(a) is displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in Fig. 10(b). The image shown in Fig. 10(b) is a predetermined area image represented by the initial (default) predetermined area information. The following description will be given using the imaging direction (ea, aa) and the angle of view (α) of the virtual camera IC. Note that the predetermined area T may be represented by the position coordinates (X, Y, Z) of the imaging area of ​​the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

[0087] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 11. FIG. 11 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 11, "ea" indicates an elevation angle, "aa" indicates an azimuth angle, and "α" indicates an angle of view. That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the imaging direction (ea, aa) becomes the center point CP(x, y) of the predetermined area T, which is the imaging area of ​​the virtual camera IC. As shown in FIG. 11, the center point CP(x, y) when the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC is α becomes the parameter ((x, y)) of the predetermined area information. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between any vertex of the predetermined region T and the center point CP(x, y) (2L is the diagonal). In FIG. 11, the trigonometric function generally expressed by the following (Equation 1) holds.

number

[0088] FIG. 12 is a diagram showing the relationship described in FIG. 11 by points in a three-dimensional Euclidean space by spherical coordinates. Here, the position coordinates when the center point CP shown in FIG. 11 is expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are a radius, a polar angle, and an azimuth angle, respectively. The radius r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to the center point CP, and is therefore equal to the distance f shown in FIG. 11. FIG. 12 is a diagram showing these relationships. Hereinafter, the position coordinates (r, θ, φ) of the virtual camera IC will be used as an example of viewpoint information for explanation. Note that the viewpoint information may be parameter information that can specify the predetermined area T (predetermined area image Q) displayed as an image of the imaging area of ​​the virtual camera IC on the predetermined display shown in FIG. 10 as described above, and also includes the coordinates of the diagonal vertices of the predetermined area T. In addition, the viewpoint information may be information indicating the angle of view α of the virtual camera IC described in FIG. 11 and information indicating the center point CP (x, y). The viewpoint information may be information indicating the angle of view α and the azimuth angle aa of the virtual camera IC described in FIG. 11. The viewpoint information may include not only position coordinate information based on spherical coordinates, but also position coordinate information based on Cartesian coordinates and a difference value of coordinates from the initially set (default) predetermined area information. The viewpoint information may be information other than coordinate information such as angle and distance as shown in FIG. 11. Although the center point of the predetermined area T is used as a reference in FIG. 11 and FIG. 12, the predetermined area T may be specified by parameter information based on one of the vertices of the predetermined area T. Note that the viewpoint information has been described above by taking the case where the wide-field image is a celestial sphere image as an example, but in the case of other wide-field images, information specifying the predetermined area T in the wide-field image becomes the viewpoint information. The viewpoint information may include parameter information such as the height and width of the predetermined area T and parameter information such as the magnification ratio by zooming the virtual camera IC. In addition, when the position of each pixel of the equirectangular projection image EC as shown in Fig. 7(c) is associated with the coordinates on the surface of a sphere (for example, coordinates on two axes of latitude and longitude), parameter information such as the direction and angle of view of the virtual camera IC may be used as viewpoint information, or information such as latitude and longitude may be included in the viewpoint information. In this way, viewpoint information is not necessarily limited to information indicating a point.

[0089] <About the function> Next, the functional configuration of the communication system 1 according to the present embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the functional configuration of the communication system 1 according to the present embodiment. Note that Fig. 13 shows those terminals, devices, and servers shown in Fig. 1 that are related to the processes or operations described below.

[0090] <<Functional configuration of the imaging device>> First, the functional configuration of the imaging device 10 will be described with reference to Fig. 13. The imaging device 10 has a communication unit 11, a reception unit 12, an imaging processing unit 13, an analysis unit 14, a registration request unit 15, a connection unit 16, a storage processing unit 17, an image transmission control unit 18, and a storage / readout unit 19. Each of these units is a function or a means for performing a function that is realized by operating any of the components shown in Fig. 3 in response to an instruction from the CPU 111 according to a program loaded on the SRAM 113 or the DRAM 114. The imaging device 10 also has a storage unit 1000 constructed by the ROM 112 shown in Fig. 3, etc.

[0091] The communication unit 11 has a function of connecting to the communication network N using a wireless communication means such as Wi-Fi and transmitting / receiving various data or information to / from other devices. In this embodiment, a form in which the wide-field image acquired by the imaging processing unit 13 is transmitted to the information processing system 50 via the connection unit 16 will be mainly described. However, it is also possible for the communication unit 11 to transmit the wide-field image to the information processing system 50.

[0092] The reception unit 12 is a function that receives operational input from a user to the imaging device 10. The reception unit 12 receives operational input from a user to the touch panel, buttons, etc., such as turning the power on or off, turning the shutter button on or off (starting or stopping transmission of a wide-field image), etc.

[0093] The imaging processing unit 13 captures an image of a subject, a landscape image, or the like, and acquires a captured image. The captured image acquired by the imaging processing unit 13 may be a moving image or a still image (or both), and may include sound together with the image. The imaging processing unit 13 also captures, for example, a two-dimensional code (see FIG. 19) displayed on the display 306 of the communication terminal 30. The imaging processing unit 13 may also generate a wide-field image by executing the image processing described in FIG. 7 and FIG. 8 on the captured image.

[0094] The analysis unit 14 analyzes the two-dimensional code captured and acquired by the imaging processing unit 13, and extracts information included in the two-dimensional code (a URL for registering the imaging device to a tenant, and a temporary ID and password).

[0095] The registration request unit 15 uses the information included in the two-dimensional code read by the analysis unit 14 to transmit a request to the information processing system 50 to register the imaging device 10 with the tenant of the information processing system 50 .

[0096] The connection unit 16 is realized by, for example, a short-distance communication circuit 117, and has a function of receiving power supply from the communication terminal 30A and performing data communication.

[0097] The storage processing unit 17 performs processing to store a wide-field image captured in response to an imaging request from an arbitrary location in a URL notified by the information processing system 50 (for example, the storage 90, etc.).

[0098] The image transmission control unit 18 is a function for controlling the transmission of wide-field images to the information processing system 50. For example, the image transmission control unit 18 transmits captured images acquired by the imaging processing unit 13 to the information processing system 50 periodically or in response to a user operation if the captured images are still images, or at a predetermined FPS (Frame Per Second) if the captured images are moving images. The image transmission control unit 18 also switches between the communication unit 11 and the connection unit 16.

[0099] The storage / readout unit 19 has a function of storing various data in the storage unit 1000 or reading out various data from the storage unit 1000. The storage unit 1000 also stores captured image data, an imaging device ID, and the like acquired by the imaging processing unit 13. The captured image data stored in the storage unit 1000 may be configured to be deleted when a predetermined time has elapsed since the captured image data was acquired by the imaging processing unit 13, or data transmitted to the information processing system 50 may be deleted.

[0100] In addition, an application (also called a plug-in) for supporting the communication system 1 is installed in the imaging device 10. This application is used when associating the imaging device 10 with a virtual room and accepting external control. Some of the functions shown in FIG. 13 (e.g., the registration request unit 15) are included in this application. In addition, the application for supporting the communication system 1 may be placed on a network, and the same function may be realized by accessing the application using a Web browser or the like that the imaging device 10 has.

[0101] <<Functional configuration of communication terminal>> Next, the functional configuration of the communication terminal 30 will be described with reference to Fig. 13. The communication terminal 30 has a communication unit 31, a reception unit 32, a display control unit 33, an imaging unit 34, a storage / readout unit 35, and a connection unit 36. Each of these units is a function or a means for performing a function that is realized by operating any of the components shown in Fig. 4 according to an instruction from the CPU 301 in accordance with a program (which may be a web browser or a dedicated application) deployed on the RAM 303. The communication terminal 30 also has a storage unit 3000 constructed by the ROM 302 or the recording medium 315 shown in Fig. 4.

[0102] The communication unit 31 (an example of a second communication unit) is realized by, for example, the network I / F 309, and has a function of connecting to the communication network N and transmitting / receiving various data or information to / from other devices.

[0103] The reception unit 32 is a function that receives various selections or operation inputs to the communication terminal 30. The display control unit 33 is a function that displays a wide-field image, an image with a normal angle of view, and various screens on the display 306 of the communication terminal 30. The display control unit 33 displays, for example, a two-dimensional code transmitted from the information processing system 50 on the display 306. The two-dimensional code is, for example, a QR code (registered trademark), DataMatrix (DataCode), MaxiCode, PDF417, or the like. The two-dimensional code may be a barcode.

[0104] The connection unit 36 ​​is realized by, for example, a short-range communication circuit 320, and has a function of supplying power to the image capture device 10 and performing data communication.

[0105] 4, and has the function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. An image management information storage unit 3001 is formed in the storage unit 3000. The image management information storage unit 3001 will be described in detail in the description of the information processing system 50.

[0106] <<Functional configuration of information processing system>> Next, the functional configuration of the information processing system 50 will be described. The information processing system 50 has a communication unit 51, a screen generation unit 52, an association processing unit 53, an image delivery unit 54, an authentication unit 55, a communication group management unit 56, a communication control unit 57, a connection management unit 58, a storage / reading unit 59, and an API management unit 60. Each of these units is a function or a means for performing a function that is realized by operating any of the components shown in Fig. 4 by an instruction from the CPU 501 according to a program loaded on the RAM 503. The information processing system also has a storage unit 5000 constructed by the ROM 502, HDD 504, or recording medium 515 shown in Fig. 4.

[0107] The communication unit 51 (an example of a first communication unit) has a function of transmitting and receiving various data or information to and from other devices via the communication network N.

[0108] The screen generation unit 52 generates screen information to be displayed by the communication terminal 30. When the communication terminal 30 executes a Web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), and the like. When the communication terminal 30 executes a native application, the screen information is held by the communication terminal 30, and the information to be displayed is transmitted in XML or the like. The screen generation unit 52 generates screen information on which the wide-field image and the like delivered by the image delivery unit 54 are arranged.

[0109] The association processing unit 53 controls association and sharing of viewpoint information of a wide-field-of-view image. When the association processing unit 53 receives an imaging request together with viewpoint information from the communication terminal 30, it performs processing to associate the acquired wide-field-of-view image with the viewpoint information by requesting imaging from the imaging device 10. Furthermore, it stores the associated wide-field-of-view image and viewpoint information in the image management information storage unit 5001. The association processing unit 53 also transmits storage destination information (e.g., URL) to the communication terminal 30 as information indicating a storage location where the associated wide-field-of-view image and viewpoint information are stored. Note that the information processing system 50 does not need to simultaneously receive the viewpoint information and the imaging request from the communication terminal 30, and may perform processing to associate them after receiving them separately. Also, the URL is an example of storage destination information indicating a storage location, and may be in another format such as a URI.

[0110] The image delivery unit 54 delivers images such as wide-field images sent by the imaging device 10 associated with the same virtual room to the communication terminal 30 operated by a user who is in the virtual room. Images with normal angles of view captured by the camera of the communication terminal 30 or the connected cameras 8 and 9 are also delivered in the same manner. The delivered images include streaming video, videos, still images, etc.

[0111] The authentication unit 55 is a function that authenticates a requester based on an authentication request received by the communication unit 51. The authentication unit 55 authenticates a user based on, for example, whether or not authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches authentication information previously stored. Note that the authentication information may be a card number of an IC card, biometric authentication information such as face, fingerprint, or voiceprint, a device ID, a passcode, an access token, a security key, a ticket, or the like. The authentication unit 55 may also perform authentication using an external authentication system or an authentication method such as OAuth. The authentication unit 55 may also authenticate not only a user but also a device such as an imaging device.

[0112] The communication group management unit 56 manages the entry of the communication terminal 30 and the user into the virtual room, the association of devices, etc. When the authentication unit 55 is successful, the communication group management unit 56 registers the user ID and the IP address of the communication terminal 30 in the virtual room information storage unit 5002, and associates the imaging device 10 with the virtual room.

[0113] The communication control unit 57 manages the start, establishment, and end of communication with the imaging device 10 associated with each virtual room. The communication control unit 57 also manages the start, establishment, and end of communication for delivering wide-field images and audio in response to the communication terminal 30 entering or leaving a virtual room.

[0114] The connection management unit manages the communications (connections) established between the communication terminal 30 and the imaging device 10 and the information processing system 50 in association with the virtual room.

[0115] The API management unit 60 manages the API used when the platform contractor provides an image delivery service of a wide-field image. When using the API, the platform contractor may separately develop software that calls the API. The developed software may run on a server or on a client such as a communication terminal. Any function that the information processing system 50 has, such as the image delivery unit 54, the association processing unit 53, the communication control unit 57, etc., can be provided as an API. In addition, it is also possible to provide a function added to the information processing system 50 later as an API. Whether or not to provide it as an API is determined by a communication terminal operated by the platform provider accessing the information processing system 50 and accepting the public setting of the API, so that the API management unit 60 can control the API based on the public setting. In addition, the API management unit 60 may perform an authentication process to confirm whether the software that requests the API call is software developed by a legitimate platform contractor. The authentication process can be confirmed by comparing information registered and stored in advance as information of the platform contractor in the storage unit 5000 with information transmitted from the requesting software. As a specific example of the authentication process, the information processing system 50 receives from the requesting software an application ID issued in advance by the API management unit 60 for software developed by a platform contractor, and if the API management unit 60 determines that the application ID is stored in the storage unit 5000, the API management unit 60 performs control to permit the provision of the API as legitimate software. On the other hand, if the software cannot be determined as legitimate, the API management unit 60 performs control not to permit the provision of the API. Note that the application ID is an example of authentication information for determining legitimacy, and the API management unit 60 may confirm the legitimacy of the requesting software using authentication information such as an access token, ticket, security key, password, or PIN code issued in advance by the API management unit 60 of the information processing system or an external system.In this embodiment, the form of using the functions of the information processing system 50 as an API is not described, but the processing flow may be the same except that software such as an application developed by a platform subscriber utilizes the functions of the information processing system 50 via judgment in the API management unit 60.

[0116] The storage / reading unit 59 has a function of storing various data in the storage unit 5000 or reading various data from the storage unit 5000. The storage unit 5000 also includes an image management information storage unit 5001, a virtual room information storage unit 5002, and a tenant information storage unit 5003.

[0117] "Image management information storage unit 5001" Fig. 14(a) is a conceptual diagram showing image management information stored in image management information storage unit 5001. Image management information such as that shown in Fig. 14 is stored in image management information storage unit 5001. Image management information is information for managing wide-field images captured in response to an imaging request, and when a user transmits an imaging request from communication terminal 30, one record of image management information is generated. Each item contained in the image management information will be described.

[0118] The data ID is identification information for identifying image data such as a wide-field image. The data ID is assigned by the information processing system 50. ID is an abbreviation for Identification and means identifier or identification information. The ID refers to a name, code, character string, number, or a combination of one or more of these that is used to uniquely distinguish a specific object from multiple objects. Note that the data ID may be associated not only with the wide-field image, but also with an image captured at a normal angle of view by the imaging device 10 associated with the virtual room.

[0119] The data name is the name of the wide-field image set by the user of the communication terminal 30. The data name can be set by the user, but may also be set automatically.

[0120] The imaging date and time information is information for identifying the imaging date and time of a captured image such as a wide-field image, such as the date and time when a user inputs an imaging request to the communication terminal 30, or the date and time when the imaging device 10 captures a captured image such as a wide-field image, etc. The imaging date and time information may be replaced with timestamp information of the captured image such as a wide-field image.

[0121] The image capturer information is identification information (including the user ID and user name) of the user who inputs an image capture request to the communication terminal 30. Since the user inputs an image capture request to the communication terminal 30 while in the virtual room, the user registered in the image capturer information is identified by authentication to the information processing system 50 or the virtual room. The image capturer information is transmitted to the information processing system 50 together with the image capture request. Note that the image capture request and the image capturer information do not necessarily need to be transmitted to the information processing system 50 at the same time, and may be transmitted to the information processing system 50 at different times.

[0122] The imaging device information is identification information (imaging device ID) of the imaging device 10 that captured the wide-field image. The imaging device ID is assigned by the information processing system 50 and shared with the imaging device 10, but information specific to the imaging device 10, such as a MAC address or serial number, may also be used. The imaging device ID is transmitted to the information processing system 50 together with the wide-field image. Note that the imaging device ID and the wide-field image do not necessarily need to be transmitted to the information processing system 50 at the same time, and may be transmitted to the information processing system 50 at different times.

[0123] The viewpoint information of the image taker is viewpoint information specified in the communication terminal 10 of the image taker. For example, the viewpoint information indicates the center coordinates of the wide-field-of-view image being displayed by the communication terminal 30, and is parameter information used to specify a predetermined area of ​​the wide-field-of-view image displayed on the communication terminal 30. Here, the radius vector, polar angle, and azimuth angle are shown as examples of parameter information, but other parameter information described in Figs. 10 to 12 may also be used. The viewpoint information is transmitted from the communication terminal 30 that requests imaging. Note that the viewpoint information may include information specifying the width and height of the display range of the predetermined area. Alternatively, the viewpoint information may be only the width and height of the display range.

[0124] The virtual room ID at the time of imaging is identification information of the virtual room with which the imaging device 10 is associated.

[0125] Data storage location information (storage destination information) is information indicating the location where the wide-field image is stored, such as a URL or a file path. The storage location specified by the storage location information may be information indicating a specified folder. The folder may be a folder associated with the virtual room at the time of image capture. Alternatively, the folder may be associated with identification information (additional information such as a name) indicating one or a combination of two or more classifications such as image capture date and time, image capture device, image capturer, and virtual room at the time of image capture. Alternatively, the data storage location may be specified by combining the data storage location information with information such as a data ID or a data name.

[0126] Fig. 14(b) is also a conceptual diagram showing image management information. In Fig. 14(b), wide-field images with the same virtual room ID at the time of capture are stored. In this way, image management information may be classified by virtual room.

[0127] "Virtual room information storage unit 5002" FIG. 15(a) is a conceptual diagram showing virtual room information stored in virtual room information storage unit 5002. Virtual room information as shown in FIG. 15(a) is stored in virtual room information storage unit 5002. The virtual room information is information about a virtual room, and is stored for each virtual room. Each item of the virtual room information will be described. Note that although the virtual room is registered with a tenant here, registration with a tenant is not essential, and information about a temporarily created virtual room or a shared virtual room is also stored in virtual room information storage unit 5002.

[0128] The virtual room ID is identification information for identifying a virtual room. In this embodiment, it is assumed that a user can create a virtual room at will.

[0129] The virtual room name is a name that allows the user to distinguish the virtual room and can be set by the user. Note that the virtual room ID and the virtual room name may be the same information.

[0130] The device information is identification information (device ID) of a device including the image capture device 10 associated with the virtual room. The currently active user is the user ID of the user currently in the virtual room. This user is a user who can view images such as wide-field images that are distributed to the users in the virtual room. The method of entering the virtual room will be described later. The user ID may also be associated with the IP address of the communication terminal 30 operated by the user. The user ID may also be stored in association with a user name.

[0131] "Tenant information storage unit 5003" FIG. 15(b) is a conceptual diagram showing tenant information stored in the tenant information storage unit 5003. The tenant information storage unit 5003 stores tenant information as shown in FIG. 15(b). The tenant information is information about a tenant (user group) and is held for each tenant. Each item of the tenant information will be described. Note that the tenant information registers various information other than that shown in the figure, such as user information, and FIG. 15(b) shows only a part of the information. The tenant ID is identification information for identifying a tenant. The tenant name is a name that allows users to identify the tenant. Note that the tenant ID and the tenant name may be the same information. The tenant registered virtual room ID is identification information of the virtual room registered to the tenant. Tenant registered devices are information about devices registered with a tenant. The tenant information storage unit, tenant ID, tenant name, tenant registered virtual room ID, and tenant registered device can be rephrased as a user group information storage unit, user group ID, user group name, user group registered virtual room ID, and user group registered device, respectively.

[0132] <Entering a virtual room from a communication device> Next, the process of user B entering a virtual room will be described with reference to Fig. 16 and Fig. 17. It is assumed that the imaging device 10 has already been associated with the virtual room, and the communication terminal 30A has already transmitted a wide-field image and an image with a normal angle of view to the information processing system 50 (association of the imaging device 10 with the virtual room will be described in Fig. 18 and subsequent figures). In the following description, there is no particular distinction between user B entering a virtual room and communication terminal 30B operated by user B entering a virtual room.

[0133] FIG. 16 shows an example of a screen displayed by the communication terminal 30B when user B enters a virtual room. FIG. 16(a) is an example of an entrance screen 200. More specifically, before the entrance screen 200 is displayed, user B logs in to the information processing system 50. By logging in, the tenant to which user B belongs is identified. Virtual rooms are associated with tenants. User B displays a list of virtual rooms associated with the tenants on the communication terminal 30B (see FIG. 21), and selects a virtual room to enter from the list. FIG. 16(a) shows the entrance screen 200 for the virtual room selected by user B in this way. Note that a temporarily created virtual room not associated with a tenant or a shared virtual room may be displayed on the screen of FIG. 16(a).

[0134] Alternatively, the creator of the virtual room may request the information processing system 50 to issue a URL corresponding to the virtual room, and send this URL by email or the like to user B. When user B presses the URL displayed on communication terminal 30B, communication terminal 30B displays entry screen 200 of FIG.

[0135] The entry screen 200 has a virtual room name 201, a participant name input field 202, and an enter button 203. The virtual room name 201 is the same as that stored in the virtual room information storage unit 5002. The participant name input field 202 is a field for inputting a user name to be displayed in the virtual room, and may be a nickname or other name for user B. When user B logs in, a user name linked to user B's user ID may be specified, and this specified user name may be automatically displayed. The enter button 203 is a button for user B to request entry into the virtual room.

[0136] Note that authentication for entering the virtual room may be required at the time of entry, separately from logging in to the tenant.

[0137] FIG. 16(b) shows an image viewing screen 210 displayed by the communication terminal 30B when the user B enters the virtual room. In the image viewing screen 210 of FIG. 16(b), the imaging device 10 has already started distributing a wide-field-of-view image via the information processing system 50, and the communication terminal 30A has already started distributing an image with a normal field of view. Therefore, the image viewing screen 210 has a first image field 211 and a second image field 212. The wide-field-of-view image is displayed in the first image field 211, and the image with a normal field of view is displayed in the second image field 212. If there are three or more locations transmitting images, the image viewing screen 210 is divided according to the number of locations from which the images are transmitted.

[0138] A wide-field-of-view image mark 213 is displayed in the first image field 211. The wide-field-of-view image mark 213 is set by the screen generation unit 52 of the information processing system 50 when it determines that the image to be displayed in the first image field 211 is a wide-field-of-view image. The communication terminal 30B may determine and display the mark. By looking at the wide-field-of-view image mark 213, user B knows that a wide-field-of-view image with a changeable viewpoint is being distributed. In addition, a device name 214 (transmitted from the imaging device 10 together with the wide-field-of-view image) is displayed in the first image field 211. The device name 214 is information set by user A or the like, as described below (see FIG. 19).

[0139] A participant name 215 is displayed in the second image field 212. The participant name 215 is a user name, and the participant name input field 202 displays the participant name of a user who has already entered the room (in this case, since user A has already entered the room, "AAA" entered by user A in the participant name input field 202).

[0140] FIG. 17 is a sequence diagram illustrating a process in which user B (or communication terminal 30B) enters a virtual room.

[0141] S1: First, user B at site B performs an operation to display a list screen of virtual rooms. Note that the communication terminal 30B has previously accessed the information processing system 50 in response to an operation by user B, and has received information on the virtual rooms stored in the virtual room information storage unit 5002 from the information processing system 50 for displaying the list screen of the virtual rooms. At this time, the communication terminal 30B may be authenticated by the authentication unit 55 of the information processing system 50 by transmitting authentication information required for login or the like to the information processing system 50. The authentication information may be authentication information associated with user B or authentication information associated with the communication terminal 30B. In such a case, the virtual room displayed on the list screen may be a virtual room registered in a tenant associated with user B or a virtual room registered in a tenant associated with the communication terminal 30B. When the reception unit 32 receives an operation to display the list screen, the display control unit 33 of the communication terminal 30B displays a selection screen on the display 306.

[0142] S2: When user B selects a selection button for a certain virtual room, the reception unit 32 of the communication terminal 30B receives the selection of the virtual room. The display control unit 33 of the communication terminal 30B causes the display 306 to display the entry screen 200 shown in FIG. 16(a).

[0143] S3: User B enters the necessary information and presses the enter button 203. When the reception unit 32 receives the press, the communication unit 31 of the communication terminal 30B transmits a request to enter the virtual room to the information processing system 50. This entry request includes information such as the virtual room ID indicating the virtual room selected in step S2, the user ID of user B authenticated by logging in or the like, and the IP address of the communication terminal 30B that is the request source terminal.

[0144] S4: As a result, the communication unit 51 of the information processing system 50 receives the entry request. The communication group management unit 56 registers the user ID and IP address authenticated by login or the like in the virtual room information specified by the virtual room ID in the virtual room information storage unit 5002.

[0145] S5: Then, the communication unit 51 of the information processing system 50 transmits a response indicating that the user has entered the room to the communication terminal 30B. As a result, the communication unit 31 of the communication terminal 30B receives the response indicating that the user has entered the room. Following S5, the display control unit 33 of the communication terminal 30B receives information on the screen generated by the screen generation unit 52 of the information processing system 50 and information on the image distributed by the image distribution unit 54, and displays the image viewing screen 210 shown in Fig. 16(b) based on the received information.

[0146] <Associating imaging devices with rooms> Next, association of the imaging device 10 with a virtual room will be described with reference to Fig. 18 to Fig. 25. Note that, although the association of the imaging device 10 with a virtual room will be described as being performed by user A at site A, it may be performed by a system administrator, a tenant administrator, or the like.

[0147] 18 is an example of a device registration screen 220 displayed by the communication terminal 30A. User A has been authenticated by logging in to the information processing system 50, for example. By logging in, the tenant to which user A belongs is identified. User A requests the information processing system 50 to display the device registration screen 220, and the communication terminal 30A displays the device registration screen 220 received from the information processing system 50. First, the device is registered in the tenant using the device registration screen 220.

[0148] The device registration screen 220 has an imaging device registration button 221, a VR goggles registration button 222, and a smart glasses registration button 223. Buttons are provided for each type of device because there are differences in the presence or absence of a camera, the information used for registration, etc. In addition, since devices are registered by device type, the information processing system 50 can also grasp the type of device.

[0149] The imaging device registration button 221 is a button for user A to register the imaging device 10, the VR goggles registration button 222 is a button for registering the VR goggles 89, and the smart glasses registration button 223 is a button for registering the smart glasses 88.

[0150] Fig. 19 is an example of a screen displayed when the imaging device registration button 221 is pressed. Fig. 19(a) shows an imaging device registration dialogue 230. The imaging device registration dialogue 230 has a name field 231 for the imaging device 10, a description field 232, and a next button 233. User A sets an arbitrary name in the name field 231 for the imaging device 10 so that it is clear that this is the imaging device 10 to be registered, and sets a description in the description field 232.

[0151] When user A presses the next button 233, the communication terminal 30A requests the two-dimensional code from the information processing system 50, and the communication terminal 30A displays the two-dimensional code.

[0152] 19(b) is an example of a two-dimensional code screen 240 displayed by the communication terminal 30A. The two-dimensional code screen 240 has a message 241 saying "Please scan the following two-dimensional code to register a device called XX (the name entered in the name field)" and a two-dimensional code 242. User A captures the two-dimensional code 242 with the imaging device 10 that he / she wishes to register. The two-dimensional code 242 contains authentication information required for registration, such as a URL that the imaging device 10 connects to for registering itself, and a temporary ID and password.

[0153] When user A captures an image of the two-dimensional code 242 with the imaging device 10, the imaging device 10 connects to a URL and is authenticated with a temporary ID and password. If the authentication is successful, a formal imaging device ID is exchanged, and the name, description, and imaging device ID of the imaging device 10 are registered with the tenant. The imaging device 10 also holds this imaging device ID, name, and description. The imaging device 10 registered with the tenant is associated with a virtual room in response to an operation by user A, which will be described later. Note that the two-dimensional code 242 is an example of code information, and any other form of code, such as a barcode, may be used as long as similar authentication information is embedded therein.

[0154] Next, an example of a method for registering a communication terminal such as a VR goggle 89 or a smart glass 88 to a tenant will be described with reference to Fig. 20. Fig. 20 shows an example of a VR goggle registration screen 250 that is displayed when the VR goggle registration button 222 is pressed. The VR goggle registration screen 250 has a temporary code input field 251 and a secret input field 252.

[0155] If the VR goggles 89 do not have a camera, the two-dimensional code cannot be captured. For this reason, the user A may cause the VR goggles 89 to output (display) a temporary code (temporary ID) and a secret (password), and input them into the temporary code input field 251 and the secret input field 252. The communication terminal 30A registers the VR goggles 89 with the tenant by transmitting the temporary code and secret to the information processing system 50. The VR goggles 89 connect to the information processing system 50 and are authenticated by transmitting the temporary code and secret. If the authentication is successful, a formal VR goggle ID is exchanged, and the VR goggle ID is registered with the tenant. The VR goggles 89 also hold this VR goggle ID. The VR goggles 89 registered with the tenant are associated with a virtual room in response to an operation by the user A, which will be described later. The smart glasses 88 will be described in detail, but the user A can register them in the same way as the imaging device 10 or the VR goggles 89. The temporary code and secret are examples of authentication information, and other information may be used as the authentication information. The imaging device ID, the VR goggles ID, and the smart glasses ID are each an example of a device ID, and can be referred to as a device ID. Therefore, when registering a device other than the imaging device 10, the VR goggles, or the smart glasses, the device ID can be used to associate the device with a virtual room or a tenant by a similar procedure. The device ID may be identification information linked to the owner of the device.

[0156] FIG. 21 shows an example of a virtual room association screen (part 1) 260 for associating the imaging device 10 with a virtual room. The screen configuration may be the same for the VR goggles 89 and the smart glasses 88. The virtual room association screen (part 1) 260 has a list 261 of virtual rooms. The list 261 of virtual rooms displays individual virtual room fields 262 to 264 based on the virtual rooms created in the tenant. Each of the individual virtual room fields 262 to 264 has a link issue button 265, an entry button 266, a setting button 267, and a virtual room name 268. The link issue button 265 is a button for issuing a link (URL for invitation) and a passcode to the virtual room. The entry button 266 is a button for user A to enter the virtual room. The setting button 267 is a button for associating the imaging device 10 with the virtual room. The virtual room name 268 is the same as that stored in the virtual room information storage unit 5002. Therefore, user A presses set button 267. By pressing set button 267, communication terminal 30A displays virtual room association screen (part 2) 270.

[0157] Furthermore, if a device has already been associated with a virtual room, the name 269 of the device is displayed in the individual virtual room field (individual virtual room field 264 in the figure).

[0158] 22 shows an example of virtual room matching screen (part 2) 270. Note that virtual room matching screen (part 2) 270 is displayed as a pop-up on virtual room matching screen (part 1) 260. A screen transition from virtual room matching screen (part 1) 260 to virtual room matching screen (part 2) 270 does not go through information processing system 50, but a screen transition that goes through information processing system 50 is also possible.

[0159] The virtual room association screen (part 2) 270 has a name 271 (not registered yet and therefore not registered in the figure) of the imaging device 10 currently (already) associated with the virtual room, a connection button 272, and a storage button 273. The connection button 272 is a button for displaying a list of devices registered in the tenant as candidates for association in order to associate a device with the virtual room. The storage button 273 is a button for displaying a list of storages 90 for storing wide-field images and normal-angle images captured by the imaging device 10 associated with the virtual room. The list of storages 90 may include not only a list of storages 90 associated with the virtual room, but also a list of specific storage locations such as folders on the storage 90. The user can associate the storage 90 with the virtual room by selecting a specific storage 90 or a specific storage location such as a folder on the storage 90. The information of the storage 90 associated in this way (address information for accessing the storage 90 and a storage location such as a folder on the storage 90) can be stored in association with the virtual room ID in the virtual room information storage unit 5002. When the connect button 272 is pressed, the communication terminal 30A displays the virtual room association screen (third).

[0160] The communication terminal 30A sends a virtual room ID to the information processing system 50 and obtains the names of devices (including device IDs, etc.) registered in the tenant in which the virtual room is generated, and the names of devices (including device IDs, etc.) associated with the virtual room.

[0161] FIG. 23 shows an example of a virtual room association screen (part 3) 280. The virtual room association screen (part 3) 280 has a name 281 of the imaging device 10 currently (already) associated with the virtual room, a list of devices that can be added 282, and a save button 283. The user A selects a device that he / she wishes to additionally associate with the virtual room from the list of devices that can be added 282, and presses the save button 283. This associates the device with the virtual room (a device ID such as an imaging device ID is registered in the virtual room information storage unit 5002). Note that, as shown in FIG. 23, the number of imaging devices that can be associated with the virtual room may be limited. For example, when the upper limit is two, the remaining number of devices that can be additionally registered may be displayed on the virtual room association screen (part 3) by referring to the number of imaging device IDs already registered in the virtual room information storage unit 5002.

[0162] <Wide-field image transmission start process for imaging device> In the above manner, devices such as the image capture device 10 are associated with the virtual room, but user A needs to operate the device to start transmitting images.

[0163] For the VR goggles 89 and smart glasses 88, user A turns on and off the transmission of images by operating the device itself. This is because currently, a dedicated application is not running in the communication system 1 for the VR goggles 89 and smart glasses 88. If a dedicated application runs in the communication system 1 for the VR goggles 89 and smart glasses 88 as well, user A can remotely turn on and off the transmission of images.

[0164] In the case of the imaging device 10, if the application is enabled, user A can enter the virtual room and turn on and off the transmission of wide-field images from the menu.

[0165] FIG. 24 is an example of a wide-field-of-view image transmission control dialogue 290 displayed by the communication terminal 30A. The wide-field-of-view image transmission control dialogue 290 is displayed as a pop-up on the image viewing screen 210. It is assumed that the user A operates the communication terminal 30A to enter a virtual room to which the imaging device 10 is associated. The wide-field-of-view image transmission control dialogue 290 displays a name 292 of the imaging device 10 associated with this virtual room. A toggle button 291 is displayed near the name 292, and the user A can operate the toggle button 291 to set the transmission of the wide-field-of-view image by the imaging device 10 to ON (start transmission) or OFF (stop transmission). Note that the method of setting ON or OFF using the toggle button is one example, and it is sufficient as long as it can be set according to the input of a user operation. For example, it may be set by selecting a radio button or a predetermined icon, operating a menu, or the like. In addition, it may be possible to automatically start transmitting the wide-field-of-view image after the imaging device 10 enters the room without requiring a user operation. In addition, certain conditions, such as the date and time, the number of users who have entered, or the participation of a specific user, may be determined in advance, and the transmission of the wide-field image may begin when it is determined that the conditions are met.

[0166] The communications terminal 30A transmits setting information for transmission control set by operating the toggle button 291 to the information processing system 50. The information processing system 50 transmits a transmission start request or a transmission stop request to the imaging device 10 in accordance with the setting information for transmission control.

[0167] Fig. 24(a) shows a state where the toggle button 291 is set to OFF. For this reason, no wide-field image is displayed in Fig. 24(a). On the other hand, in Fig. 24(a), when the communication terminal 30A enters the room, an image with a normal angle of view captured by the camera 9 of the communication terminal 30A has already been shared and is displayed on the image viewing screen 210.

[0168] 24(b) shows a state in which the toggle button 291 is set to ON. When the toggle button 291 is set to ON, the information processing system 50 transmits a transmission start request to the imaging device 10, and the imaging device 10 starts transmitting the wide-field image. As a result, two images are shared in one virtual room, and the image viewing screen 210 is divided into two. Furthermore, when the setting is changed from ON to OFF, the communication terminal 30A transmits setting information for OFF, and the information processing system 50 transmits a transmission stop request to the imaging device 10 in response to receiving the setting information for OFF, and the imaging device 10 stops transmitting the wide-field image.

[0169] As described in FIG. 25, even if the user is at the site, the imaging device 10 can be associated with the virtual room by a simple operation of capturing the code information with the imaging device 10. Since the user at the site may not have a PC or the like, it is particularly useful for the user at the site to be able to perform the association process on the spot with only the code information issued in advance and the imaging device 10. Furthermore, if the association process is performed in advance, the user can connect the imaging device 10 to a specific virtual room without having to select a virtual room, and the start or stop of transmission can also be instructed from the remote location, thereby reducing the burden on the user who wants to concentrate on the work at the site. Therefore, it is possible to provide a system that allows efficient communication between the site and the remote location even during the advance preparation process.

[0170] <<How to register an imaging device in a virtual room>> Next, a procedure for registering the imaging device 10 in the virtual room described in the series of screen transitions in Fig. 18 to Fig. 24 will be described with reference to Fig. 25. Fig. 25 is an example of a sequence diagram showing a procedure in which user A registers the imaging device 10 in the virtual room.

[0171] S11: First, user A connects communication terminal 30A to information processing system 50, inputs authentication information (user ID, password, etc.), and requests to log in. Accepting unit 32 of communication terminal 30A accepts the operation.

[0172] S12: The communication unit 31 of the communication terminal 30A specifies authentication information and transmits a login request to the information processing system 50. The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 55 performs authentication based on the specified authentication information. Here, it is assumed that the authentication is successful. At this time, the information processing system 50 can specify the tenant ID associated with the authenticated user ID by referring to the tenant information storage unit 5003.

[0173] S13: In response to a user operation, the screen generation unit 52 of the information processing system 50 generates the device registration screen 220, and the communication unit 51 transmits screen information of the device registration screen 220 to the communication terminal 30A.

[0174] S14: The communication unit 31 of the communication terminal 30A receives the screen information of the device registration screen 220, and the display control unit 33 displays the device registration screen 220 shown in Fig. 18. User A selects the type of device (here, it is assumed that the imaging device 10 (e.g., omnidirectional camera) is selected), and then inputs the name and description of the imaging device 10 as shown in Fig. 19. The reception unit 32 receives the input.

[0175] S15: The communication unit 31 of the communication terminal 30A transmits a request for code information (for example, a two-dimensional code) to the information processing system 50, specifying the name and description input by the user A.

[0176] S16: The communication unit 51 of the information processing system 50 receives a request for code information (e.g., a two-dimensional code). The communication group management unit 56 generates a URL (a connection destination for registration) in association with the name and description, and generates code information (e.g., a two-dimensional code) including the URL, a temporary ID, and a password. The communication unit 51 of the information processing system 50 transmits the code information (e.g., a two-dimensional code) to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the code information (e.g., a two-dimensional code), and the display control unit 33 displays the code information (e.g., a two-dimensional code) as shown in FIG. 19.

[0177] S17: Next, the user A operates the imaging device 10 that he / she wishes to associate with the virtual room, and captures an image of code information (for example, a two-dimensional code). The reception unit 12 of the imaging device 10 receives the operation.

[0178] S18: The imaging processing unit 13 of the imaging device 10 performs imaging processing on an imaging target including code information (e.g., a two-dimensional code) to generate image data, and the analysis unit 14 analyzes the image data to extract a URL, a temporary ID, and a password. As a result, the registration request unit 15 connects to the URL via the connection unit 16, and transmits a registration request for the imaging device 10 to the information processing system 50 by specifying the temporary ID and password. Note that when implementing the registration method using the registration screen described in FIG. 20, the imaging device 10 is replaced with a communication terminal such as VR goggles 89 or smart glasses 88, and steps S15 to S17 can be omitted since the code information is not captured.

[0179] S19: The communication unit 51 of the information processing system 50 receives the temporary ID and password, and judges whether or not they match the temporary ID and password associated with the URL to which the authentication unit 55 is connected. Here, it is assumed that they match.

[0180] S20: Since the registration of the imaging device 10 is requested, the communication group management unit 56 of the information processing system 50 generates an imaging device ID as an example of a device ID, and registers the imaging device ID in the tenant corresponding to the tenant ID identified when the user A logs in. Note that a name and a description are associated with the imaging device ID. Specifically, the communication group management unit 56 refers to the tenant information storage unit 5003, and adds and registers the imaging device ID to the tenant registered device associated with the identified tenant ID. Note that, although the communication group management unit 56 generates and registers the imaging device ID here, the imaging device ID received from the imaging device 10 may be registered. Note that, when a communication terminal such as the VR goggles 89 or the smart glasses 88 is to be registered in the tenant instead of the imaging device 10, the corresponding device ID can be registered in the tenant information storage unit 5003 in a similar procedure.

[0181] S21: The communication unit 51 of the information processing system 50 transmits the imaging device ID to the imaging device 10. The connection unit 16 of the imaging device 10 receives the imaging device ID and stores it in the storage unit 1000.

[0182] S22: The communication terminal 30A is notified of the completion of registration from the communication unit 51 of the information processing system 50, and thus the user A can start associating the imaging device 10 with the virtual room. The user A causes the communication terminal 30A to display the virtual room association screen (part 1) 260, and selects the virtual room to which the imaging device 10 registered in the tenant is to be associated. The reception unit 32 of the communication terminal 30A accepts an operation input indicating the selection. Specifically, the reception unit 32 of the communication terminal 30A accepts an operation input from the user A, and the display control unit 33 causes the virtual room association screen (part 1) 260 to be displayed. At this time, the communication unit 31 may transmit a screen update request to the communication unit 57 of the information processing system 50. When the information processing system 50 receives the update request, it refers to the tenant information storage unit 5003 and identifies the virtual room ID registered in the tenant associated with the authenticated user ID. It then refers to the virtual room information storage unit 5002 and acquires the virtual room name associated with the identified virtual room ID. The communication unit 51 of the information processing system 50 transmits information on the virtual room ID and the corresponding virtual room name thus identified (which may be information on a screen generated by the screen generation unit 52 based on this information) to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the information on the virtual room ID and the virtual room name, and the display control unit 33 can update and display the virtual room matching screen (type 1) 260 based on the received information. Note that such information can be identified based on the user ID, and therefore may be received in S13 after authentication. The reception unit 32 receives an operation input indicating a selection from user A on the virtual room matching screen (type 1) 260 displayed in this manner, whereby the communication terminal 30A can identify the selected virtual room ID.

[0183] S23: Next, user A causes communication terminal 30A to display virtual room matching screen (part 2) 270, and presses connect button 272 to additionally match the device to the virtual room. Reception unit 32 of communication terminal 30A receives an operation input indicating the press. Specifically, display control unit 33 of communication terminal 30A displays virtual room matching screen (part 2) 270 corresponding to the selected virtual room ID identified in S22. Furthermore, reception unit 32 receives an instruction from user A to additionally match the device to the virtual room (pressing connect button 272).

[0184] S24: In response to the operation input in S23, the communication unit 31 of the communication terminal 30A requests, from the information processing system 50, information on devices registered with the tenant that are candidates for devices to be associated with the virtual room, and information on devices that are already associated with the virtual room ID selected in step S22.

[0185] S25: The communication unit 51 of the information processing system 50 receives a request for information on the devices registered with the tenant and the devices associated with the selected virtual room ID, and the screen generation unit 52 generates a virtual room matching screen (third) 280 including the device IDs of the devices registered with the tenant and the devices associated with the selected virtual room ID. The communication unit 51 of the information processing system 50 transmits screen information of the virtual room matching screen (third) 280 to the communication terminal 30A.

[0186] S26: The communication unit 31 of the communication terminal 30A receives the screen information of the virtual room matching screen (part 3) 280, and the display control unit 33 displays the virtual room matching screen (part 3) 280. The user A selects a device (here, the imaging device 10 is described as an example) to be matched with the virtual room. The reception unit 32 of the communication terminal 30A receives the selection, and the imaging device ID is specified as the device ID of the selected device.

[0187] S27: The communication unit 31 of the communication terminal 30A transmits an association request to the information processing system 50, specifying the virtual room ID selected in step S22 and the device ID (eg, the imaging device ID) selected in S26.

[0188] S28: The communication unit 51 of the information processing system 50 receives the association request, and the communication group management unit 56 registers the device (e.g., the imaging device 10) in the virtual room. That is, the communication group management unit 56 refers to the virtual room information storage unit 5002, and registers the device ID (e.g., the imaging device ID) in association with the virtual room ID designated in the request of S27.

[0189] S29: Since the device ID (for example, the imaging device ID) is associated with the virtual room, the communication unit 51 of the information processing system 50 transmits the virtual room ID, name, and description to the imaging device 10. The information processing system 50 may use push notification, or the imaging device 10 may transmit by polling. The connection unit 16 of the imaging device 10 receives the virtual room ID, name, and description, and stores them in the storage unit 1000. Thereby, when the imaging device 10 transmits a wide-field image, the imaging device ID, virtual room ID, name, description, and the like can be added. Devices other than the imaging device 10 can also be associated with the virtual room in a similar procedure. Furthermore, the communication unit 51 of the information processing system 50 may continue to transmit a notification indicating the completion of the association to the communication terminal 30A. After this step, the device (imaging device 10) registered with the virtual room can be connected to the associated virtual room. Here, we will continue the explanation assuming that the imaging device 10 has connected to the virtual room by sending a connection request to the virtual room by specifying the virtual room ID received in S29 to the information processing system 50, but the timing at which the imaging device 10 connects to the virtual room can be changed by user operation.

[0190] S30: The communication terminal 30A and the information processing system 50 perform the room entry process described with reference to FIG. 17, thereby allowing the communication terminal 30A to enter the virtual room in which the device (imaging device 10) has been associated.

[0191] S31: After entering the virtual room, user A turns on the toggle button 291 of the imaging device 10 associated with the virtual room on the image viewing screen 210. The reception unit 32 of the communication terminal 30A receives the on-state.

[0192] S32: The communication unit 31 of the communication terminal 30A designates the device ID (imaging device ID) and transmits a request to start transmitting the wide-field image to the information processing system 50. Note that user A may directly operate a button on the imaging device 10 to start transmitting the wide-field image. Note that user A's operation may also cause the communication unit 31 of the communication terminal 30A to transmit a transmission stop request to the information processing system 50.

[0193] S33: The communication unit 51 of the information processing system 50 receives the transmission start request and requests the imaging device 10 identified by the device ID (imaging device ID) to start transmission. The information processing system 50 may use push notification, or the imaging device 10 may use polling. The connection unit 16 of the imaging device 10 receives the transmission start request, and the imaging processing unit 13 starts imaging. The image transmission control unit 18 repeats transmission of the wide-field image via the connection unit 16 at a determined FPS or an FPS according to the bandwidth. Therefore, the communication terminal 30 that has entered the virtual room can display the situation at site A on the image viewing screen 210 in real time.

[0194] <Distribution of wide-field images, etc.> With reference to FIG. 26, a flow in which a wide-field image or an image with a normal angle of view is shared will be described. FIG. 26 is an example of a sequence diagram illustrating a flow in which a wide-field image is shared. In FIG. 26, the communication terminals 30A and 30B execute the entry process described in FIG. 17 and have entered the virtual room. In addition, the communication terminal 30A has a camera 9 with a normal angle of view, which is shared with the communication terminal 30B. Images captured by smart glasses 88 associated with the virtual room, instead of the camera 9 of the communication terminal 30A, may be shared. In addition, in FIG. 26, the imaging device 10 has already been connected to the same virtual room by the registration procedure described in FIG. 25.

[0195] S41: The imaging unit 34 of the communication terminal 30A captures an image, and the communication unit 31 transmits to the information processing system 50 video and audio including the captured image by specifying the virtual room ID of the entered room.

[0196] S42, S43: When the communication unit 51 of the information processing system 50 receives the video and audio including the image, the image distribution unit 54 acquires the IP addresses of the communication terminals 30A, 30B that have entered the same virtual room from the virtual room information storage unit 5002, and transmits the video and audio including the image via the communication unit 51. Note that in Fig. 26, the communication unit 31 of the communication terminal 30A receives and displays an image with a normal angle of view from the information processing system 50, but the image with a normal angle of view captured by the imaging unit 34 may be displayed without receiving it.

[0197] S44: Next, in response to a transmission start request based on the transmission start setting of the imaging device 10, the imaging processing unit 13 captures a wide-field image, and the image transmission control unit 18 specifies the virtual room ID, imaging device ID, name, and description in which the imaging device 10 is registered, via the connection unit 16, and transmits video and audio including the wide-field image to the information processing system 50.

[0198] S45, S46: When the communication unit 51 of the information processing system 50 receives video and audio including a wide-field image, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A, 30B that have entered the same virtual room from the virtual room information storage unit 5002, and transmits the video and audio including the wide-field image via the communication unit 51.

[0199] S47: Next, the communication terminal 30C equipped with the camera 9 executes the entry process described with reference to FIG. 17 and thereby newly enters the virtual room.

[0200] S48: The communication unit 31 of the communication terminal 30C transmits to the information processing system 50 video and audio including an image with a normal angle of view.

[0201] S49-S51: The communication unit 51 of the information processing system 50 receives video and audio including an image with a normal angle of view from the communication terminal 30C, obtains the IP addresses of the communication terminals 30A-30C that have entered the same virtual room from the virtual room information storage unit 5002, and the image distribution unit 54 transmits the video and audio including the image with the normal angle of view.

[0202] S52: Furthermore, the communication unit 51 of the information processing system 50 also transmits video and audio including the wide-field image to the communication terminal 30C that has entered the same virtual room.

[0203] In this way, users A and B who enter the same virtual room can share in real time a video including a wide-field image captured by the imaging device 10 associated with the virtual room. Note that the order of transmission of each image shown in Fig. 26 is an example, and the wide-field image may be shared first, or the normal angle of view image may be shared first.

[0204] A supplementary explanation about the smart glasses 88 and the VR goggles 89. The smart glasses 88 have a camera with a normal angle of view and a display function. An image with a normal angle of view captured by a camera held by the smart glasses 88 is distributed in the same manner as the cameras 8 and 9. The display function held by the smart glasses 88 is flat like a normal display, so a part of a wide-field image is displayed at a viewpoint specified by the user. The VR goggles 89 have a display function (may further have a camera with a normal angle of view). The display function held by the smart glasses 88 projects a wide-field image with a viewpoint determined by the direction of the user's head, so a part of the wide-field image is displayed at a viewpoint according to the direction of the user's head. While viewing a wide-field image with the smart glasses 88 or the VR goggles 89, the user can transmit an imaging request specifying the viewpoint information being viewed to the information processing system 50.

[0205] <Capturing images with an imaging device according to instructions from a communication terminal> Next, a process in which the imaging device 10 captures an image in response to an instruction from any communication terminal 30 will be described with reference to FIGS.

[0206] First, a screen will be described when a user operates the communication terminal 30 to request imaging from the imaging device 10 when a wide-field image is shared. Any user can instruct imaging, but here it is assumed that user B instructs imaging. Since a wide-field image is being shared, strictly speaking the imaging device 10 is already "imaging" in real time. An imaging request when a wide-field image is shared means capturing a wide-field image for storage. Simply put, when a wide-field image is being distributed in real time, the wide-field image is not stored, and therefore cannot be checked later (a scene is not arbitrarily stored).

[0207] FIG. 27 is an example of an image viewing screen 400 displayed by communication terminal 30B. In the explanation of FIG. 27, the difference from is mainly explained. Communication terminals 30A and 30B each display an image including a wide-field image distributed in real time, and communication terminals 30A and 30B also display an image including an image with a normal angle of view captured by camera 9 of communication terminal 30A. The image of the wide-field image can be changed by arbitrarily changing the virtual viewpoint by the operation of the user at each base, thereby changing a predetermined area of ​​the displayed wide-field image. Therefore, it is possible to check the situation at the site to an extent that cannot be checked with a normal angle of view.

[0208] As shown in FIG. 27, an image capture button 401 is displayed in the first image field 211 in which the wide-field image is displayed. The image capture button 401 is a button for the communication terminal 30B to transmit an image capture request to the image capture device 10 by using viewpoint information for identifying a predetermined area of ​​a video including a wide-field image currently displayed in the first image field 211. Note that, in FIG. 27, the image capture button 401 is given a description such as "take a 360-degree photo", but this is an example of a button for capturing a still image of a celestial sphere image, and may be a button for capturing a video instead of a photo or a button for instructing recording. Also, it may be a button for instructing capturing a wide-field image other than a celestial sphere image. The image capture button 401 may be any GUI component capable of instructing capturing an image, and is not limited to the example of FIG. 27.

[0209] FIG. 28 shows an image viewing screen 400 on which a download button 402 is displayed by pressing an image capture button 401 on the communication terminal 30B. In FIG. 28, the download button 402 (an example of a display component) is displayed below the first image field 211. The download button 402 displays a message saying "Download captured still image." The download button 402 is embedded with storage destination information (e.g., URL) to which a wide-field-of-view image captured by the image capture device 10 in response to an image capture request transmitted by pressing the image capture button 401 is uploaded. When the user B presses the download button 402, the communication terminal 30B connects to the URL and can download the wide-field-of-view image. Since the wide-field-of-view image is associated with viewpoint information, when the communication terminal 30B displays the downloaded wide-field-of-view image, a virtual viewpoint is set so that a predetermined area of ​​the wide-field-of-view image specified by the viewpoint information matches the center of the first image field 211 and displayed. Note that it is not necessary to completely match the viewpoint with the center, and the viewpoint may be set to be included in a range near the center of the first image field 211 and displayed. Although an example of displaying a downloaded wide-field image in the first image field 211 in which a video including a wide-field image distributed in real time was displayed has been described here, the first image field 211 may continue to display a video including a wide-field image distributed in real time, and a new image field may be added to the image viewing screen 400, and the downloaded wide-field image may be displayed in the new image field. In this way, it is possible to check the change in the situation at the site through the video of the wide-field image distributed in real time, while simultaneously checking a wide-field image capturing a specific situation at the site.

[0210] The above-described form of the download button 402 is merely an example, and the download button 402 may display a message such as "Spherical image URL." Alternatively, instead of a button, a link corresponding to destination information (e.g., a URL) may be displayed. The user may be able to download the image in the same way by clicking the link.

[0211] Furthermore, the communication terminal 30B may automatically receive and display the stored wide-field image and associated viewpoint information without displaying the download button 402.

[0212] 28 shows an image viewing screen 400 displayed by the communication terminal 30B which has issued an image capturing request, but the download button 402 may also be displayed on the image viewing screen 400 displayed by the communication terminal 30A which is displaying a video including a wide-field-of-view image distributed in real time. As one embodiment, the download button 402 may be displayed on the communication terminal 30A of the participant who has entered the same virtual room in response to an operation by user B to instruct sharing of a wide-field-of-view image captured in response to an image capturing request. In this way, the imager (user B) who has instructed the image capturing can share the image with other participants after checking the image captured at hand, thereby preventing a wide-field-of-view image captured by mistake or a wide-field-of-view image that needs to be shared from being shared. However, all communication terminals 30 which have entered the virtual room may automatically display the download button 402 in response to saving a wide-field-of-view image.

[0213] With reference to Fig. 29, a description will be given assuming that communication terminal 30A displays download button 402. Fig. 29(a) is an example of an image viewing screen 410 that is displayed before user A presses download button 402. Before user A presses download button 402, communication terminal 30A displays in first image field 211 a wide-field image with an angle of view that includes, for example, a work machine equipped with a drill as an arbitrary virtual viewpoint designated by user A. In this state, user A presses download button 402.

[0214] FIG. 29(b) is an example of an image viewing screen 420 displayed after user A presses the download button 402. Since the viewpoint information when user B presses the image capture button 401 is the first image field 211 in FIG. 27, the downloaded wide-field image is displayed in the first image field 211 of the communication terminal 30A from the same viewpoint as the first image field 211 in FIG. 27, as shown in FIG. 29(b). Therefore, the wide-field image is displayed with an angle of view that allows a scene of lifting building materials with a crane from the roof of a building to be confirmed, rather than a work machine equipped with a drill. In addition, user A can arbitrarily change the virtual viewpoint for the downloaded wide-field image displayed in the first image field 211, just like the video of the wide-field image delivered in real time. Therefore, the viewpoint of the user who requested the image capture can be reflected as an initial value even in the wide-field image that cuts out a certain scene, and the specific scene of the site can be confirmed to a range that cannot be confirmed with a normal angle of view.

[0215] In this way, users at different locations can share a virtual viewpoint of a wide-field image of a specific scene later. The wide-field image saved in response to a save request may be displayed as a pop-up instead of in the first image field, or in a separate window. In this way, users can check the situation at the site that changes over time with the wide-field image video delivered in real time, and can use the wide-field image of a specific scene to improve the efficiency of their communication.

[0216] As shown in FIG. 29(b), while the communication terminal 30A is displaying the downloaded wide-field image, the imaging button 401 is not displayed. This is because the real-time wide-field image is not displayed in the first image field 211. Also, the device name 214 in the first image field 211 displays the name of the imaging device 10 and that the image is a shared image. The device name 214 can be identified from the information of the tenant registered device stored in the tenant information storage unit 5003. Also, the device name 214 may be a device ID.

[0217] Furthermore, in the communication terminals 30A and 30B on which the download button 402 is displayed, the users A and B can arbitrarily delete the download button 402.

[0218] 30, together with the download button 402 (serving also as the download button 402), the communication terminals 30A and 30B may display a thumbnail image 404 of the wide-field image. The thumbnail image 404 may be created by the information processing system 50 or the communication terminal 30. The viewpoint of the thumbnail image 404 is determined by viewpoint information.

[0219] Furthermore, it is preferable that each of users A and B can set a data name (registered in the image management information storage unit 3001 of each communication terminal 30) for the wide-field image captured in response to an imaging request.

[0220] Fig. 31 shows an example of an image viewing screen 430 in the case where there are three image columns. For example, the image viewing screen 430 in Fig. 31 displays a first image column 211 in which a wide-field image captured by the imaging device 10 is displayed, a second image column 212 in which an image with a normal angle of view captured by the camera 9 of the communication terminal 30A is displayed, and a third image column 431 in which an image with a normal angle of view captured by the camera 8 of the communication terminal 30C is displayed. Any user among users A, B, and C can press the imaging button, and users A, B, and C at each location can share a wide-field image from the same viewpoint.

[0221] <<Action or processing in response to imaging requests when sharing wide-field images>> Next, with reference to Fig. 32, an operation or process in response to an imaging request of the communication system 1 when a wide-field-of-view image is shared will be described. Fig. 32 is an example of a sequence diagram illustrating a process in which user B requests imaging from the imaging device 10 via the communication terminal 30 when a wide-field-of-view image is shared. In the description of Fig. 32, it is assumed that the communication terminals 30A and 30B have entered the virtual room by the entry process described in Fig. 17. In addition, in the description of Fig. 32, a case is described in which user B presses the imaging button 401 to share viewpoint information with user A, but user A can also press the imaging button 401 to share viewpoint information with user B. In addition, in Fig. 32, the imaging device 10 has already been connected to the same virtual room by the registration procedure described in Fig. 25.

[0222] S61: For example, on the image viewing screen 400, the user B performs an operation to move (change) the virtual viewpoint of the wide-field-of-view image included in the video being distributed in real time from the imaging device 10 via the information processing system 50. The reception unit 32 of the communication terminal 30B receives an operation input indicating a viewpoint movement (change) operation, and the display control unit 33 displays in the first image column 211 a predetermined area of ​​the wide-field-of-view image identified based on the viewpoint information after the movement (change) as a display range according to the viewpoint after the movement (change).

[0223] S62: Next, user B, who wants to save a wide-field-of-view image captured from a specific scene, presses the image capture button 401. In response to receiving the press, the reception unit 32 of the communication terminal 30B identifies viewpoint information corresponding to the viewpoint currently set as viewpoint information when instructing imaging. Pressing the image capture button 401 is one example of a method for transmitting an image capture request, and an image capture request may also be transmitted in response to an input to another user interface. User interfaces include not only GUIs but also interfaces using voice or gestures.

[0224] S63: The communication unit 31 of the communication terminal 30B transmits an imaging request to the information processing system 50 by specifying the identified viewpoint information, the entered virtual room ID, and the imaging device ID corresponding to the imaging device that instructed the imaging request.

[0225] S64: The communication unit 51 of the information processing system 50 receives the imaging request, and the association processing unit 53 transmits the imaging request to the imaging device 10 identified by the imaging device ID by specifying the received viewpoint information. This imaging request may be a response to the imaging device 10 that transmitted the wide-field image, or may be a push notification.

[0226] S65: The connection unit 16 of the imaging device 10 receives the imaging request, and the imaging processing unit 13 captures the image in response to the received imaging request. Since the imaging device 10 distributes the wide-field image in real time, if imaging in response to the imaging request cannot be performed in parallel, the imaging processing unit 13 may control to temporarily stop the video distribution of the wide-field image, perform imaging in response to the imaging request, and then start the video distribution of the wide-field image again. The imaging device 10 can capture the image of the wide-field image without using viewpoint information. The imaging processing unit 13 may capture a still image or a video. In the case of a video, the image may be captured for a predetermined time (for example, 10 seconds), or the user B may specify the recording time. In addition, when capturing a video, the imaging device 10 may capture the video by recording the video being distributed while distributing the wide-field image in real time.

[0227] S66: The storage processing unit 17 of the imaging device 10 transmits a request to the information processing system 50 via the connection unit 16 to inquire about storage destination information (for example, URL) of the upload destination of the wide-field image.

[0228] S67: The communication unit 51 of the information processing system 50 receives the inquiry request, and the association processing unit 53 creates storage destination information (e.g., URL) for the storage 90 set in the virtual room. Specifically, the association processing unit 53 refers to the virtual room information storage unit 5002 and identifies the virtual room ID associated with the device ID (imaging device ID) of the imaging device 10 that is the device that originated the inquiry. Furthermore, information on the storage 90 associated with the identified virtual room ID is acquired. As the information on the storage 90, for example, address information of a specific storage 90 or information on a folder on the storage can be acquired as storage destination information (e.g., URL). In the case of a service for a construction site, folder information of a classification suitable for the site to which the service is applied, such as information on a folder associated with construction, may be prepared in advance. The communication unit 51 of the information processing system 50 transmits the acquired storage destination information (e.g., URL) to the imaging device 10. The communication unit 51 may transmit the storage destination information (e.g., URL) to the imaging device 10 together with the imaging request of S64.

[0229] S68: The connection unit 16 of the imaging device 10 receives the storage destination information (e.g., URL), and the storage processing unit 17 associates and stores the data (still image or video) of the wide-field image captured in S65 with viewpoint information corresponding to the virtual viewpoint set in the communication terminal 30B in the storage destination indicated by the storage destination information (e.g., URL) via the connection unit 16. The association process here may be a process of linking the viewpoint information so that it can be reflected when viewing the stored wide-field image. The viewpoint information may be added to the wide-field image as metadata, the viewpoint information may be managed as data separate from the wide-field image, and common identification information may be assigned so that the correspondence can be specified, or the storage location of the viewpoint information may be assigned to the wide-field image as metadata. The association process may also be performed by embedding the viewpoint information in a predetermined data item defined in the format of a standardized wide-field image. Since the viewpoint information is also stored in the image management information storage unit 5001 as described later, only the wide-field image may be stored in the storage 90.

[0230] S69: The storage processing unit 17 of the imaging device 10 transmits a notification indicating completion of storage in the storage destination information (e.g., URL) to the information processing system 50 via the connection unit 16. At this time, together with the notification, information indicating the capture date and time of the wide-field image captured by the imaging device 10 may be received.

[0231] S70: When the communication unit 51 of the information processing system 50 receives a notification indicating completion of storage, the association processing unit 53 executes a process of associating the series of information received in step S63 and the information received from the imaging device 10, etc., in the image management information storage unit 5001 and stores them. That is, the association processing unit 53 assigns a data ID, and associates and stores the imaging date and time information (which may be the date and time information when the imaging device 10 actually captured an image, or the date and time information when the communication terminal transmitted an imaging request), the imager information (linked to the IP address of the communication terminal 30B, for example), the imaging device information (acquired in step S63), the viewpoint information of the imager, the virtual room ID at the time of imaging (acquired in step S63), and the data storage location information (URL) in the image management information storage unit 5001. The association processing unit 53 may specify the virtual room ID at the time of imaging, the imaging device information, and the imager information by referring to the virtual room information storage unit 5002. The virtual room ID at the time of imaging is the virtual room ID of the virtual room in which the communication terminal that is the request source of the imaging request is present. The imaging device information is imaging device information associated with the identified virtual room ID. The imager information is user information associated with the communication terminal that transmitted the imaging request, among the user information of the users currently in the virtual room associated with the identified virtual room ID. The data name may be arbitrarily set by the user B, or may be automatically set by the information processing system 50.

[0232] S71: The communication unit 51 of the information processing system 50 designates the save destination information (e.g., a URL) and transmits a notification indicating completion of saving to the communication terminal 30B. The communication unit 51 transmits the information stored in association with the image management information storage unit 5001 to the communication terminal 30B. The communication terminal 30B can construct the image management information storage unit 3001 in the storage unit 3000 based on the received information.

[0233] S72: When the communication unit 31 of the communication terminal 30B receives a notification indicating completion of saving, the display control unit 33 displays the download button 402. Here, if the save destination information is not automatically shared with all users who have entered the virtual room, user B instructs communication terminal 30B to perform a sharing operation to share the save destination information. In this way, user B can share only the wide-field images that are determined to be worth sharing with user A who has entered the virtual room. Specifically, the reception unit 32 of the communication terminal 30B receives an operation input indicating a sharing operation.

[0234] S73: In response to the reception of the operation input indicating the sharing operation, the communication unit 31 of the communication terminal 30B transmits a sharing request to the information processing system 50 by specifying the save destination information (for example, a URL) and the virtual room ID.

[0235] S74: When the communication unit 51 of the information processing system 50 receives the sharing request, the association processing unit 53 refers to the virtual room information storage unit 5002 and identifies the communication terminal 30A that has entered the virtual room based on the virtual room ID, and the communication unit 51 transmits the save destination information (e.g., a URL) to the communication terminal 30A. When the communication unit 31 of the communication terminal 30A receives the save destination information (e.g., a URL), the display control unit 33 displays the download button 402. Note that if a sharing operation is not required, steps S72 and S73 may be omitted, and step S74 may be executed without receiving a sharing request.

[0236] The communication unit 51 transmits to the communication terminal 30A the information stored in association with the image management information storage unit 5001. The communication terminal 30A can construct the image management information storage unit 3001 in the storage unit 3000 based on the received information.

[0237] S75, S76: Users A and B press the download button 402. The reception units 32 of the communication terminals 30A and 30B receive the operation input indicating the press. If user B is already displaying a wide-field image from the viewpoint he or she wishes to share, he or she does not need to press the download button 402 again, and may simply display on the screen a notification that the save destination information has been shared with other users.

[0238] S77, S78: The communication units 31 of the communication terminals 30A and 30B connect to the destination information (e.g., URL) embedded in the download button 402, and request the storage 90 for the wide-field image and viewpoint information associated with the destination information (e.g., URL).

[0239] S79, 80: The communication unit 31 of the communication terminals 30A, B receives the wide-field image from the storage 90. The display control unit 33 of the communication terminals 30A, B, while receiving the video including the wide-field image distributed from the imaging device 10 in real time, displays the wide-field image received from the storage 90 in the first image field 211 or displays it as a pop-up. In the initial state of displaying the wide-field image, the display control unit 33 sets the viewpoint at which a predetermined area of ​​the wide-field image specified by the viewpoint information is displayed to match the center of the first image field 211 and displays it. Note that the viewpoint does not need to be completely matched to the center, and the viewpoint may be set to be included in a range near the center of the image field and displayed. The angle of view may be within a predetermined range. The user may set the initial value of the angle of view. In addition, when the wide-field image is a video, the entire file may be downloaded or may be streamed from the storage 90.

[0240] Similarly, when the smart glasses 88 are the communication terminal 30, in the initial state where the received wide-field image is displayed, the viewpoint where a predetermined area of ​​the wide-field image specified by the viewpoint information is displayed is set to match the center of the display function in front of the user's eyes and displayed. Similarly, when the VR goggles 89 are the communication terminal 30, in the initial state where the received wide-field image is displayed, the viewpoint of the wide-field image specified by the viewpoint information is set to match the center of the display function in front of the user's eyes regardless of the orientation of the head and displayed. In either case, it is not necessary to match the viewpoint completely to the center, and the viewpoint may be set to be included in a range near the center of the display function and displayed.

[0241] 32, the communication terminals 30A and 30B press a download button 402 (URL of the upload destination of the wide-field image) received from the information processing system 50 to acquire the wide-field image from the storage 90. However, as shown in FIG 33, the information processing system 50 may directly transmit the wide-field image and viewpoint information to the communication terminals 30A and 30B.

[0242] Fig. 33 is a modified example of a sequence diagram illustrating a process in which user B requests imaging from the imaging device 10 via the communication terminal 30 when a wide-field-of-view image is shared. Note that the explanation of Fig. 33 will mainly focus on the differences from Fig. 32.

[0243] In FIG. 33, the imaging device 10 returns the wide-field-of-view image and the viewpoint information to the information processing system 50 in step S91. The communication unit 51 of the information processing system 50 executes a process of associating the received wide-field-of-view image and the viewpoint information by the association processing unit 53, and then transmits the viewpoint information associated with the received wide-field-of-view image to the communication terminals 30A and 30B in the same virtual room as the imaging device 10 (S92, S93). Note that the viewpoint information to be associated may not be the viewpoint information received in step S91, but may be the viewpoint information received in step S63. In this way, the users A and B can display the wide-field-of-view image from the same viewpoint even if they do not press the download button 402 to display the wide-field-of-view image. Note that in the case of FIG. 33 as well, each communication terminal 30A and 30B may once display the wide-field-of-view image as a thumbnail, and then display the wide-field-of-view image enlarged in response to a user operation.

[0244] 33, the storage destination information (e.g., a URL) is not transmitted to the communication terminals 30A and 30B, but the information processing system 50 may transmit the storage destination information (e.g., a URL) to the communication terminals 30A and 30B. Even if the storage destination information (e.g., a URL) is not transmitted to the communication terminals 30A and 30B, the storage 90 is associated with each virtual room (see FIG. 22), so that the user can connect the communication terminal 30 to the storage 90 and view the stored wide-field image at any timing, reflecting the associated viewpoint information.

[0245] <Major Effects> As described above, in the communication system 1 of this embodiment, when a wide-field image is being distributed, even if a wide-field image captured in response to a request from a specific communication terminal is to be shared with other communication terminals, the captured wide-field image is associated with the viewpoint information so that the viewpoint information of the requesting communication terminal is shared at each location, making it easier for users to communicate their intentions.

[0246] <Application of communication systems in remote medical care> FIG. 34 is a diagram for explaining an example of remote communication in which the communication system 1 is applied to remote medical care. In the explanation of FIG. 34, the difference from FIG. 1 will be mainly explained. The site A in FIG. 34 is an operating room, but the process flow of (1) to (6) may be the same as that in FIG. 1. In FIG. 34, a patient is placed on an operating table 355 and undergoes surgery by a medical professional such as a doctor. The medical professional (corresponding to a user) operates on the patient using various surgical tools 354 such as forceps and a scalpel. In addition, the medical professional can wear smart glasses 88, and can transmit an image of the medical professional's surgical field to the communication network N. In addition, in the operating room, various cameras such as an operating field camera 351, an operating field camera 352, and an endoscope 353 are arranged as imaging devices similar to the imaging device 10. In addition, each of these imaging devices may have an imaging function for capturing an image for generating a wide-field image. All imaging devices and the smart glasses 88 in the operating room can be associated with a virtual room.

[0247] In the operating room, a main unit 356 is disposed to monitor the vital signs of the patient and the operating state of the medical equipment. The main unit 356 corresponds to the communication terminal 30 of this embodiment. The communication terminal 30 (main unit 356) in the operating room may also have a function of receiving images from the endoscope 353 and the surgical field camera 352 in addition to the functions shown in FIG. 1. The communication terminal 30 can display the received images including the wide-field image on the display 306, and can transmit the images to the information processing system 50 as images of the base of the communication terminal 30. The operation panel 357 is an input interface that accepts various operations, and may allow medical personnel to operate the equipment in the operating room via the operation panel 357. In addition, the endoscope 353, the surgical field camera 351, and the surgical field camera 352 may communicate directly with the information processing system 50 without going through the communication terminal 30. In this way, since a plurality of imaging devices 10 can be associated with the same virtual room, a user at a remote base can request the imaging of wide-field images that capture various scenes from the site of the base A. For example, if one wishes to capture images of the inside of a patient's body, one can send an imaging request to an imaging device corresponding to the endoscope 353, and if one wishes to capture images of the entire operating room, one can send an imaging request to an imaging device corresponding to the operating room camera 351.

[0248] The communication terminal 30 may also have a function of an electronic medical record system, or may have a function of communicating with the electronic medical record system. The communication terminal 30 may display electronic medical record information together with the wide-field image on the display 306. The storage 90 may also be an electronic medical record system. In such a case, the wide-field image (and associated viewpoint information) captured in response to an imaging request may be stored in association with the patient's electronic medical record by the association processing unit 53. The folders indicating the storage location of the storage 90 may also be classified by patient or surgery. The virtual room information storage unit 5002 may also store information indicating the patient and the contents of the surgery in association with each other. In this way, information regarding the patient and the surgery can be constantly displayed on the viewing screen of the communication terminal.

[0249] 35 is a diagram showing an example of a virtual room association screen (part 1) 360 for associating an imaging device with a virtual room in the case of remote medical care. In the explanation of FIG. 35, differences from FIG. 21 will be mainly explained.

[0250] In the case of remote medical care, a list of virtual rooms 361 associated with, for example, surgeries or examinations performed remotely is displayed on the virtual room association screen (part 1) 360. Medical cameras 362 including omnidirectional cameras are associated with the virtual rooms. The medical cameras 362 include endoscopes, surgical field cameras used to capture images of the surgical field in the operating room, cameras that capture microscopic images, and the like.

[0251] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0252] For example, the configuration example in Fig. 13 and the like is divided according to main functions in order to facilitate understanding of the processing by the information processing system 50, the imaging device 10, and the communication terminal 30. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the information processing system 50, the imaging device 10, and the communication terminal 30 can be divided into more processing units according to the processing contents. Also, one processing unit can be divided so as to include more processes.

[0253] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0254] Additionally, the devices described in the examples are merely one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the information processing system 50 includes a plurality of computing devices, such as a server cluster. The computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processes disclosed herein.

[0255] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figures 25, 26, 32, and 33, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Also, the information processing system 50 may be integrated into one server device, or may be divided into multiple devices. [Explanation of symbols]

[0256] 10. Imaging device 30 Communication terminals 50 Information Processing Systems [Prior art documents] [Patent documents]

[0257] [Patent Document 1] JP 2019-121857 A

Claims

1. A communication system including a plurality of communication terminals that display a wide-field-of-view image having a wide range of viewing angles and an information processing system, The first communication terminal a first display control unit that displays a first wide-field image; When the first display control unit receives an operation input by a user of the first communication terminal to change a viewpoint of the first wide-field-of-view image, the first display control unit displays the first wide-field-of-view image from a changed viewpoint; a first communication unit that transmits viewpoint information corresponding to a currently displayed viewpoint and an imaging request to the information processing system when an operation to capture a wide-field image is received from a user of the first communication terminal; The information processing system includes: a second communication unit that transmits an imaging request to an imaging device that captured the first wide-field image when the imaging request is received from the first communication terminal; an association processing unit that associates a second wide-field-of-view image captured by the imaging device with the viewpoint information received from the first communication terminal, The association processing unit: Transmitting the second wide-field image captured by the imaging device and the viewpoint information received from the first communication terminal to a second communication terminal; The second communication terminal, a second display control unit that displays the second wide-field image at a viewpoint corresponding to the viewpoint information received from the information processing system; A communication system having the above configuration.

2. When receiving an imaging request for a wide-field-of-view image from the first communication terminal, the association processing unit receives viewpoint information arbitrarily designated for the first wide-field-of-view image in the first communication terminal, and transmits an imaging request for the second wide-field-of-view image, storage destination information, and the viewpoint information to the imaging device; The imaging device includes:

2. The communication system according to claim 1, wherein the second wide-field image and the viewpoint information are stored in a storage destination specified by the storage destination information.

3. When receiving an imaging request for a wide-field-of-view image from the first communication terminal, the association processing unit receives viewpoint information arbitrarily designated for the first wide-field-of-view image in the first communication terminal, and transmits an imaging request for the second wide-field-of-view image and the viewpoint information to the imaging device; the imaging device transmits the second wide-field image and the viewpoint information to the information processing system; The communication system according to claim 2 , wherein the second communication unit transmits the second wide-field-of-view image and the viewpoint information received from the imaging device to a second communication terminal.

4. The association processing unit transmits the storage destination information to the first communication terminal, The communication system according to claim 3, characterized in that, when a request to share the storage destination information is received from the first communication terminal, the second communication unit transmits the storage destination information to the second communication terminal.

5. The information processing system includes: providing a screen for accepting a start or stop of transmission of a wide-field-of-view image by the imaging device to a communication terminal; The communication system according to any one of claims 1 to 4, characterized in that the imaging device is requested to start or stop transmitting the wide-field image in response to a transmission start request or transmission stop request from a communication terminal.

6. The information processing system includes: Accepting registration of the imaging device for a tenant; a communication group management unit that provides a communication terminal with a list of communication groups that are registered in advance in the tenant and the imaging device that is registered in the tenant; The communication system according to any one of claims 1 to 5, characterized in that the communication group management unit associates the imaging device, the selection of which is accepted by the communication terminal, with the communication group, the selection of which is accepted by the communication terminal.

7. The second communication terminal, receiving the second wide-field-of-view image and the viewpoint information from a storage destination designated by the storage destination information; 5. The communication system according to claim 3, wherein the second wide-field-of-view image specified by the viewpoint information is displayed with its viewpoint aligned with the center of an image field.

8. The second communication terminal, The communication system according to claim 7, characterized in that the storage destination information is received from the information processing system, and a display component is displayed that accepts acquisition of the second wide-field image stored in the storage destination indicated by the storage destination information.

9. The second communication terminal, The communication system according to claim 8, wherein the storage destination information and a thumbnail image of the second wide-field-of-view image are received from the information processing system, and the thumbnail image is displayed together with the display component.

10. The communication system according to claim 1 , wherein the wide-field image is a spherical image.

11. when the information processing system receives from the imaging device a request for storage destination information for storing the second wide-field image captured by the imaging device, the information processing system transmits the storage destination information to the imaging device; receiving, from the imaging device that has stored the wide-field image in association with the viewpoint information in a storage destination designated by the storage destination information acquired by the imaging device, a notice that the wide-field image has been stored and the storage destination information; The viewpoint information received from the imaging device is associated with the wide-field image and recorded. The communication system according to claim 2.

12. An information processing system capable of communicating with a plurality of communication terminals that display a wide-field-of-view image having a wide range of viewing angles, a communication unit that displays a first wide-field-of-view image, and transmits an imaging request to the imaging device when viewpoint information corresponding to a currently displayed viewpoint and an imaging request are received from a first communication terminal that has received an operation by a user to capture the wide-field-of-view image; an association processing unit that associates a second wide-field-of-view image captured by the imaging device with the viewpoint information received from the first communication terminal, The association processing unit: Transmitting the second wide-field-of-view image captured by the imaging device and the viewpoint information received from the first communication terminal to the second communication terminal so that the second communication terminal displays the second wide-field-of-view image at a viewpoint corresponding to the viewpoint information; Information processing system.

13. An information processing system capable of communicating with a plurality of communication terminals that display a wide-field-of-view image having a wide range of viewing angles, a communication unit that displays a first wide-field-of-view image, and transmits an imaging request to the imaging device when viewpoint information corresponding to a currently displayed viewpoint and an imaging request are received from a first communication terminal that has received an operation by a user to capture the wide-field-of-view image; an association processing unit that associates a second wide-field image captured by the imaging device with the viewpoint information received from the first communication terminal; The association processing unit: Transmitting the second wide-field-of-view image captured by the imaging device and the viewpoint information received from the first communication terminal to the second communication terminal so that the second communication terminal displays the second wide-field-of-view image at a viewpoint corresponding to the viewpoint information; program.

14. An image sharing method performed by an information processing system capable of communicating with a plurality of communication terminals that display wide-field-of-view images having a wide range of viewing angles, comprising: a process of transmitting an imaging request to an imaging device when viewpoint information corresponding to a currently displayed viewpoint and an imaging request are received from a first communication terminal that displays a first wide-field-of-view image and that has received an operation by a user to capture the wide-field-of-view image; A process of associating a second wide-field-of-view image captured by the imaging device with the viewpoint information received from the first communication terminal; A process of transmitting the second wide-field-of-view image captured by the imaging device and the viewpoint information received from the first communication terminal to the second communication terminal so that the second communication terminal displays the second wide-field-of-view image at a viewpoint corresponding to the viewpoint information; How to share images.

Images