Communication systems, information processing systems, video creation methods, programs

Abstract

To provide a communication system that reproduces in parallel a plurality of attention point moving images concerning different types of attention points.SOLUTION: The present invention is a communication system including an imaging device capable of capturing a wide-field image having a wide range of viewing angles and an information processing system that transmits video including the wide-field image captured by the imaging device to a communication terminal. The information processing system includes: a recording unit that records recorded video of the wide-field image displayed on the communication terminal; a moving image creation unit that creates a mixed moving image in which a plurality of attention point moving images is arranged, each reproducing a predetermined area of the recorded video recorded in the recording unit at different types of attention points; and a communication unit that transmits screen information of the mixed moving image in which the plurality of the attention point moving images is reproduced in parallel to the communication terminal.SELECTED DRAWING: Figure 30

Description

Technical Field

[0006] , ,

[0001] The present invention relates to an information processing system, a communication system, a moving image creation method, and a program.

Background Art

[0002] There is known a communication system that transmits images and sounds in real time from one base point to one or more other base points and enables remote communication using images and sounds between users located at remote locations. As such an image, there is known a wide-angle image having a wide imaging range captured in a wide imaging range including, for example, a 360-degree image (also referred to as an omnidirectional image, a panoramic image, or a full-surround image) in which a 360-degree entire circumference including portions that cannot be fully confirmed within a normal angle of view is imaged. A user can view a part of the area of the wide-angle image from an arbitrary virtual viewpoint by operating a communication terminal to change the virtual viewpoint for a part of the wide-angle image displayed on the display screen of the communication terminal.

[0003] Techniques for efficiently browsing a wide-angle image are known (see, for example, Patent Document 1). Patent Document 1 discloses a technique for simultaneously reproducing images of a plurality of viewpoints in a wide-angle image.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional techniques have a problem in that a plurality of focus videos related to different types of focus points are not reproduced in parallel. For example, a plurality of focus videos are not reproduced in parallel with two or more of a fixed point, an arbitrary object, or the viewing viewpoints of others as focus points.

[0005] In view of the above problems, an object of the present invention is to provide a communication system that reproduces a plurality of focus videos related to different types of focus points in parallel.

Means for Solving the Problems

[0006] In view of the above problems, the present invention provides a communication system comprising an imaging device capable of capturing a wide-field image having a wide field of view, and an information processing system that transmits a video including the wide-field image captured by the imaging device to a communication terminal, wherein the information processing system includes a recording unit that records a recorded video of the wide-field image displayed on the communication terminal, a video creation unit that creates a hybrid video in which a plurality of point-of-interest videos are arranged, each playing back a predetermined area of ​​the recorded video recorded by the recording unit at different points of interest, and a communication unit that transmits screen information of the hybrid video in which the plurality of point-of-interest videos are played back in parallel to the communication terminal. [Effects of the Invention]

[0007] The present invention can provide a communication system that plays multiple videos of different types of points of interest in parallel. [Brief explanation of the drawing]

[0008] [Figure 1] This figure illustrates an example of remote communication using wide-field imaging. [Figure 2] This is an example of a schematic diagram of a communication system configuration. [Figure 3] This is an example of a hardware configuration diagram for an imaging device. [Figure 4] This is an example of a hardware configuration diagram for a communication terminal and an information processing system. [Figure 5] (a) is a left side view of the imaging device, (b) is a front view of the imaging device, and (c) is a top view of the imaging device. [Figure 6] This is a diagram illustrating the use of the imaging device. [Figure 7] (a) is a hemispherical image (front) captured by the imaging device, (b) is a hemispherical image (back) captured by the imaging device, and (c) is an image represented using equirectangular projection. [Figure 8] (a) A conceptual diagram showing the state of covering the sphere with an equirectangular projection image, and (b) A diagram showing a full-sphere image. [Figure 9]This diagram shows the positions of a virtual camera and a predetermined region when a 360-degree image is treated as a three-dimensional sphere. [Figure 10] (a) is a stereoscopic perspective view of Figure 5, and (b) is a diagram showing the state in which an image of a predetermined area is displayed on the display. [Figure 11] This figure shows the relationship between information from a predetermined region and an image of a predetermined region T. [Figure 12] This is a diagram showing a point in three-dimensional Euclidean space using spherical coordinates. [Figure 13] This is an example of a functional configuration diagram for a communication system. [Figure 14] This is a conceptual diagram showing the image management information stored in the image management information storage unit. [Figure 15] This is a conceptual diagram showing the virtual room information stored in the virtual room information storage unit and the tenant information stored in the tenant information storage unit. [Figure 16] This is an example of a conceptual diagram showing the recorded video information stored in the recorded video memory unit. [Figure 17] This is an example of a conceptual diagram showing recorded video information with a tracking viewpoint. [Figure 18] This is an example of a conceptual diagram showing the items of a report stored in the report memory unit. [Figure 19] (a) is a diagram showing an example of the room entry screen. (b) is a diagram showing an example of the image viewing screen displayed by the communication terminal when a user enters a virtual room. [Figure 20] This is an example of a sequence diagram illustrating the process of a user (or communication terminal) entering a virtual room. [Figure 21] This figure shows an example of a device registration screen displayed by a communication terminal. [Figure 22] (a) is an example of a diagram showing the imaging device registration dialog. (b) is a diagram showing an example of a two-dimensional code screen. [Figure 23] This figure shows an example of the VR goggles registration screen that appears when the VR goggles registration button is pressed. [Figure 24]This is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room. [Figure 25] This is a diagram showing an example of a virtual room association screen (part 2). [Figure 26] This is a diagram showing an example of a virtual room association screen (part 3). [Figure 27] This is a diagram showing an example of a wide - field image transmission start / stop dialog displayed on a communication terminal. [Figure 28] This is an example of a sequence diagram showing the procedure for a user to register an imaging device in a virtual room. [Figure 29] This is an example of a sequence diagram explaining the flow in which a wide - field image is shared. [Figure 30] This is a diagram explaining the outline of the generation of a hybrid video. [Figure 31] This is an example of a sequence diagram showing a communication terminal recording video during remote communication. [Figure 32] This is an example of a sequence diagram showing the overall process in which a communication terminal and an information processing system communicate to create a hybrid video. [Figure 33] This is a diagram showing an example of a home screen displayed on a communication terminal when creating a hybrid video. [Figure 34] This is a diagram showing an example of a recorded video selection screen displayed on a communication terminal. [Figure 35] This is an example of a sequence diagram showing the process in which a communication terminal and an information processing system communicate, and the information processing system accepts the setting of a point of interest and creates point - of - interest information. [Figure 36] This is a diagram showing an example of a point - of - interest setting screen displayed on a communication terminal. [Figure 37] This is a diagram showing an example of a fixed - point setting screen displayed on a communication terminal. [Figure 38] This is a diagram showing an example of a viewing - point setting screen displayed on a communication terminal. [Figure 39] This is a diagram showing an example of a follow - up setting screen displayed on a communication terminal. [Figure 40] This is a diagram showing an example of a follow - up setting screen (part 2) displayed on a communication terminal. [Figure 41] This is an example of a sequence diagram showing the process in which a communication terminal and an information processing system communicate, and the information processing system uses the point of interest information to create a hybrid video. [Figure 42] This is an example of a flowchart illustrating the process by which the video creation department creates a hybrid video. [Figure 43] This diagram explains how to create a video highlighting key points. [Figure 44] This diagram illustrates how to create a hybrid video using videos highlighting key points. [Figure 45] This figure shows an example of a hybrid video containing one or two point-of-interest videos. [Figure 46] This is an example of a sequence diagram showing the process in which a communication terminal and an information processing system communicate, and the information processing system provides a hybrid video to the communication terminal. [Figure 47] This diagram illustrates an example of remote communication where a communication system is applied to telemedicine. [Figure 48] This figure shows an example of a mixed video playback screen when an imaging device is used in a medical setting. [Modes for carrying out the invention]

[0009] The following describes an example of an embodiment for carrying out the present invention: an information processing system and a video creation method performed by the information processing system.

[0010] <An example of remote communication> Figure 1 illustrates an example of remote communication using wide-field imaging. In Figure 1, three locations A to C communicate via an information processing system 50. The number of locations is just an example; there could be two locations, four or more locations, or any number of locations.

[0011] Location A is, for example, a construction site. Locations B and C can be anywhere, such as an office, as long as they are locations where wide-field images can be transmitted. Location A is equipped with an imaging device 10 capable of capturing wide-field images, such as a 360-degree spherical image, or wide-angle wide-field images of the surroundings, such as 180 to 360 degrees in the vertical or horizontal direction. Hereafter, such wide-angle images will simply be referred to as wide-field images. Locations A to C are equipped with various communication terminals 30A to 30C for viewing wide-field images. Hereafter, any of the communication terminals 30A to 30C will be referred to as "communication terminal 30".

[0012] At the construction site, various construction activities are being carried out by workers at each location. The entire site is captured using wide-field images, and users at each location A to C can arbitrarily change their virtual viewpoint to check any construction or work they wish to focus on. The viewpoint is the center position or range of a predetermined area displayed on a screen such as a display within the entire wide-field image.

[0013] The imaging device 10 can be mounted on a tripod 86 or on an arm 85 via a gimbal 87. A relay device (in Figure 1, the communication terminal 30A also serves as the relay device) is installed at the construction site, and the communication terminal 30A transmits the wide-field images received from the imaging device 10 via wired or wireless connection to the information processing system 50. The communication terminal 30A can also serve as a terminal for viewing the wide-field images. A camera 9 is connected to (or built into) the communication terminal 30A, and images of a normal field of view (or 360-degree images) captured by the camera 9 can also be transmitted to the information processing system 50. In addition, user A (for example, a worker) can wear smart glasses 88, and images of a normal field of view (or 360-degree images) captured by the smart glasses 88 may be transmitted to the information processing system 50. Smart glasses 88 are information terminals that display information acquired via the internet on a display while maintaining a field of view. Smart glasses 88 may be placed at any location.

[0014] At site B, a PC (Personal Computer) or smartphone is provided as an example of a communication terminal 30B. Furthermore, any device capable of communicating with the information processing system 50 can be a communication terminal 30B; other devices such as tablet terminals, PDAs (Personal Digital Assistants), electronic whiteboards, or projectors are also acceptable. A camera may be built into or connected to the communication terminal 30B.

[0015] At base C, a PC, smartphone, VR goggles 89, etc., are placed as examples of communication terminals 30C. In Figure 1, a camera 8 is built into or connected to the communication terminal 30C. VR goggles 89 are information terminals that display an artificial world on a computer or a 360-degree image according to the direction of movement of the head or body. The camera 8 can be either wide-angle or standard-angle. Furthermore, the communication terminal 30C can be any device capable of communicating with the information processing system 50, such as a tablet terminal, PDA, electronic whiteboard, or projector. VR goggles 89 may be placed at any base.

[0016] In this embodiment, the imaging device 10 and the communication terminal 30 are managed in a communication group called a virtual room. The imaging device 10 is associated with the virtual room, and the communication terminal 30 (the user operating the communication terminal 30) can enter this virtual room to receive the wide-field image transmitted by the imaging device 10 and view the wide-field image. Smart glasses 88 and VR goggles 89 can also be associated with the virtual room. Cameras 8 and 9 enter the virtual room together with the communication terminal 30.

[0017] Users A to C at locations A to C can arbitrarily change the viewpoint of the wide-field image. Therefore, since each user A to C viewing the wide-field image in real time may be seeing a different viewpoint, there is a risk that communication may become difficult. To address this, in this embodiment, for example, the viewpoint of the communication terminal 30 at location B is transmitted to the communication terminals 30 at locations A and C, so that the information of the virtual viewpoint set at the communication terminal 30 of any location can be shared with the communication terminals 30 at other locations. The outline of the sharing will be explained below. In the following explanation, for the sake of explanation, we will assume that the viewpoint specified by the user at location B is shared with the users at locations A and C.

[0018] (1) Communication terminals 30A to 30C share a wide-field image (an example of a first wide-field image) captured by the imaging device 10. When user B requests the capture of a wide-field image while viewing from an arbitrary viewpoint on communication terminal 30B, communication terminal 30B (an example of a first communication terminal) transmits viewpoint information and the capture request to the information processing system 50.

[0019] (2) In response to an imaging request, the information processing system 50 specifies viewpoint information and requests the imaging device 10 to take an image (either a still image or a video).

[0020] (3) The imaging device 10 takes an image in response to an imaging request and stores the wide-field image (an example of a second wide-field image) and viewpoint information in association with a URL (an example of storage location information; in Figure 1, the storage location on the storage 90) notified by the information processing system 50. The wide-field image stored on the storage 90 can be downloaded and displayed by any communication terminal 30.

[0021] (4) The information processing system 50 sends a URL to the communication terminal 30B.

[0022] (5) The information processing system 50 also automatically or at the request of user B transmits the URL to the communication terminals 30A and 30C (an example of a second communication terminal) that are in the same virtual room.

[0023] (6) The communication terminals 30A and 30C connect to a URL to receive viewpoint information and a wide-field image, and set the viewpoint of the wide-field image identified by the viewpoint information to align with the center of the image field and display it. However, it is not necessary to align the viewpoint perfectly with the center; the viewpoint may be set to be included in the area near the center of the image field and displayed.

[0024] The same applies when sharing the perspective of users at location A with users at locations B and C, and when sharing the perspective of users at location C with users at locations A and B.

[0025] As described above, even when a wide-field image is distributed, the communication system 1 of this embodiment facilitates user communication because viewpoint information is shared without requiring instructions to move the viewpoint relative to the wide-field image captured so that a predetermined area of ​​focus at each location is displayed.

[0026] In (3), the imaging device 10 can transmit the wide-field image itself to the information processing system 50, and in (4), the information processing system 50 can transmit the wide-field image to the communication terminals 30A to 30C.

[0027] While Figure 1 illustrates an example where the imaging device 10 is deployed at a construction site, this embodiment can also be applied to VR (Virtual Reality) education, event distribution, remote customer service, telemedicine, and more. In VR education, the imaging device 10 is deployed at a site such as a laboratory, allowing students to remotely change their viewpoint and view blackboards, equipment, samples, experimental results, etc. In event distribution, the imaging device 10 is deployed at the event venue, allowing event participants, such as spectators, to remotely change their viewpoint and view the venue online. The venue view includes images of performers, contestants, presenters, products and exhibits explained at the event, images of materials explained at the event, and images of the venue's condition. The event venue can be indoors or outdoors and includes venues for sports, concerts, plays, etc. In remote customer service, for example, when applied to customer service at a travel agency, the imaging device 10 is deployed at the travel destination, allowing customers to remotely change their viewpoint and review their itinerary. In telemedicine, the imaging device 10 is placed in a medical setting such as an operating room, and doctors, students, medical equipment personnel, etc., can remotely change their viewpoint and view the actions of doctors and nurses performing medical procedures in the medical setting, the placement of equipment, the patient's condition, vital signs, etc.

[0028] The locations where images are captured are not limited to these; any space where users (viewers) at the viewing location have a need to understand the situation at a remote location, such as schools, factories, warehouses, construction sites, server rooms, or stores, is acceptable.

[0029] <About Terminology> A tenant refers to a group of users associated with a contract unit when contracting to receive an image distribution service from a service provider (in this embodiment, an information processing system). This includes contracting companies, organizations, individuals, etc. Therefore, a tenant can also be referred to as a user group. While a user may belong to a tenant as an example, an individual user may also subscribe to the service. In addition to users, imaging devices, virtual rooms, etc., are registered within a tenant (user group).

[0030] A base of operations refers to a place that serves as the foundation for activities. In this embodiment, a conference room is used as an example of a base of operations. A conference room is a room set up primarily for the purpose of holding meetings. Meetings are also called gatherings, meetings, discussions, assemblies, and other similar terms.

[0031] A device refers to an apparatus other than a general-purpose communication terminal 30 such as a PC or smartphone, and is an imaging device or a wide-field image viewing device. In this embodiment, examples include an imaging device 10, smart glasses 88, and VR goggles 89.

[0032] Viewpoint information refers to parameter information that specifies which predetermined region of a wide-field image to display on the display screen. In this embodiment, the radial, polar angle, and azimuth angle corresponding to the center of the wide-field image displayed on the display screen are described as examples of viewpoint information, but it may also be specified by other parameter information such as the coordinates of diagonal vertices.

[0033] • Wide-field image A wide-field-of-view image is an image with a wide field of view captured over a wide imaging range, including 360-degree images (also called omnidirectional images or all-around images) that capture the entire 360 ​​degrees surrounding the subject. A wide-field-of-view image is an image that has a wide field of view that is wider than the display range that can be displayed at once on the display screen (the area where the wide-field-of-view image is displayed) in a predetermined display method. A wide-field-of-view image has a display range of up to 360 degrees (or 180 degrees) vertically and 360 degrees horizontally, but an image with a wide field of view that is wider than the display range that can be displayed at once on a display is also included as a wide-field-of-view image, even if it is less than 360 degrees vertically or horizontally. For example, an image with a display range wider than the range that a human can see at once is also included as a wide-field-of-view image. In addition, an image with a display range of 160 degrees or more vertically and horizontally is also included as a wide-field-of-view image. Furthermore, even if an image can be displayed at once on the display screen depending on the display method, it can be included as a wide-field-of-view image if it has a wide field of view by switching to or changing to a predetermined display method. In this embodiment, an equirectangular spherical image is used as an example of a wide-field image, but omnidirectional images, hemispherical images, 3D panoramic images, 2D panoramic images, and VR (Virtual Reality) images are also included in the wide-field image category. The wide-field image may also be an image in the form of cube mapping, dome master, etc. Furthermore, the spherical image may be in a format other than equirectangular.

[0034] Images captured with a normal field of view are not wide-field images, but in this embodiment, they will be described as non-wide-field images (planar images).

[0035] A communication group is a collection of users from whom wide-field images are shared (distributed). In a normal space, a communication group is described as a virtual room, meaning that when each user enters the same room, each user can share the wide-field image. Here, "virtual" means that it is realized through information processing via a network.

[0036] Users at each location communicate remotely with each other. Remote communication refers to communicating with someone who is physically far away, using IT tools to convey information through images and audio. It does not necessarily have to be two-way communication. Therefore, a virtual room can also be called a virtual meeting room.

[0037] • VR education, remote customer service VR education refers to educational tools that utilize VR (Virtual Reality) to create an unrealistic space before the viewer's eyes, immersing them in a realistic experience and deepening their understanding. Remote customer service refers to customer service where staff in remote locations provide product information to users via video conferencing using tablets or displays.

[0038] The viewing viewpoint is a chronological arrangement of viewpoints that the viewer manually changed in real time while viewing the wide-field image.

[0039] A point of focus is a viewpoint used to identify a predetermined area from recorded video and create a video. The type of point of focus refers to the difference in the method of determining the point of focus. In this embodiment, examples of types include a fixed point, a viewing viewpoint, or a tracking viewpoint.

[0040] A "point of interest video" is a video that focuses on a specific point of interest within a designated area, while a "composite video" is a video containing multiple point of interest videos that can be played in parallel. "Parallel" means that they can be played on a single screen. A single screen includes the area that can be displayed by scrolling. "Composite" means combining two or more things to form one thing or a group. "Composite" may also be referred to as "composition," "mixture," or "integration."

[0041] The object to be tracked can be tangible or intangible, and it only needs to have a shape that is at least temporarily visible in the wide-field image. The object does not need to be constantly visible in part of the wide-field image from the beginning to the end of the recorded video. Tracking is possible if it is continuously visible for a portion of the recorded video.

[0042] <Example of a communication system configuration> Figure 2 is an example of a schematic diagram of the configuration of communication system 1. Figure 1 is an example of applying communication system 1 from Figure 2 to remote communication with the field. Communication system 1 is a system that transmits and receives wide-field images and normal-angle images captured by the imaging device 10 bidirectionally between multiple locations. By displaying images distributed from one location at other locations, users at other locations can view the images. As an example of a wide-field image, a 360-degree spherical image captured by the imaging device 10 is distributed. Communication system 1 can, for example, allow a wide-field image captured at a predetermined location to be viewed at other locations in remote locations.

[0043] As shown in Figure 2, in the communication system 1, the imaging device 10, communication terminal 30A, and information processing system 50 located at base A, and communication terminals 30B and 30C located at each of the multiple bases (bases B and C), are connected in a way that enables communication.

[0044] If the imaging device 10 has a communication function that allows it to connect directly to the communication network N, then a communication terminal 30A acting as a relay device (e.g., a router) is not necessary. In this case, the communication terminal 30A connects to the communication network N without the imaging device 10. However, if the communication terminal 30A is located at site A, the communication terminal 30A also functions as a relay device, allowing user A to view wide-field images in the same way as with communication terminals 30B and 30C. Furthermore, imaging devices 10 may be located at other sites besides site A, or multiple imaging devices 10 may be located at site A.

[0045] Each communication terminal 30 and information processing system 50 can communicate via the communication network N. The communication network N is constructed using the Internet, mobile communication networks, LANs (Local Area Networks), etc. The communication network N may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0046] The imaging device 10 is a digital camera capable of capturing a full-sphere image by capturing a subject, landscape, etc., to obtain two hemispherical images as the basis. The wide-field image obtained by the imaging device 10 may be a video, a still image, or both. Furthermore, the captured image may be video containing sound along with the image.

[0047] The communication terminal 30 is a computer such as a PC used by users at each location. The communication terminal 30 displays images captured at its own location, wide-field images (still images or videos) distributed from other locations, and images with a normal field of view. For example, the communication terminal 30 acquires wide-field images captured by the imaging device 10 via the communication network N. The communication terminal 30 also has image processing software such as OpenGL ES installed, and can display images based on viewpoint information that identifies a portion of the wide-field image. Note that OpenGL ES is just one example of image processing software, and other software may be used. Furthermore, even if image processing software is not installed, image processing may be performed by software received from an external source, or image display may be performed by receiving the results of image processing performed by external software. In other words, the communication terminal 30 can display a predetermined portion of the wide-field image.

[0048] The communication terminal 30 can arbitrarily change the viewpoint relative to the display range of a wide-field image in response to user operation. The communication terminal 30 can change the field of view (predetermined area) based on viewpoint information corresponding to the changed viewpoint by moving the virtual viewpoint in response to user operation input (including key input, drag, scrolling, etc.) to a touch panel, directional buttons, mouse, keyboard, touchpad, etc. Furthermore, if the communication terminal 30 is a communication terminal worn by a user, such as VR goggles, the communication terminal 30 may detect the change in posture information of the communication terminal 30 in response to changes in the movements of the user wearing it, and change the field of view (predetermined area) based on viewpoint information corresponding to the changed viewpoint by moving the virtual viewpoint in response to the detected posture information.

[0049] The communication terminal 30A distributes wide-field images acquired from the imaging device 10 via a wired cable such as a USB (Universal Serial Bus) cable to communication terminals 30 at other locations via the information processing system 50. The connection between the imaging device 10 and the communication terminal 30A may be a wireless connection using short-range wireless communication or the like, rather than a wired connection using a wired cable. Multiple communication terminals 30A may be located at location A.

[0050] In some cases, a user at location A may wear smart glasses 88, and the smart glasses 88 may connect to a communication network. In this case, the images captured by the smart glasses 88 are transmitted to the information processing system 50 via the communication network, and the information processing system 50 can distribute them to the communication terminals 30 at each location.

[0051] Communication terminal 30B is located at site B where user B is located, and communication terminal 30C is located at site C where user C is located. Multiple communication terminals 30B and 30C may be located at sites B and C. Furthermore, communication terminals 30B and 30C may be carried by user B and user C respectively.

[0052] Furthermore, the communication terminals 30A to 30C at locations A to C can have cameras 8 and 9 built-in or externally attached as imaging units, and the communication terminals 30A to 30C can distribute images of their own location captured by their own cameras 8 and 9 to other locations. In addition, any devices may be placed at locations A to C.

[0053] The arrangement of each terminal and device (communication terminal 30 and imaging device) and the user shown in Figure 2 is an example, and other examples may be used. Furthermore, the communication terminal 30 is not limited to a PC, but may be, for example, a tablet terminal, smartphone, PDA, wearable device (including smart glasses and VR goggles), projector, electronic blackboard (a whiteboard with electronic blackboard functionality that enables mutual communication), or autonomous robot. The communication terminal 30 can be any computer on which a dedicated application for a web browser or image distribution service runs.

[0054] Furthermore, if the imaging device 10 has a display, it may be configured to display images distributed from other locations.

[0055] The information processing system 50 has one or more information processing devices. The information processing system 50 manages and controls communication between the imaging devices 10 and communication terminals 30 at each location, and manages the wide-field images that are transmitted and received. The information processing system 50 provides a platform that allows users to utilize the functions necessary to provide an image distribution service that distributes wide-field images. This platform may be made available to service providers, such as individuals or companies that wish to provide image distribution services, by contract. Hereinafter, in order to distinguish them from tenants that receive image distribution services, service providers who provide image distribution services to users using the contracted platform will be referred to as platform subscribers.

[0056] Therefore, the information processing system 50 may, as a platform, expose an API (Application Programming Interface), allowing platform subscribers to use this API to provide various image distribution services. Platform subscribers only need to develop software such as applications that handle screens displayed by the communication terminal 30 and API calls, and do not need to develop functions provided by APIs such as image distribution from scratch.

[0057] The information processing system 50 may be constructed using a single computer, or it may be constructed using multiple computers, each part (function or means) of which is arbitrarily assigned. Furthermore, all or part of the functions of the information processing system 50 may be server computers located in a cloud environment, or server computers located in an on-premises environment.

[0058] Storage 90 is a memory device that stores data such as captured wide-field images. Storage 90 may be an external storage device separate from the information processing system 50 (it may be storage located on the cloud or storage located on-premises), or it may be storage included in the information processing system 50.

[0059] <Example Hardware Configuration> Next, the hardware configuration of each device or terminal in the image communication system according to this embodiment will be described using Figures 3 and 4. Note that the hardware configuration shown in Figures 3 and 4 may have components added or removed as needed.

[0060] <<Hardware configuration of the imaging device>> First, the hardware configuration of the imaging device 10 will be explained using Figure 3. Figure 3 is a diagram showing an example of the hardware configuration of the imaging device 10. In the following, the imaging device 10 will be described as a 360-degree (omnidirectional) imaging device using two image sensors, but it may have one image sensor or two or more. Furthermore, it is not necessarily required to be a device dedicated to omnidirectional imaging; an omnidirectional imaging unit can be attached to a regular digital camera or smartphone, etc., to provide essentially the same functionality as the imaging device 10.

[0061] As shown in Figure 3, the imaging device 10 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output interface 116, a short-range communication circuit 117, an antenna 117a for the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network interface 121.

[0062] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lens 102 unless otherwise specified) each having a field of view of 180° or more for forming a hemispherical image, and two image sensors 103a and 103b provided corresponding to each lens. The image sensors 103a and 103b have an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the lenses 102a and 102b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers for setting various commands or parameters necessary for the operation of the image sensors. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example; it may have only one, or three or more.

[0063] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, operation unit 115, input / output I / F 116, short-range communication circuit 117, electronic compass 118, gyro sensor 119, acceleration sensor 120, and network I / F 121, etc.

[0064] The image processing unit 104 receives image data output from image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create data for an equirectangular projection image (an example of a wide-field image), which will be described later.

[0065] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 acting as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0066] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 10, there may also be functions to display a preview or video on a display (for example, the display of an external terminal such as a smartphone that communicates with the imaging device 10 via a short-range communication circuit 117). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0067] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0068] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress of processing by the image processing unit 104 and data of completed equirectangular projection images.

[0069] The control unit 115 is a collective term for various operation buttons, a power switch, a shutter button, and a touch panel that combines display and operation functions. The user inputs various imaging modes, imaging conditions, etc., by operating the control unit 115.

[0070] The input / output interface (I / F) 116 is a general term for interface circuits (such as USB I / F) to external media such as SD cards or personal computers. The I / F 116 can be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded to external media via the I / F 116, or transmitted to an external terminal (device) via the I / F 116 as needed.

[0071] The short-range communication circuit 117 communicates with an external terminal (device) via an antenna 117a provided on the imaging device 10 using a short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi. The short-range communication circuit 117 can transmit equirectangular projection image data to the external terminal (device).

[0072] The electronic compass 118 calculates the orientation of the imaging device 10 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) according to Exif and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data. The gyro sensor 119 is a sensor that detects changes in angle (Roll angle, Pitch angle, Yaw angle) associated with the movement of the imaging device 10. The changes in angle are an example of related information (metadata) according to Exif and are used for image processing such as image correction of captured images. Furthermore, the acceleration sensor 120 is a sensor that detects acceleration in three axes. The imaging device 10 calculates its own orientation (angle relative to the direction of gravity) based on the acceleration detected by the acceleration sensor 120. By providing the acceleration sensor 120, the accuracy of image correction of the imaging device 10 is improved.

[0073] Network I / F121 is an interface for data communication using a communication network N, such as the Internet, via a router or similar device.

[0074] Furthermore, the hardware configuration of the imaging device 10 is not limited to that shown herein; any configuration that can realize the functional configuration of the imaging device 10 is acceptable. In addition, at least a part of the above hardware configuration may reside on a network.

[0075] <<Hardware configuration of the communication terminal>> Figure 4 shows an example of the hardware configuration of the communication terminal 30 and the information processing system 50. First, the communication terminal 30 will be described. The hardware configuration of the communication terminal 30 is indicated by codes in the 300 series. The communication terminal 30 is built by a computer and, as shown in Figure 4, is equipped with a CPU 301, ROM 302, RAM 303, HDD (Hard Disk Drive) 304, HDD controller 305, display 306, external device connection I / F 308, network I / F 309, bus line 310, keyboard 311, pointing device 312, DVD-RW (Digital Versatile Disk Rewritable) drive 314, media I / F 316, sound input / output I / F 317, microphone 318, speaker 319, short-range communication circuit 320, and camera 321.

[0076] Of these components, the CPU 301 controls the overall operation of the communication terminal 30. The ROM 302 stores programs used to drive the CPU 301, such as IPLs. The RAM 303 is used as the work area for the CPU 301. The HDD 304 stores various data, such as programs and data. The HDD controller 305 controls the reading or writing of various data to the HDD 304 according to the control of the CPU 301. The display 306 displays various information such as cursors, menus, windows, characters, or images. The display 306 is an example of a display unit. The display 306 may also be a touch panel display equipped with input means. The external device connection I / F 308 is an interface for connecting various external devices. In this case, external devices include, for example, USB memory or printers. The network I / F 309 is an interface for data communication using the communication network N. The bus line 310 is an address bus or data bus, etc., for electrically connecting each component, such as the CPU 301 shown in Figure 4. Note that the HDD304 and HDD controller305 are examples of storage devices that store programs, data, etc., and may also be SSDs (Solid State Drives) and SSD controllers, respectively.

[0077] The keyboard 311 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting or executing various instructions, selecting processing targets, or moving a cursor, etc. Note that the input means may not be limited to the keyboard 311 and the pointing device 312, but may also be a touch panel or an audio input device, etc. The DVD-RW drive 314 controls the reading or writing of various data to the DVD-RW 313, which is an example of a removable recording medium. Note that the DVD-RW 313 may be a DVD-R or a Blu-ray® Disc, etc. The media I / F 316 controls the reading or writing (storage) of data to the recording medium 315, such as flash memory. The microphone 318 is a type of built-in sound collection means for inputting sound. The sound input / output I / F 317 is a circuit that processes the input and output of sound signals between the microphone 318 and the speaker 319 according to the control of the CPU 301. The short-range communication circuit 320 is a communication circuit for communicating with an external terminal (device) using short-range wireless communication technology such as NFC, Bluetooth (registered trademark), or Wi-Fi. The camera 321 is a type of built-in imaging means that captures an image of a subject and obtains image data. Note that the microphone 318, speaker 319, and camera 321 may be external devices rather than being built into the communication terminal 30.

[0078] Furthermore, the hardware configuration of the communication terminal 30 is not limited to those shown herein; any configuration that can realize the functional configuration of the communication terminal 30 is acceptable. In addition, at least a part of the above hardware configuration may reside on the network.

[0079] <<Hardware configuration of the information processing system>> As shown in Figure 4, each hardware component of the information processing system 50 is indicated by a 500-series code in parentheses. The information processing system 50 is built using a computer and has a configuration similar to that of the communication terminal 30, as shown in Figure 4; therefore, a description of each hardware component is omitted.

[0080] Furthermore, the hardware configuration of the information processing system 50 is not limited to those shown herein; any configuration that can realize the functional configuration of the information processing system 50 is acceptable. In addition, at least a part of the above hardware configuration may reside on a network.

[0081] Furthermore, each of the above programs may be distributed as an installable or executable file recorded on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc (registered trademark), SD card, USB memory, etc. The recording media can also be provided domestically or internationally as a program product. For example, the communication terminal 30 realizes the image display method according to the present invention when the program according to the present invention is executed.

[0082] <Regarding wide-field images and viewpoint information> The following section explains how to generate wide-field images (spherical images) using Figures 5 to 12.

[0083] First, the external appearance of the imaging device 10 will be described using Figure 5. The imaging device 10 is a digital camera for obtaining captured images that will form the basis of a 360° spherical image. Figure 5(a) is a left side view of the imaging device, Figure 5(b) is a front view of the imaging device, and Figure 5(c) is a top view of the imaging device. This external view is merely one example of the imaging device 10, and other external appearances are also possible.

[0084] As shown in Figure 5(a), the imaging device 10 is small enough to be held in one hand, but this shape is merely an example, and other shapes are also possible. Also, as shown in Figures 5(a), 5(b), and 5(c), the upper part of the imaging device 10 is equipped with an image sensor 103a on the front side and an image sensor 103b on the back side. These image sensors 103a and 103b are used in conjunction with optical components (for example, lenses 102a and 102b described later) capable of capturing hemispherical images (angle of view of 180° or more). Also, as shown in Figure 5(b), an operating section 115 such as a shutter button is provided on the side of the imaging device 10 opposite to the front side. As mentioned above, there may be only one image sensor, or there may be three or more.

[0085] Next, the usage of the imaging device 10 will be explained using Figure 6. Figure 6 is an illustrative diagram of the imaging device in use. As shown in Figure 6, the imaging device 10 is used, for example, to image a subject around the imaging device 10. In this case, two hemispherical images can be obtained by imaging the subject around the imaging device 10 using the image sensors 103a and 103b shown in Figure 5, respectively.

[0086] Next, using Figures 7 and 8, we will outline the process from the image captured by the imaging device 10 to the creation of a full-sphere image. Figure 7(a) shows the hemispherical image (front side) captured by the imaging device, Figure 7(b) shows the hemispherical image (back side) captured by the imaging device, and Figure 7(c) shows the image represented by equirectangular projection (hereinafter referred to as "equistratic projection image"). Figure 8(a) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 8(b) shows the full-sphere image.

[0087] As shown in Figure 7(a), the image obtained by the image sensor 103a is a curved hemispherical image (front side) by the lens 102a described later. Similarly, as shown in Figure 7(b), the image obtained by the image sensor 103b is a curved hemispherical image (rear side) by the lens 102b described later. The imaging device 10 then combines the hemispherical image (front side) and the 180-degree inverted hemispherical image (rear side) to create an equirectangular projection image EC as shown in Figure 7(c).

[0088] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to overlay an equirectangular projection image EC so that it covers the sphere, as shown in Figure 8(a), and creates a full-sphere image (full-sphere panoramic image) CE as shown in Figure 8(b). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. OpenGL ES is merely one example of software that performs image processing, and the full-sphere image CE may be created using other software. The full-sphere image CE may be a still image or a video. Here, the imaging device 10 has been described as generating a full-sphere image, but the information processing system 50 or the communication terminal 30 may perform similar image processing or some of the image processing steps.

[0089] As described above, the 360-degree spherical image CE is an image pasted to cover a sphere, which can cause discomfort to the human eye. Therefore, by displaying a predetermined area T (hereinafter referred to as the "predetermined area image") of the 360-degree spherical image CE as a flat image with less curvature in the imaging device 10 or the communication terminal 30, a display that does not cause discomfort to the human eye can be achieved. This will be explained with reference to Figures 9 to 10.

[0090] Figure 9 shows the positions of the virtual camera and a predetermined region when the 360-degree image is treated as a three-dimensional sphere. The virtual camera IC corresponds to the position of the virtual viewpoint of the user viewing the 360-degree image CE, which is displayed as a three-dimensional sphere. Figure 10(a) is a stereoscopic perspective view of Figure 9, and Figure 10(b) shows the predetermined region image as it appears on a display. Figure 10(a) represents the 360-degree image CE shown in Figure 9 as a three-dimensional sphere CS. If the 360-degree image CE generated in this way is a sphere CS, then, as shown in Figure 9, the virtual camera IC is located inside the 360-degree image CE. The predetermined region T in the 360-degree image CE is the imaging area of ​​the virtual camera IC and is identified by predetermined region information indicating the imaging direction and field of view of the virtual camera IC in the three-dimensional virtual space including the 360-degree image CE. Furthermore, zooming in on the predetermined region T can also be represented by moving the virtual camera IC closer to or further away from the 360-degree image CE. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. Therefore, the predetermined region T can be determined by the field of view α and the distance f from the virtual camera IC to the 360-degree spherical image CE (see Figure 11).

[0091] Then, the predetermined region image Q shown in Figure 10(a) is displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in Figure 10(b). The image shown in Figure 10(b) is a predetermined region image represented by the predetermined region information that has been initially set (default). The following explanation will use the imaging direction (ea, aa) and field of view (α) of the virtual camera IC. Note that the predetermined region T may be represented by the position coordinates (X, Y, Z) of the imaging area of ​​the virtual camera IC, which is the predetermined region T, rather than by the field of view α and distance f.

[0092] Next, we will explain the relationship between the predetermined region information and the image of the predetermined region T using Figure 11. Figure 11 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 11, "ea" is the elevation angle, "aa" is the azimuth angle, and "α" is the field of view (Angle). That is, the orientation of the virtual camera IC is changed so that the point of fixation of the virtual camera IC, indicated by the imaging direction (ea,aa), becomes the center point CP(x,y) of the predetermined region T, which is the imaging area of ​​the virtual camera IC. As shown in Figure 11, the center point CP(x,y) when the diagonal field of view of the predetermined region T, represented by the field of view α of the virtual camera IC, is α becomes the parameter (x,y) of the predetermined region information. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. f is the distance from the virtual camera IC to the center point CP(x,y). L is the distance between any vertex of a given region T and the center point CP(x,y) (2L is the diagonal). In Figure 11, the following trigonometric function generally holds:

number

[0093] Figure 12 shows the relationship described in Figure 11 as a point in a three-dimensional Euclidean space using spherical coordinates. Here, the position coordinates of the center point CP shown in Figure 11, expressed in spherical polar coordinates, are (r, θ, φ). (r, θ, φ) are the radial, polar angle, and azimuth angle, respectively. The radial r is the distance from the origin of the three-dimensional virtual space containing the full-sphere image to the center point CP, and is therefore equal to the distance f shown in Figure 11. Figure 12 is a diagram that shows these relationships. Hereafter, the position coordinates (r, θ, φ) of the virtual camera IC will be used as an example of viewpoint information for explanation. The viewpoint information only needs to be parameter information that can identify a predetermined region T (predetermined region image Q) that is displayed as an image of the imaging area of ​​the virtual camera IC on the predetermined display shown in Figure 10, as described above, and includes the coordinates of the diagonal vertices of the predetermined region T. Alternatively, the information indicating the field of view α of the virtual camera IC and the information indicating the center point CP(x, y) described in Figure 11 may also be considered as viewpoint information. Furthermore, the information indicating the field of view α and azimuth angle aa of the virtual camera IC described in Figure 11 may also be considered viewpoint information. In addition, viewpoint information may include not only position coordinate information in spherical coordinates, but also position coordinate information in orthogonal coordinates and coordinate difference values ​​from the initially set (default) predetermined area information. Furthermore, viewpoint information may also include information other than coordinate information, such as angles and distances, as shown in Figure 11. In addition, although the center point of the predetermined area T is used as the reference in Figures 11 and 12, the predetermined area T may also be identified by parameter information based on any of the vertices of the predetermined area T. Note that in the above explanation of viewpoint information, the case where the wide-field image is a 360-degree spherical image was used as an example, but in the case of other wide-field images, the information that identifies the predetermined area T in that wide-field image will be the viewpoint information. Furthermore, viewpoint information may include parameter information such as the height and width of the predetermined area T, and parameter information such as the magnification ratio due to the zoom of the virtual camera IC. Furthermore, if the position of each pixel in the equirectangular projection image EC, as shown in Figure 7(c), is associated with the coordinates of the surface of the sphere (for example, coordinates with two axes: latitude and longitude), then parameter information such as the direction and field of view of the virtual camera IC may be used as viewpoint information, or information such as latitude and longitude may be included in the viewpoint information. Thus, viewpoint information is not necessarily limited to information that indicates a point.

[0094] <About the features> Next, the functional configuration of the communication system 1 according to this embodiment will be described using Figure 13. Figure 13 is a diagram showing an example of the functional configuration of the communication system 1 according to this embodiment. In Figure 13, the terminals, devices, and servers shown in Figure 1 that are related to the processing or operation described later are shown.

[0095] <<Functional Configuration of Imaging Device>> First, the functional configuration of the imaging device 10 will be described with reference to Figure 13. The imaging device 10 includes a communication unit 11, a reception unit 12, an imaging processing unit 13, an analysis unit 14, a registration request unit 15, a connection unit 16, a storage processing unit 17, an image transmission control unit 18, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 111 that follow a program deployed on the SRAM 113 or DRAM 114. The imaging device 10 also has a storage unit 1000 constructed from a ROM 112, etc., as shown in Figure 3.

[0096] The communication unit 11 has the function of connecting to a communication network N using wireless communication means such as Wi-Fi and sending and receiving various data or information with other devices. In this embodiment, the connection unit 16 mainly transmits the wide-field image acquired by the imaging processing unit 13 to the information processing system 50, but it is also possible for the communication unit 11 to transmit the wide-field image to the information processing system 50.

[0097] The reception unit 12 is a function that receives user input for the imaging device 10. The reception unit 12 receives user input such as power on / off, shutter button on / off (start or stop transmission of wide-field images), and input from the user via the touch panel or buttons.

[0098] The imaging processing unit 13 captures images of subjects, landscapes, etc., and acquires the captured images. The captured images acquired by the imaging processing unit 13 may be videos or still images (or both), and may include sound along with the images. The imaging processing unit 13 also captures, for example, a two-dimensional code (see Figure 22) displayed on the display 306 of the communication terminal 30. Furthermore, the imaging processing unit 13 may generate a wide-field image by performing the image processing described in Figures 7 and 8 on the captured images.

[0099] The analysis unit 14 analyzes the two-dimensional code captured and acquired by the imaging processing unit 13 and extracts the information contained in the two-dimensional code (URL for registering the imaging device with the tenant, temporary ID and password).

[0100] The registration request unit 15 uses the information contained in the two-dimensional code read by the analysis unit 14 to send a request to the information processing system 50 to register the imaging device 10 as a tenant of the information processing system 50.

[0101] The connection section 16 is implemented, for example, by a short-range communication circuit 117, and has the function of receiving power supply from the communication terminal 30A and performing data communication.

[0102] The storage processing unit 17 processes the wide-field images captured in response to imaging requests from any location and saves them to a URL (e.g., storage 90) notified by the information processing system 50.

[0103] The image transmission control unit 18 controls the transmission of wide-field images to the information processing system 50. For example, the image transmission control unit 18 transmits captured images acquired by the imaging processing unit 13 to the information processing system 50 periodically or in response to user operation if they are still images, or at a predetermined FPS (Flame Per Second) if they are videos. The image transmission control unit 18 also switches between the communication unit 11 and the connection unit 16.

[0104] The storage / reading unit 19 has the function of storing various data in the storage unit 1000 or reading various data from the storage unit 1000. The storage unit 1000 also stores the image capture data acquired by the image processing unit 13, the image capture device ID, etc. The image capture data stored in the storage unit 1000 may be deleted after a predetermined time has elapsed since it was acquired by the image processing unit 13, or it may be deleted when data has been transmitted to the information processing system 50.

[0105] Furthermore, the imaging device 10 has an application (also called a plug-in) installed to support the communication system 1. This application is used when associating the imaging device 10 with a virtual room and when receiving control from external sources. Some of the functions shown in Figure 13 (for example, the registration request unit 15) are provided by this application. Alternatively, the application for supporting the communication system 1 may be placed on a network, and the same functions may be achieved by accessing the application using a web browser or similar device on the imaging device 10.

[0106] <<Communication terminal function configuration>> Next, the functional configuration of the communication terminal 30 will be explained using Figure 13. The communication terminal 30 has a communication unit 31, a reception unit 32, a display control unit 33, an imaging unit 34, a storage / reading unit 35, and a connection unit 36. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 301 following a program (which may be a web browser or a dedicated application) deployed on the RAM 303. The communication terminal 30 also has a storage unit 3000 constructed from the ROM 302 or recording medium 315 shown in Figure 4.

[0107] The communication unit 31 (an example of a second communication unit) is implemented, for example, by a network I / F 309, connects to a communication network N, and has the function of sending and receiving various data or information with other devices.

[0108] The reception unit 32 has the function of receiving various selections or operation inputs to the communication terminal 30. The display control unit 33 has the function of displaying wide-field images, normal-angle images, and various screens on the display 306 of the communication terminal 30. The display control unit 33 displays, for example, a two-dimensional code transmitted from the information processing system 50 on the display 306. The two-dimensional code may be, for example, QR code (registered trademark), DataMatrix (DataCode), MaxiCode, or PDF417. The two-dimensional code may also be a barcode.

[0109] The connection section 36 is implemented, for example, by a short-range communication circuit 320, and has the function of supplying power to the imaging device 10 and performing data communication.

[0110] The storage / reading unit 35 is executed by instructions from the CPU 301 shown in Figure 4, and has the function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. The storage unit 3000 has an image management information storage unit 3001. The image management information storage unit 3001 will be explained in the description of the information processing system 50.

[0111] <<Functional Configuration of the Information Processing System>> Next, the functional configuration of the information processing system 50 will be described. The information processing system 50 includes a communication unit 51, a screen generation unit 52, an association processing unit 53, an image distribution unit 54, an authentication unit 55, a communication group management unit 56, a communication control unit 57, a connection management unit 58, a storage / reading unit 59, an API management unit 60, a recording unit 61, a point of interest setting reception unit 62, a tracking viewpoint creation unit 63, and a video creation unit 64. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with a program deployed on the RAM 503. The information processing system also has a storage unit 5000 constructed from a ROM 502, HDD 504, or recording media 515 shown in Figure 4.

[0112] The communication unit 51 (an example of the first communication unit) has the function of sending and receiving various data or information with other devices via the communication network N.

[0113] The screen generation unit 52 generates screen information to be displayed by the communication terminal 30. When the communication terminal 30 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc. When the communication terminal 30 runs a native application, the screen information is held by the communication terminal 30, and the information to be displayed is transmitted in XML, etc. The screen generation unit 52 generates screen information on which wide-field images, etc., distributed by the image distribution unit 54 are placed.

[0114] The association processing unit 53 controls the association and sharing of viewpoint information for wide-field images. When the association processing unit 53 receives an imaging request along with viewpoint information from the communication terminal 30, it requests imaging from the imaging device 10 and processes the acquisition of a wide-field image and the association of the viewpoint information. Furthermore, the association processing unit 53 stores the associated wide-field image and viewpoint information in the image management information storage unit 5001. The association processing unit 53 also sends destination information (e.g., a URL) to the communication terminal 30 as information indicating the storage location where the associated wide-field image and viewpoint information are stored. Note that the information processing system 50 does not need to receive viewpoint information and imaging requests from the communication terminal 30 simultaneously; it may receive them separately and then perform the association process. Also, a URL is just one example of destination information indicating a storage location, and other formats such as URIs may be used.

[0115] The image distribution unit 54 distributes images such as wide-field images transmitted by the imaging device 10, which is associated with the same virtual room, to the communication terminal 30 operated by the user who is in the virtual room. Images of the normal field of view captured by the camera on the communication terminal 30 or connected cameras 8 and 9 are also distributed in the same manner. The distributed images include streaming video, video, still images, etc.

[0116] The authentication unit 55 is a function that authenticates the requester based on the authentication request received by the communication unit 51. For example, the authentication unit 55 authenticates the user by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, biometric authentication information such as face, fingerprint or voiceprint, device ID, passcode, access token, security key, ticket, etc. The authentication unit 55 may also authenticate using an external authentication system or an authentication method such as OAuth. Furthermore, the authentication unit 55 may authenticate not only the user but also devices such as an imaging device.

[0117] The communication group management unit 56 manages the entry of communication terminals 30 and users into the virtual room, as well as the mapping of devices. When authentication by the authentication unit 55 is successful, the communication group management unit 56 registers the user ID and the IP address of the communication terminal 30 in the virtual room information storage unit 5002, and maps the imaging device 10 to the virtual room.

[0118] The communication control unit 57 manages the start, establishment, and termination of communication with the imaging device 10 associated with each virtual room. The communication control unit 57 also manages the start, establishment, and termination of communication for distributing wide-field images and audio in response to the communication terminal 30 entering or leaving the virtual room.

[0119] The connection management unit manages the communications (connections) established between the communication terminal 30 and the imaging device 10 and the information processing system 50, associating them with virtual rooms.

[0120] The API management unit 60 manages the APIs used by platform subscribers when providing wide-field image distribution services. When using APIs, platform subscribers only need to develop software to call the APIs separately. The developed software may run on a server or on a client such as a communication terminal. Any function provided by the information processing system 50, such as the image distribution unit 54, association processing unit 53, and communication control unit 57, can be provided as an API. It is also possible to provide functions added to the information processing system 50 later as APIs. Whether or not to provide an API is determined by the communication terminal operated by the platform provider accessing the information processing system 50 and accepting the API disclosure setting, allowing the API management unit 60 to control the API based on the disclosure setting. The API management unit 60 may also perform authentication processing to verify whether the requesting software requesting the API call is software developed by a legitimate platform subscriber. Authentication processing can be performed by comparing information pre-registered and stored as platform subscriber information in the storage unit 5000 with information transmitted from the requesting software. As a specific example of the authentication process, the information processing system 50 receives an application ID, which has been pre-issued by the API management unit 60, from the requesting software developed by the platform contractor. If the API management unit 60 determines that the application ID is stored in the storage unit 5000, the API management unit 60 performs control to permit the provision of the API as legitimate software. On the other hand, if it cannot determine that the software is legitimate, the API management unit 60 performs control to deny the provision of the API. Note that the application ID is just one example of authentication information used to determine legitimacy, and the API management unit 60 may also verify the legitimacy of the requester using authentication information such as an access token, ticket, security key, password, or PIN code that has been pre-issued by the API management unit 60 of the information processing system or an external system.In this embodiment, the form in which the functions provided by the information processing system 50 are used as an API is not described, but the processing flow is the same except that the software such as an application developed by the platform subscriber uses the functions provided by the information processing system 50 via a decision made by the API management unit 60.

[0121] The memory / read unit 59 has the function of storing various data in the memory unit 5000 or reading various data from the memory unit 5000. The memory unit 5000 also includes an image management information storage unit 5001, a virtual room information storage unit 5002, a tenant information storage unit 5003, a recorded video storage unit 5004, a point of interest storage unit 5005, and a mixed video information storage unit 5006.

[0122] The recording unit 61 records a video of the wide-field image viewed by the viewer while changing their viewpoint. The recording unit 61 attaches the viewing viewpoint, which the viewer changed while viewing the recorded video, to the recorded video.

[0123] The focus point setting reception unit 62 accepts from the user whether the focus point of the recorded video should be a fixed point, a viewing point, or a following point when creating a mixed video.

[0124] The tracking viewpoint creation unit 63 creates a tracking viewpoint that follows the selected object when the user sets the tracking viewpoint as a point of interest. Since the recorded video is a wide-field video, the tracking viewpoint creation unit 63 creates tracking viewpoint information that records viewpoints that keep the object within a predetermined area in chronological order.

[0125] The video creation unit 64 creates a hybrid video by combining point-of-interest videos created based on a fixed point, a viewer's viewpoint, or a follow-up viewpoint (none of these may be present, or there may be multiple points of interest).

[0126] "Image Management Information Storage Unit 5001" Figure 14(a) is a conceptual diagram showing the image management information stored in the image management information storage unit 5001. The image management information storage unit 5001 stores the image management information as shown in Figure 14. The image management information is information for managing wide-field images captured in response to an imaging request, and when a user sends an imaging request from the communication terminal 30, one record of image management information is generated. Each item of the image management information will be explained below.

[0127] The data ID is identification information that identifies image data such as wide-field images. The data ID is assigned by the information processing system 50. ID is an abbreviation of Identification and means identifier or identification information. An ID is a name, code, string, number, or a combination of one or more of these used to uniquely distinguish a particular object from multiple objects. Note that the data ID may be associated not only with wide-field images but also with images captured at a normal field of view by the imaging device 10 associated with the virtual room.

[0128] The data name is the name of the wide-field image set by the user of the communication terminal 30. The data name can be set by the user, but it may also be set automatically.

[0129] The imaging date and time information is information used to identify the imaging date and time of the wide-field image, etc., such as the date and time when the user entered an imaging request into the communication terminal 30, and the date and time when the imaging device 10 captured the wide-field image, etc. The imaging date and time information may be substituted with the timestamp information of the wide-field image, etc.

[0130] The imager information is the identification information (including user ID and username) of the user who entered the imaging request into the communication terminal 30. Since the user enters the imaging request into the communication terminal 30 while inside the virtual room, the user registered in the imager information is identified by authentication to the information processing system 50 or the virtual room. The imager information is transmitted to the information processing system 50 along with the imaging request. Note that the imaging request and the imager information do not necessarily have to be transmitted to the information processing system 50 at the same time; they may be transmitted to the information processing system 50 at different times.

[0131] The imaging device information is the identification information (imaging device ID) of the imaging device 10 that captured the wide-field image. The imaging device ID is assigned by the information processing system 50 and shared with the imaging device 10, but information unique to the imaging device 10, such as a MAC address or serial number, may also be used. The imaging device ID is transmitted to the information processing system 50 along with the wide-field image. Note that the imaging device ID and the wide-field image do not necessarily have to be transmitted to the information processing system 50 at the same time; they may be transmitted to the information processing system 50 at different times.

[0132] The viewpoint information of the imager is the viewpoint information specified in the imager's communication terminal 30. For example, the viewpoint information indicates the center coordinates of the wide-field image displayed by the communication terminal 30 and is parameter information used to identify a predetermined area of ​​the wide-field image displayed by the communication terminal 30. Here, radial radius, polar angle, and azimuth angle are shown as examples of parameter information, but other parameter information as described in Figures 10 to 12 may also be used. The viewpoint information is transmitted from the communication terminal 30 requesting imaging. The viewpoint information may also include information specifying the width and height of the display range of the predetermined area. Alternatively, the viewpoint information may only include the width and height of the display range.

[0133] The virtual room ID used during imaging is the identification information of the virtual room to which the imaging device 10 is associated.

[0134] The data storage location information (storage location information) is information indicating where the wide-field image is stored, and may be a URL or file path. The storage location identified by the storage location information may also be information indicating a predetermined folder. The folder may be a folder associated with the virtual room used during imaging. Alternatively, it may be a folder associated with identification information (additional information such as a name) indicating one or more combinations of classifications such as the date and time of imaging, imaging device, imager, and virtual room used during imaging. Furthermore, the data storage location information may be combined with information such as the data ID and data name to identify the data storage location.

[0135] Figure 14(b) is also a conceptual diagram showing image management information. In Figure 14(b), wide-field images with the same virtual room ID at the time of acquisition are stored. In this way, image management information may be classified by virtual room.

[0136] "Virtual Room Information Storage Unit 5002" Figure 15(a) is a conceptual diagram showing the virtual room information stored in the virtual room information storage unit 5002. The virtual room information storage unit 5002 stores virtual room information as shown in Figure 15(a). Virtual room information is information about a virtual room and is maintained for each virtual room. The items contained in the virtual room information will be explained below. Note that although virtual rooms are registered with tenants here, registration with tenants is not mandatory, and information on temporarily created virtual rooms and virtual rooms that can be used for sharing is also stored in the virtual room information storage unit 5002.

[0137] The virtual room ID is identification information that identifies the virtual room. In this embodiment, virtual rooms can be created at the user's discretion.

[0138] The virtual room name is a name used by users to identify a virtual room and can be set by the user at will. The virtual room ID and virtual room name may be the same information.

[0139] The device information is the identification information (device ID) of the device, including the imaging device 10, which is associated with the virtual room. The user currently in the virtual room is the user ID of the user currently in the virtual room. This user is able to view images such as wide-field images distributed to the virtual room entrants. The method of entering the room will be described later. In addition, the user ID may be associated with the IP address of the communication terminal 30 operated by that user. In addition, the username may be associated with and stored as the user ID.

[0140] "Tenant Information Storage Unit 5003" Figure 15(b) is a conceptual diagram showing the tenant information stored in the tenant information storage unit 5003. The tenant information storage unit 5003 stores tenant information as shown in Figure 15(b). Tenant information is information about tenants (user groups) and is maintained for each tenant. The items contained in the tenant information will be explained below. Note that the tenant information includes various other information besides that shown in the diagram, such as user information, and Figure 15(b) is only a part of it. • The tenant ID is identification information that identifies the tenant. The tenant name is a name used by users to identify the tenant. Note that the tenant ID and tenant name may be the same information. The tenant registration virtual room ID is the identification information for the virtual room registered with the tenant. • Tenant registration devices are information about devices registered with the tenant. Furthermore, the tenant information storage unit, tenant ID, tenant name, tenant registration virtual room ID, and tenant registration device can be rephrased as the user group information storage unit, user group ID, user group name, user group registration virtual room ID, and user group registration device, respectively.

[0141] "Recording and Video Memory Unit 5004" Figure 16 is a conceptual diagram showing the recorded video information stored in the recorded video storage unit 5004. The recorded video storage unit 5004 stores recorded video information related to the recorded video as shown in Figure 16. The recorded video information includes the following items: recorded video ID, virtual room ID, recorded video, viewing viewpoint information, storage location information, recording start date and time and end date and time, and viewing viewpoint name. The recorded video ID is identification information that identifies the recorded video. The recorded video is a wide-field-of-view video. The recorded video is recorded while a viewer is watching it (although a viewer is not required), but because it is a wide-field-of-view image, the specific area that was viewed cannot be identified without a viewing viewpoint. The virtual room ID is the identification information of the virtual room to which the imaging device 10 used for recording the recorded video is associated. The recorded video section includes items such as the file name of the recorded video. In Figure 16, the file name is the recording start date and time, but it would be better if the user could set the file name arbitrarily. • Viewing viewpoint information is a file in which viewing viewpoints are associated with time. • The storage location information refers to the location (URL, path, etc.) where the recorded video and viewing viewpoint information are stored. The recording start and end times are the time when the recording of the video footage began (in the standard time of the country or region in question) and the time when the recording stopped. The viewer name is the name, user ID, or nickname of the viewer who recorded the video. The viewing point name serves as identification information to identify the video. The viewing point name does not need to include the viewer's name as long as it can identify the viewer.

[0142] "Points of Interest Memory Unit 5005" Figure 17 is a conceptual diagram showing the point of interest information stored in the point of interest storage unit 5005. As shown in Figure 17(a), the point of interest storage unit 5005 stores point of interest information related to the point of interest. The point of interest information is associated with the recorded video by the recorded video ID. The point of interest information has the following items: point of interest ID, type, and storage location. • The point of interest ID is identification information used to identify each point of interest. The "type" refers to the type of point of interest, and in this embodiment, examples include viewing viewpoint, fixed point, and tracking. The viewing viewpoint is the viewpoint from which the viewer viewed the recorded video when it was being recorded. A fixed point is a viewpoint used to view the recorded video only from a specific viewpoint. Tracking is a viewpoint used to track an object set by the user. Each point of interest is shown in Figures 17(b) to 17(e). • The storage location indicates where the information of the points of interest is stored. Note that if the type is "following," the storage location may indicate either the object features or the following viewpoint.

[0143] Figure 17(b) shows an example of viewing viewpoint information. The viewing viewpoint information includes the elapsed time from the start of recording of the recorded video and the viewing viewpoint associated with that elapsed time. Since the recording start date and time of the recorded video is recorded, the elapsed time can be converted to standard time. The viewing viewpoint identifies a predetermined area viewed by the viewer. In Figure 17(b), the viewing viewpoint is recorded only when it changes, but the viewing viewpoint may be recorded periodically regardless of any changes.

[0144] Figure 17(c) shows an example of a fixed point. Since the fixed point remains constant regardless of time, only one is needed.

[0145] Figure 17(d) shows an example of object features. The type of information that object features represent varies depending on the feature. Generally, features are represented as vectors.

[0146] Figure 17(e) shows an example of tracking viewpoint information. Tracking viewpoint information is the same as viewing viewpoint information in that the viewpoint that tracks the object changes over time, and it has the same structure as viewing viewpoint information.

[0147] "Hybrid video information storage unit 5006" Figure 18 is a conceptual diagram showing the hybrid video information stored in the hybrid video information storage unit 5006. The hybrid video information is information about a hybrid video created by the user by setting points of interest. • The mixed video ID is identification information that identifies each mixed video. • The user ID is the identification information of the user who created the hybrid video. Each user can play at least the hybrid video they created. The recorded video ID is the identification information of the recorded video from which the composite video was created. If a single composite video is created from multiple recorded videos, there will be multiple recorded video IDs. • The format of a composite video differs depending on whether it was created using one of the methods described below, (i) or (ii). Composite video 1 is a composite video created using method (i) described below, and represents a single video. • Hybrid video 2 is a hybrid video created using the method described in (ii) below, and is a format for a web application or native application to play the hybrid video using recorded video and point of interest information, as described below.

[0148] <Entering a virtual room using a communication terminal> Next, referring to Figures 19 and 20, the process of user B entering the virtual room will be explained. It is assumed that the imaging device 10 has already been associated with the virtual room, and that the communication terminal 30A has transmitted wide-field images and normal-angle images to the information processing system 50 (the association of the imaging device 10 with the virtual room will be explained from Figure 21 onwards). Furthermore, in the following explanation, there will be no particular distinction made between user B entering the virtual room and the communication terminal 30B operated by user B entering the virtual room.

[0149] Figure 19 shows an example of the screen displayed by the communication terminal 30B when user B enters a virtual room. Figure 19(a) is an example of the entry screen 200. Prior to the display of the entry screen 200, user B is logged into the information processing system 50. Logging in identifies the tenant to which user B belongs. Virtual rooms are associated with tenants. User B displays a list of virtual rooms associated with the tenant on the communication terminal 30B (see Figure 24) and selects a virtual room to enter from the list. Figure 19(a) is the entry screen 200 for the virtual room selected by user B in this way. Note that temporarily created virtual rooms or shared virtual rooms not associated with a tenant may also be displayed on the screen in Figure 19(a).

[0150] Alternatively, the creator of the virtual room may request the information processing system 50 to issue a URL corresponding to the virtual room and send this URL to user B via email or other means. When user B clicks the URL displayed on the communication terminal 30B, the communication terminal 30B displays the room entry screen 200 shown in Figure 19(a).

[0151] The entry screen 200 has a virtual room name 201, a participant name input field 202, and an entry button 203. The virtual room name 201 is the same as the one stored in the virtual room information storage unit 5002. The participant name input field 202 is a field for entering the username to be displayed in the virtual room, which may be a nickname or other designation for user B. When user B logs in, the username associated with user B's user ID is identified, and this identified username may be displayed automatically. The entry button 203 is a button for user B to request entry into the virtual room.

[0152] Furthermore, authentication for entering the virtual room may be required separately from the tenant login process.

[0153] Figure 19(b) shows the image viewing screen 210 displayed by the communication terminal 30B when user B enters the virtual room. In the image viewing screen 210 of Figure 19(b), the imaging device 10 has already started distributing wide-field images via the information processing system 50, and the communication terminal 30A has already started distributing images with a normal field of view. Therefore, the image viewing screen 210 has a first image section 211 and a second image section 212. The first image section 211 displays the wide-field image, and the second image section 212 displays the image with a normal field of view. If there are three or more locations transmitting images, the image viewing screen 210 is divided according to the number of transmitting locations.

[0154] The first image field 211 displays a wide-field image mark 213. The wide-field image mark 213 is set by the screen generation unit 52 of the information processing system 50 when it determines that the image to be displayed in the first image field 211 is a wide-field image. The communication terminal 30B may also determine and display it. By seeing the wide-field image mark 213, user B can understand that a wide-field image that allows the viewpoint to be changed is being delivered. In addition, the first image field 211 displays the device name 214 (transmitted from the imaging device 10 along with the wide-field image). The device name 214 is information set by user A, etc., as will be described later (see Figure 22).

[0155] The second image field 212 displays the participant name 215. The participant name 215 is a username, and the participant name of a user who has already entered the room (in this case, user A has already entered, so "AAA" entered by user A in the participant name input field 202) is displayed in the participant name input field 202.

[0156] Figure 20 is a sequence diagram illustrating the process by which user B (or communication terminal 30B) enters a virtual room.

[0157] S1: First, user B at site B performs an operation to display the virtual room list screen. The communication terminal 30B has previously accessed the information processing system 50 in response to the operation by user B and has received virtual room information stored in the virtual room information storage unit 5002 for displaying the virtual room list screen from the information processing system 50. At this time, the communication terminal 30B may be authenticated by the authentication unit 55 of the information processing system 50 by sending authentication information necessary for login, etc. to the information processing system 50. The authentication information may be authentication information associated with user B or authentication information associated with the communication terminal 30B. In such cases, the virtual rooms displayed on the list screen may be virtual rooms registered in the tenant associated with user B or virtual rooms registered in the tenant associated with the communication terminal 30B. When the reception unit 32 receives the operation to display the list screen, the display control unit 33 of the communication terminal 30B displays the selection screen on the display 306.

[0158] S2: When user B selects a virtual room selection button, the reception unit 32 of the communication terminal 30B accepts the selection of the virtual room. The display control unit 33 of the communication terminal 30B displays the room entry screen 200 shown in Figure 19(a) on the display 306.

[0159] S3: User B enters the required information and presses the enter button 203. Upon receiving the press from the reception unit 32, the communication unit 31 of the communication terminal 30B sends an entry request to the information processing system 50. This entry request includes information such as the virtual room ID indicating the virtual room selected in step S2, the user ID of user B authenticated by login, etc., and the IP address of the requesting terminal, the communication terminal 30B.

[0160] S4: As a result, the communication unit 51 of the information processing system 50 receives an entry request. The communication group management unit 56 registers the user ID and IP address authenticated by login, etc., in the virtual room information identified by the virtual room ID in the virtual room information storage unit 5002.

[0161] S5: The communication unit 51 of the information processing system 50 then sends a response to the communication terminal 30B indicating that it has entered the room. As a result, the communication unit 31 of the communication terminal 30B receives the response indicating that it has entered the room. Following S5, the display control unit 33 of the communication terminal 30B receives the screen information generated by the screen generation unit 52 of the information processing system 50 and the image information distributed by the image distribution unit 54, and displays the image viewing screen 210 shown in Figure 19(b) based on the received information.

[0162] <Mapping of imaging equipment to rooms> Next, the mapping of the imaging device 10 to the virtual room will be explained with reference to Figures 21 to 28. Note that this explanation assumes that user A at site A performs the mapping of the imaging device 10 to the virtual room, but this can also be done by a system administrator, tenant administrator, etc.

[0163] Figure 21 shows an example of a device registration screen 220 displayed by the communication terminal 30A. User A is authenticated by logging into the information processing system 50. Logging in identifies the tenant to which User A belongs. User A requests the information processing system 50 to display the device registration screen 220, and the communication terminal 30A displays the device registration screen 220 received from the information processing system 50. The device is first registered with the tenant via the device registration screen 220.

[0164] The device registration screen 220 has buttons for registering an imaging device 221, a VR goggles 222, and smart glasses 223. Buttons are provided for each type of device because there are differences in the presence or absence of a camera, the information used for registration, etc. Furthermore, since devices are registered according to their type, the information processing system 50 can also determine the type of device.

[0165] The imaging device registration button 221 is for user A to register the imaging device 10, the VR goggles registration button 222 is for registering the VR goggles 89, and the smart glasses registration button 223 is for registering the smart glasses 88.

[0166] Figure 22 shows an example of a screen displayed when the imaging device registration button 221 is pressed. Figure 22(a) shows the imaging device registration dialog 230. The imaging device registration dialog 230 has a name field 231 for the imaging device 10, a description field 232, and a next button 233. User A sets an arbitrary name in the name field 231 for the imaging device 10 so that it is clear that it is the imaging device 10 to be registered, and sets a description in the description field 232.

[0167] When user A presses the next button 233, the communication terminal 30A requests a two-dimensional code from the information processing system 50, and the communication terminal 30A displays the two-dimensional code.

[0168] Figure 22(b) shows an example of a two-dimensional code screen 240 displayed by the communication terminal 30A. The two-dimensional code screen 240 contains a message 241 that reads, "Please scan the following two-dimensional code to register the device named XX (the name entered in the name field)," and a two-dimensional code 242. User A scans the two-dimensional code 242 with the imaging device 10 that they wish to register. The two-dimensional code 242 contains authentication information necessary for registration, such as the URL to which the imaging device 10 connects for its own registration, and a temporary ID and password.

[0169] When user A scans the two-dimensional code 242 with the imaging device 10, the imaging device 10 connects to a URL and is authenticated with a temporary ID and password. If authentication is successful, the official imaging device ID is exchanged, and the name, description, and imaging device ID of the imaging device 10 are registered with the tenant. The imaging device 10 also retains this imaging device ID, name, and description. The imaging device 10 registered with the tenant is associated with a virtual room according to the operations of user A described later. Note that the two-dimensional code 242 is just one example of code information; any code with similar authentication information embedded may be used, and other forms of codes such as barcodes may also be used.

[0170] Next, with reference to Figure 23, an example of how to register communication terminals such as VR goggles 89 and smart glasses 88 to a tenant will be explained. Figure 23 is an example of the VR goggles registration screen 250 that is displayed when the VR goggles registration button 222 is pressed. The VR goggles registration screen 250 has a temporary code input field 251 and a secret input field 252.

[0171] If the VR goggles 89 do not have a camera, they cannot capture the QR code. For this reason, user A may have the VR goggles 89 output (display) a temporary code (temporary ID) and a secret (password), and enter them into the temporary code input field 251 and the secret input field 252. The communication terminal 30A registers the VR goggles 89 with the tenant by transmitting the temporary code and secret to the information processing system 50. The VR goggles 89 connects to the information processing system 50 and receives authentication by transmitting the temporary code and secret. If authentication is successful, a formal VR goggle ID is exchanged, and the VR goggle ID is registered with the tenant. The VR goggles 89 also holds this VR goggle ID. The VR goggles 89 registered with the tenant is associated with a virtual room according to the operation of user A described later. The smart glasses 88 will be explained in detail later, but user A can register them in the same way as the imaging device 10 or the VR goggles 89. Note that the temporary code and secret are just examples of authentication information, and other information may be used as authentication information. The imaging device ID, VR goggles ID, and smart glasses ID are examples of device IDs and can therefore be referred to simply as "device ID." Thus, when registering devices other than the imaging device 10, VR goggles, and smart glasses, the same procedure can be used to associate the device ID with virtual rooms and tenants. The device ID may also be identification information linked to the device owner.

[0172] Figure 24 shows an example of a virtual room mapping screen (part 1) 260 for associating the imaging device 10 with a virtual room. The screen configuration can be the same for VR goggles 89 and smart glasses 88. The virtual room mapping screen (part 1) 260 has a virtual room list 261. The virtual room list 261 displays individual virtual room fields 262 to 264 based on the virtual rooms created in the tenant. Each individual virtual room field 262 to 264 has a link issuance button 265, an entry button 266, a settings button 267, and a virtual room name 268. The link issuance button 265 is a button for issuing a link to the virtual room (URL for invitation) and a passcode. The entry button 266 is a button for user A to enter the virtual room. The settings button 267 is a button for associating the imaging device 10 with the virtual room. The virtual room name 268 is the same as that stored in the virtual room information storage unit 5002. Therefore, user A presses the settings button 267. Pressing the settings button 267 causes the communication terminal 30A to display the virtual room mapping screen (part 2) 270.

[0173] Additionally, if a device is already associated with a virtual room, the device name 269 will be displayed in the individual virtual room section (individual virtual room section 264 in the diagram).

[0174] Figure 25 shows an example of the virtual room mapping screen (part 2) 270. Note that the virtual room mapping screen (part 2) 270 is displayed as a pop-up on the virtual room mapping screen (part 1) 260. The screen transition from the virtual room mapping screen (part 1) 260 to the virtual room mapping screen (part 2) 270 does not go through the information processing system 50, but a screen transition that does go through the information processing system 50 is also possible.

[0175] The virtual room mapping screen (part 2) 270 has the name 271 of the imaging device 10 currently (already) mapped to the virtual room (not yet registered, so not shown in the diagram), a connection button 272, and a storage button 273. The connection button 272 is a button that displays a list of devices registered in the tenant as mapping candidates for mapping a device to a virtual room. The storage button 273 is a button that displays a list of storage 90s that store wide-field images and normal-angle images captured by the imaging device 10 mapped to the virtual room. The list of storage 90s may include not only a list of storage 90s mapped to the virtual room, but also a list of specific storage locations such as folders on the storage 90. By selecting a predetermined storage 90 or a specific storage location such as a folder on the storage 90, the storage 90 can be mapped to the virtual room. The information of the storage 90 mapped in this way (address information for accessing the storage 90 and storage locations such as folders on the storage 90) can be stored in the virtual room information storage unit 5002 in association with the virtual room ID. When the connect button 272 is pressed, the communication terminal 30A displays the virtual room mapping screen (part 3).

[0176] The communication terminal 30A transmits the virtual room ID to the information processing system 50 and obtains the name of the device registered in the tenant where the virtual room is created (including the device ID, etc.), and the name of the device associated with the virtual room (including the device ID, etc.).

[0177] Figure 26 shows an example of the virtual room mapping screen (part 3) 280. The virtual room mapping screen (part 3) 280 has the name 281 of the imaging device 10 currently (already) mapped to the virtual room, a list of additional devices 282, and a save button 283. User A selects a device that they want to add to the virtual room from the list of additional devices 282 and presses the save button 283. This maps the device to the virtual room (the device ID, such as the imaging device ID, is registered in the virtual room information storage unit 5002). Note that, as shown in Figure 26, the number of imaging devices that can be mapped to a virtual room may be limited. For example, if the upper limit is two devices, the number of remaining devices that can be added may be displayed on the virtual room mapping screen (part 3) by referring to the number of imaging device IDs already registered in the virtual room information storage unit 5002.

[0178] <Processing to initiate transmission of wide-field images to the imaging device> With the above steps, devices such as the imaging device 10 are now associated with the virtual room, but user A needs to initiate image transmission to the device.

[0179] For the VR goggles 89 and smart glasses 88, user A turns image transmission on and off by operating the device itself. This is because currently, there is no dedicated application running on the communication system 1 for the VR goggles 89 and smart glasses 88. If a dedicated application were running on the communication system 1 for the VR goggles 89 and smart glasses 88, user A would be able to remotely turn image transmission on and off.

[0180] In the case of the imaging device 10, if the application is enabled, user A can enter the virtual room and turn the transmission of wide-field images on or off from the menu.

[0181] Figure 27 shows an example of a wide-field image transmission control dialog 290 displayed by the communication terminal 30A. The wide-field image transmission control dialog 290 is displayed as a pop-up on the image viewing screen 210. Assume that user A has entered a virtual room associated with the imaging device 10 by operating the communication terminal 30A. The wide-field image transmission control dialog 290 displays the name 292 of the imaging device 10 associated with this virtual room. A toggle button 291 is displayed near the name 292, and user A can operate the toggle button 291 to turn on (start transmission) or off (stop transmission) the transmission of wide-field images by the imaging device 10. Note that the method of setting on or off using the toggle button is just one example, and it is sufficient if it can be set according to user input. For example, the user may set it by selecting radio buttons or predetermined icons, or by operating a menu. Alternatively, the transmission of wide-field images may start automatically after the imaging device 10 enters the room, without requiring user operation. Alternatively, certain conditions such as the date and time, the number of users who entered the room, or whether a specific user participated may be predetermined, and the transmission of wide-field images may begin when it is determined that these conditions have been met.

[0182] The communication terminal 30A transmits transmission control setting information to the information processing system 50 by operating the toggle button 291. The information processing system 50 transmits a transmission start request or a transmission stop request to the imaging device 10 according to the transmission control setting information.

[0183] Figure 27(a) shows the state where the toggle button 291 is set to OFF. Therefore, the wide-field image is not displayed in Figure 27(a). On the other hand, in Figure 27(a), when the communication terminal 30A enters the room, the image of the normal field of view captured by the camera 9 of the communication terminal 30A has already been shared and is displayed on the image viewing screen 210.

[0184] Figure 27(b) shows the state where the toggle button 291 is set to ON. When the toggle button 291 is turned ON, the information processing system 50 sends a transmission start request to the imaging device 10, and the imaging device 10 starts transmitting the wide-field image. As a result, two images are shared in one virtual room, and the image viewing screen 210 is divided into two. Also, when the setting is changed from ON to OFF, the communication terminal 30A sends OFF setting information, and the information processing system 50, upon receiving the OFF setting information, sends a transmission stop request to the imaging device 10, and the imaging device 10 stops transmitting the wide-field image.

[0185] As explained in Figure 28, even if a user is on-site, the imaging device 10 can be associated with a virtual room through a simple operation such as capturing code information with the imaging device 10. Since on-site users may not have a PC, the ability to perform the association process on the spot, using only pre-issued code information and the imaging device 10, is particularly useful for on-site users. Furthermore, by performing the association process in advance, the user can connect the imaging device 10 to a designated virtual room without having to select a virtual room, and the start or stop of transmission can be instructed from a remote location, thus reducing the burden on users who want to concentrate on their work on-site. Therefore, a system can be provided that enables efficient communication between the on-site and remote locations even during the preparation process.

[0186] <<Procedure for registering the imaging device in the virtual room>> Next, referring to Figure 28, the procedure for registering the imaging device 10 to the virtual room, as described in the series of screen transitions in Figures 21 to 27, will be explained. Figure 28 is an example of a sequence diagram showing the procedure for user A to register the imaging device 10 to the virtual room.

[0187] S11: First, user A connects the communication terminal 30A to the information processing system 50, enters authentication information (user ID, password, etc.), and requests login. The reception unit 32 of the communication terminal 30A accepts the operation.

[0188] S12: The communication unit 31 of the communication terminal 30A sends a login request to the information processing system 50, specifying authentication information. The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 55 performs authentication based on the specified authentication information. Here, we assume that the authentication was successful. At this time, the information processing system 50 can also identify the tenant ID associated with the authenticated user ID by referring to the tenant information storage unit 5003.

[0189] S13: In response to user operation, the screen generation unit 52 of the information processing system 50 generates a device registration screen 220, and the communication unit 51 transmits the screen information of the device registration screen 220 to the communication terminal 30A.

[0190] S14: The communication unit 31 of the communication terminal 30A receives screen information from the device registration screen 220, and the display control unit 33 displays the device registration screen 220 shown in Figure 21. User A selects the type of device (here, the imaging device 10 (for example, a 360-degree camera) is selected), and then enters the name and description of the imaging device 10 as shown in Figure 22. The reception unit 32 accepts the input.

[0191] S15: The communication unit 31 of the communication terminal 30A sends a request for code information (e.g., a two-dimensional code) to the information processing system 50, specifying the name and description entered by user A.

[0192] S16: The communication unit 51 of the information processing system 50 receives a request for code information (e.g., a two-dimensional code). The communication group management unit 56 generates a URL (a connection destination for registration) associated with the name and description, and generates code information (e.g., a two-dimensional code) including the URL, a temporary ID, and a password. The communication unit 51 of the information processing system 50 transmits the code information (e.g., a two-dimensional code) to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the code information (e.g., a two-dimensional code), and the display control unit 33 displays the code information (e.g., a two-dimensional code) as shown in Figure 22.

[0193] S17: Next, user A operates the imaging device 10 that they want to associate with the virtual room to capture code information (e.g., a two-dimensional code). The reception unit 12 of the imaging device 10 receives the operation.

[0194] S18: The imaging processing unit 13 of the imaging device 10 generates image data by performing imaging processing on the target to be imaged, which includes code information (e.g., a two-dimensional code), and the analysis unit 14 analyzes the image data to extract a URL, a temporary ID, and a password. As a result, the registration request unit 15 connects to the URL via the connection unit 16 and sends a registration request from the imaging device 10 to the information processing system 50, specifying the temporary ID and password. Note that if the registration method using the registration screen described in Figure 23 is implemented, the code information is not imaged, so the imaging device 10 is replaced by a communication terminal such as VR goggles 89 or smart glasses 88, and steps S15 to S17 can be omitted.

[0195] S19: The communication unit 51 of the information processing system 50 receives a temporary ID and password, and the authentication unit 55 determines whether they match the temporary ID and password associated with the connected URL. Here, it is assumed that the temporary ID and password match.

[0196] S20: The communication group management unit 56 of the information processing system 50 receives a request to register the imaging device 10. As an example of a device ID, it generates an imaging device ID and registers it in the tenant corresponding to the tenant ID identified when user A logged in. The imaging device ID is associated with a name and description. Specifically, the communication group management unit 56 refers to the tenant information storage unit 5003 and adds the imaging device ID to the tenant registration devices associated with the identified tenant ID and registers it. Although the communication group management unit 56 generates and registers the imaging device ID here, it may also register the imaging device ID received from the imaging device 10. If a communication terminal such as VR goggles 89 or smart glasses 88 is to be registered in the tenant instead of the imaging device 10, the corresponding device ID can be registered in the tenant information storage unit 5003 using the same procedure.

[0197] S21: The communication unit 51 of the information processing system 50 transmits the imaging device ID to the imaging device 10. The connection unit 16 of the imaging device 10 receives the imaging device ID and stores it in the storage unit 1000.

[0198] S22: The communication terminal 30A is notified of the completion of registration from the communication unit 51 of the information processing system 50, which allows user A to start associating the imaging device 10 with a virtual room. User A displays the virtual room association screen (part 1) 260 on the communication terminal 30A and selects the virtual room to which the imaging device 10 registered with the tenant should be associated. The reception unit 32 of the communication terminal 30A receives the operation input indicating the selection. Specifically, when the reception unit 32 of the communication terminal 30A receives the operation input from user A, the display control unit 33 displays the virtual room association screen (part 1) 260. At this time, the communication unit 31 may send a screen update request to the communication control unit 57 of the information processing system 50. When the information processing system 50 receives the update request, it refers to the tenant information storage unit 5003 and identifies the virtual room ID registered with the tenant associated with the authenticated user ID. The information processing system 50 then refers to the virtual room information storage unit 5002 and obtains the virtual room name associated with the identified virtual room ID. The information processing system 50's communication unit 51 transmits the identified virtual room ID and corresponding virtual room name information (which may also be information from a screen generated by the screen generation unit 52 based on this information) to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the virtual room ID and virtual room name information, and the display control unit 33 can update and display the virtual room mapping screen (part 1) 260 based on the received information. Note that such information can be identified based on the user ID, so it may also be received in S13 after authentication. The reception unit 32 receives an operation input from user A indicating a selection on the virtual room mapping screen (part 1) 260 displayed in this way, allowing the communication terminal 30A to identify the selected virtual room ID.

[0199] S23: Next, user A displays the virtual room mapping screen (part 2) 270 on the communication terminal 30A and presses the connect button 272 to map an additional device to a virtual room. The reception unit 32 of the communication terminal 30A receives the operation input indicating the press. Specifically, the display control unit 33 of the communication terminal 30A displays the virtual room mapping screen (part 2) 270 corresponding to the selected virtual room ID identified in S22. Furthermore, the reception unit 32 receives the instruction from user A to map an additional device to a virtual room (press of connect button 272).

[0200] S24: In response to the operation input in S23, the communication unit 31 of the communication terminal 30A requests information from the information processing system 50 regarding devices registered with tenants that are candidates for devices to be associated with the virtual room, and devices that have already been associated with the virtual room ID selected in step S22.

[0201] S25: The communication unit 51 of the information processing system 50 receives a request for information on devices registered with the tenant and devices associated with the selected virtual room ID, and the screen generation unit 52 generates a virtual room mapping screen (part 3) 280 that includes the device IDs of the devices registered with the tenant and devices associated with the selected virtual room ID. The communication unit 51 of the information processing system 50 transmits the screen information of the virtual room mapping screen (part 3) 280 to the communication terminal 30A.

[0202] S26: The communication unit 31 of the communication terminal 30A receives screen information from the virtual room mapping screen (part 3) 280, and the display control unit 33 displays the virtual room mapping screen (part 3) 280. User A selects a device to associate with the virtual room (here, the imaging device 10 is used as an example). The reception unit 32 of the communication terminal 30A accepts the selection, and the imaging device ID is identified as the device ID of the selected device.

[0203] S27: The communication unit 31 of the communication terminal 30A sends an association request to the information processing system 50, specifying the virtual room ID selected in step S22 and the device ID (e.g., imaging device ID) selected in S26.

[0204] S28: The communication unit 51 of the information processing system 50 receives the mapping request, and the communication group management unit 56 registers the device (e.g., imaging device 10) in the virtual room. That is, the communication group management unit 56 refers to the virtual room information storage unit 5002 and registers the device ID (e.g., imaging device ID) associated with the virtual room ID specified in the request of S27.

[0205] S29: Since a device ID (e.g., imaging device ID) has been associated with a virtual room, the communication unit 51 of the information processing system 50 transmits the virtual room ID, name, and description to the imaging device 10. The information processing system 50 may use push notifications or transmit the information by utilizing polling by the imaging device 10. The connection unit 16 of the imaging device 10 receives the virtual room ID, name, and description and stores them in the storage unit 1000. This allows the imaging device 10 to add the imaging device ID, virtual room ID, name, description, etc. when transmitting wide-field images. Other devices besides the imaging device 10 can also be associated with virtual rooms using the same procedure. Furthermore, the communication unit 51 of the information processing system 50 may send a notification to the communication terminal 30A indicating that the association is complete. From this step onward, devices registered with a virtual room (imaging device 10) can connect to the associated virtual room. Here, we will continue the explanation assuming that the imaging device 10 has connected to the virtual room by sending a connection request to the information processing system 50, specifying the virtual room ID received in S29. However, the timing of the imaging device 10 connecting to the virtual room can be changed by user operation.

[0206] S30: The communication terminal 30A and the information processing system 50 perform the entry process described in Figure 20, thereby allowing the communication terminal 30A to enter the virtual room associated with the device (imaging device 10).

[0207] S31: After entering the room, user A turns on the toggle button 291 of the imaging device 10 associated with the virtual room on the image viewing screen 210. The reception unit 32 of the communication terminal 30A receives the ON signal.

[0208] S32: The communication unit 31 of the communication terminal 30A sends a request to the information processing system 50 to start transmitting wide-field images, specifying the device ID (imaging device ID). Alternatively, user A may directly start transmitting wide-field images by operating a button on the imaging device 10. In addition, user A may also cause the communication unit 31 of the communication terminal 30A to send a request to stop transmission to the information processing system 50.

[0209] S33: The communication unit 51 of the information processing system 50 receives a transmission start request and requests the imaging device 10, identified by its device ID (imaging device ID), to start transmission. The information processing system 50 may use push notifications or the imaging device 10 may use polling. The connection unit 16 of the imaging device 10 receives the transmission start request and the imaging processing unit 13 starts imaging. The image transmission control unit 18 repeatedly transmits wide-field images via the connection unit 16 at a fixed FPS or an FPS corresponding to the bandwidth. Therefore, the communication terminal 30 that has entered the virtual room can display the status of site A on the image viewing screen 210 in real time.

[0210] <Distribution of wide-field images, etc.> Referring to Figure 29, the process for sharing wide-field images and normal-angle images will be explained. Figure 29 is an example of a sequence diagram illustrating the process for sharing wide-field images. In Figure 29, communication terminals 30A and 30B have completed the entry process described in Figure 20 and have entered the virtual room. Communication terminal 30A also has a normal-angle camera 9, which is shared with communication terminal 30B. Instead of the camera 9 of communication terminal 30A, images captured by smart glasses 88 associated with the virtual room may also be shared. Also in Figure 29, the imaging device 10 has already been connected to the same virtual room through the registration procedure described in Figure 28.

[0211] S41: The imaging unit 34 of the communication terminal 30A captures an image, and the communication unit 31 specifies the virtual room ID in which the user is present and transmits the video and audio, including the captured image, to the information processing system 50.

[0212] S42, S43: When the communication unit 51 of the information processing system 50 receives video and audio including images, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A and 30B that are in the same virtual room from the virtual room information storage unit 5002 and transmits the video and audio including images via the communication unit 51. In Figure 29, the communication unit 31 of the communication terminal 30A receives and displays an image with a normal field of view from the information processing system 50, but it may also display an image with a normal field of view captured by the imaging unit 34 without receiving it.

[0213] S44: Next, in response to a transmission start request based on the transmission start setting, the imaging device 10 captures a wide-field image, and the image transmission control unit 18 transmits the video and audio, including the wide-field image, to the information processing system 50 via the connection unit 16, specifying the virtual room ID, imaging device ID, name, and description to which the device is registered.

[0214] S45, S46: When the communication unit 51 of the information processing system 50 receives video and audio including wide-field images, the image distribution unit 54 obtains the IP addresses of communication terminals 30A and 30B that are in the same virtual room from the virtual room information storage unit 5002 and transmits video and audio including wide-field images via the communication unit 51.

[0215] S47: Next, the communication terminal 30C equipped with camera 9 entered the new virtual room by performing the entry process described in Figure 20.

[0216] S48: The communication unit 31 of the communication terminal 30C transmits video and audio, including images with a normal field of view, to the information processing system 50.

[0217] S49~S51: The communication unit 51 of the information processing system 50 receives video and audio, including images with a normal field of view, from the communication terminal 30C, obtains the IP addresses of the communication terminals 30A~30C that are in the same virtual room from the virtual room information storage unit 5002, and the image distribution unit 54 transmits video and audio, including images with a normal field of view.

[0218] S52: In addition, the communication unit 51 of the information processing system 50 transmits video, including wide-field images, and audio to the communication terminal 30C that has entered the same virtual room.

[0219] In this way, users A and B who have entered the same virtual room can share video, including wide-field images, captured by the imaging device 10 associated with the virtual room, in real time. Note that the transmission order of each image shown in Figure 29 is just an example; the wide-field images may be shared first, or the images with a normal field of view may be shared first.

[0220] Let's add some details about the smart glasses 88 and VR goggles 89. The smart glasses 88 have a camera and display function with a normal field of view. Images with a normal field of view captured by the camera held by the smart glasses 88 are distributed in the same way as cameras 8 and 9. The display function held by the smart glasses 88 is flat, like a normal display, so a portion of the wide-field image is displayed at the viewpoint indicated by the user. The VR goggles 89 have a display function (they may also have a camera with a normal field of view). The display function held by the smart glasses 88 projects a wide-field image at a viewpoint determined by the orientation of the user's head, so a portion of the wide-field image is displayed at a viewpoint corresponding to the orientation of the user's head. While viewing a wide-field image with the smart glasses 88 or VR goggles 89, the user can send an imaging request specifying the viewpoint information being viewed to the information processing system 50.

[0221] <Generating hybrid images based on multiple points of interest> Conventionally, in technologies for simultaneously playing multiple videos focusing on multiple points of focus, the communication terminal 30 does not play multiple videos created using different types of points of focus in parallel. Conventionally, it has not been possible to create multiple videos depending on settings such as whether to use a fixed point of focus, an arbitrary object as the point of focus, or the viewing perspective of another person as the point of focus. Therefore, the communication system 1 of this embodiment allows the user to set various types of points of focus and enables the parallel playback of videos with various types of points of focus.

[0222] Furthermore, until now, users could only set their preferred viewpoint, resulting in the generation of videos from multiple viewpoints tailored to individual preferences. However, the points of focus of experts are generally useful in any field, and it is expected that viewing wide-angle images from other people's viewpoints will lead to new discoveries and insights. Therefore, in this embodiment, videos are played simultaneously from multiple viewpoints, including not only the user's set viewpoint but also objective viewpoints such as those of others.

[0223] <Outline of creating a hybrid video> Figure 30 is a diagram illustrating the general process of generating a hybrid video. Figure 30(a) shows a wide-field image in an undistorted state for convenience. This wide-field image is a video. Viewer A and Viewer B each view the wide-field image, which is streamed in real time, while arbitrarily changing their viewpoints 151 and 152, and each recorded video. Each recorded video is associated with the viewing viewpoints of Viewer A and Viewer B.

[0224] From the recorded video footage saved in this way, the user can create a hybrid video in which videos viewed from multiple different points of interest can be played in parallel. The user can set whether or not to use the viewpoints of viewers A and B (hereinafter referred to as viewing viewpoints), whether or not to use a tracking viewpoint that automatically follows the object and changes viewpoint, and whether or not to use a fixed point where the viewpoint does not move, etc.

[0225] The information processing system 50 creates a hybrid video using two or more points of interest, such as a viewing viewpoint, a tracking viewpoint, and a fixed point, according to the user's settings regarding points of interest. Figure 30(b) schematically shows the playback of the hybrid video 153. In the hybrid video 153, point of interest videos 154, each with a different point of interest as a predetermined area, are played in parallel within a single screen. In Figure 30(b), a point of interest video 154 showing a crane lifting building materials as the viewing viewpoint for viewer A, and a point of interest video 154 showing a car moving as the viewing viewpoint for viewer B are played within a single hybrid video 153.

[0226] Thus, the information processing system of this embodiment allows users to set various types of points of focus, and can play multiple point-of-interest videos created using different types of points of focus in parallel. Furthermore, the communication terminal 30 can display not only the user's subjective viewpoint but also videos from objective viewpoints, such as the viewpoints of other viewers, in parallel. By referring to the viewpoints of experts and others, each user can gain new discoveries and insights.

[0227] <Recording of documentary footage> First, with reference to Figure 31, the process by which the viewer records the video along with their viewing viewpoint will be explained. Figure 31 is a sequence diagram showing how the communication terminal 30 records the video viewed by the viewer during remote communication. The communication terminal 30 has already viewed the real-time wide-field image captured by the imaging device 10.

[0228] S101: The imaging device 10 registered in the virtual room entered by the viewer repeatedly transmits wide-field images and audio to the information processing system 50.

[0229] S102: The communication unit 51 of the information processing system transmits wide-field images and audio to the communication terminals 30 that are in the virtual room. There may be two or more communication terminals 30. The communication unit 31 of the communication terminals 30 receives the wide-field images and audio, and the display control unit 33 displays them as a video.

[0230] S103: The viewer initiates recording by pressing the record button on the web application running on the communication terminal 30. The record button is used to save the recorded video, which is a wide-field image, along with the viewing viewpoint. The reception unit 32 receives the press of the record button.

[0231] S104: The communication unit 31 of the communication terminal 30 requests the information processing system 50 to start recording.

[0232] S105: The communication unit 51 of the information processing system receives a request to start recording, and the recording unit 61 starts recording (moving) wide-field images.

[0233] S106: Viewers can view wide-field images displayed by the web application while arbitrarily changing the viewpoint.

[0234] S107: The reception unit 32 receives the viewpoint changes made by the viewer as needed, and the display control unit 33 displays a predetermined area of ​​the wide-view image on the display in response to the operation. Therefore, the predetermined area moves in response to the viewer's operation.

[0235] S108: The communication unit 31 of the communication terminal 30 transmits viewing viewpoint information, including the viewing viewpoint operated by the viewer, to the information processing system 50 each time it is changed. As described above, in the viewing viewpoint information, the viewing viewpoint is associated with the elapsed time of recording based on the recording start time. The viewing viewpoint information may also be transmitted when recording stops.

[0236] S109: The viewer presses the stop recording button on the web application running on the communication terminal 30 as an operation to end the recording. The reception unit 32 receives the press of the stop recording button.

[0237] S110: The communication unit 31 of the communication terminal 30 requests the information processing system 50 to stop recording.

[0238] S111: The communication unit 51 of the information processing system receives a request to stop recording, and the recording unit 61 stops recording the video.

[0239] S112: The recording unit 61 stores, in the recording video storage unit 5004, the recorded video and the browsing viewpoint information recorded from the start to the stop of recording, in association with the recording video ID.

[0240] <Overall flow of generating a mixed video> Next, referring to FIG. 32, the overall flow of the information processing system 50 creating a mixed video will be described. FIG. 32 is a sequence diagram showing the overall process of the communication terminal 30 and the information processing system 50 communicating to create a mixed video. Note that the following communication terminal 30 is not limited to the communication terminal 30 used for browsing a wide-view image and may be any terminal device.

[0241] S201: The user enters the virtual room with the communication terminal 30 and displays the home screen 390 of FIG. 33 described later. First, the user presses the video selection button 391 to select a recorded video.

[0242] S202: The reception unit 32 receives the selection, and the communication unit 31 transmits a list request for the recorded video to the information processing system 50.

[0243] S203: The communication unit 51 of the information processing system 50 receives the list request for the recorded video, and the storage / reading unit 59 obtains a list of recorded videos from the recording video storage unit 5004. The storage / reading unit 59 obtains, for example, a list of recorded videos registered with the same tenant as the user.

[0244] S204: The screen generation unit 52 of the information processing system 50 generates the recorded video selection screen 400 shown in FIG. 34 from the list of recorded videos stored in the recording video storage unit 5004, and the attention point setting reception unit 62 transmits it to the communication terminal 30 via the communication unit 51. As a result, the communication unit 31 of the communication terminal 30 receives the list of recorded videos, and the display control unit 33 displays the list of recorded videos on the recorded video selection screen 400.

[0245] S205: The user sets a focus point on the recorded video selection screen 400 and selects the recorded videos for which a composite video is to be created. The user can specify multiple recorded videos.

[0246] S206: The reception unit 32 receives the selection, and the communication unit 31 requests the information processing system 50 for the recorded videos by specifying the selected recorded video IDs.

[0247] S207: The communication unit 51 of the information processing system 50 receives the request for the recorded videos, and the storage / reading unit 59 acquires the recorded videos specified by the recorded video IDs from the recorded video storage unit 5004.

[0248] S208: The focus point setting reception unit 62 of the information processing system 50 transmits the recorded videos to the communication terminal 30 via the communication unit 51. This recorded video is used for the user to set a fixed point or an object to be tracked.

[0249] S209: The user presses the focus point setting button 392 to set the focus point when creating the composite video.

[0250] S210: The reception unit 32 receives the selection, and thereafter, the focus point setting process shown in FIG. 35 is executed.

[0251] S211: Next, the user presses the video creation button 393 to create a composite video using the focus point.

[0252] S212: The reception unit 32 receives the selection, and thereafter, the video creation process shown in FIG. 41 is executed.

[0253] <<Home screen, recorded video selection screen>> FIG. 33 shows the home screen 390 displayed on the communication terminal 30 when creating a composite video. The home screen 390 is the screen first displayed when the user creates a composite video using multiple focus points. The home screen 390 has the following buttons. The video selection button 391 displays a list of recorded videos that will be used as the basis for creating the composite video, and accepts selections from the user. The "Focus Setting" button 392 is used to allow the user to specify which types of focus points to use when creating a mixed video. Users can specify multiple types of focus points, and multiple focus points for each type. The video creation button 393 is a button that triggers the creation of a hybrid video in which videos based on multiple points of interest are displayed in designated areas.

[0254] Figure 34 shows the recorded video selection screen 400. The recorded video selection screen 400 is displayed when the user presses the video selection button 391 on the home screen 390. The recorded video selection screen 400 displays a list 402 of recorded videos that the user can specify points of interest for. The recorded videos that the user can specify points of interest for are, for example, recorded videos associated with the same tenant as the user, but are not limited to these. For example, recorded videos uploaded to the internet may also be selectable.

[0255] The user selects the source video for creating the composite video from the video selection screen 400 and presses the OK button 401. After the user presses the OK button 401, the user returns to the home screen 390.

[0256] <Setting points of interest> Next, referring to Figure 35, the process by which the information processing system 50 receives the setting of a point of interest and creates point of interest information will be described. Figure 35 is a sequence diagram showing the process by which the communication terminal 30 and the information processing system 50 communicate, and the information processing system 50 receives the setting of a point of interest and creates point of interest information.

[0257] S301: The user presses the "Set Focus" button 392 on the home screen 390.

[0258] S302: The reception unit 32 receives the press, and the display control unit displays the point of interest setting screen 410 shown in Figure 36. First, assume that the user has pressed the fixed point setting button 412 on the point of interest setting screen 410. The order in which the user presses the buttons on the point of interest setting screen 410 does not matter.

[0259] S303: The reception unit 32 receives the press, and the display control unit 33 displays the fixed point setting screen 420 shown in Figure 37. The user sets one or more fixed points on the fixed point setting screen 420.

[0260] S304: When the user presses the selection button 422, the communication unit 31 of the communication terminal 30 transmits the fixed point (radial radius, polar angle, and azimuth angle) selected by the user to the information processing system 50.

[0261] S305: The communication unit 51 of the information processing system 50 receives a fixed point and stores it in the point of interest storage unit 5005.

[0262] S306: Next, assume that the user has pressed the viewing viewpoint setting button 411 on the focus point setting screen 410.

[0263] S307: The reception unit 32 receives the press, and the communication unit 31 of the communication terminal 30 requests the information processing system 50 for a list of viewing viewpoints associated with the recorded video selected by the user in step S205 of Figure 32.

[0264] S308: The communication unit 51 of the information processing system 50 receives a request for a list of viewing viewpoints, and the storage / reading unit 59 obtains a list of viewing viewpoints associated with the recorded video from the recorded video storage unit 5004. The storage / reading unit 59 reads out at least the names of the points of interest.

[0265] S309: The screen generation unit 52 of the information processing system 50 generates the viewing viewpoint setting screen 430 shown in Figure 38 using the list of viewing viewpoints, and the point of interest setting reception unit 62 transmits the screen information of the viewing viewpoint setting screen 430 to the communication terminal 30 via the communication unit 51.

[0266] S310: The communication unit 31 of the communication terminal 30 receives the screen information of the browsing viewpoint setting screen, and the display control unit 33 displays a list of browsing viewpoints on the browsing viewpoint setting screen 430. The user selects the browsing viewpoint for which the composite video is to be created with respect to the browsing viewpoint setting screen 430. The user can specify a plurality of browsing viewpoints.

[0267] S311: The reception unit 32 receives the press, and the communication unit 31 of the communication terminal 30 transmits the browsing viewpoint name to the information processing system 50.

[0268] S312: The communication unit 51 of the information processing system 50 receives the browsing viewpoint name, and the storage / reading unit 59 acquires the browsing viewpoint information specified by the browsing viewpoint name from the recorded video storage unit 5004.

[0269] S313: The attention point setting reception unit 62 stores this browsing viewpoint information in the attention point storage unit 5005. Therefore, the browsing viewpoint information is copied to the attention point storage unit 5005.

[0270] S314: Next, it is assumed that the user presses the follow viewpoint setting button 413 on the attention point setting screen 410.

[0271] S316: The reception unit 32 receives the press, and the display control unit 33 displays the follow setting screen 440 shown in FIG. 39. The user sets one or more objects to be displayed followingly on the follow setting screen 440. In response to the pressing of the selection button 446, the communication unit 31 of the communication terminal 30 designates a still image (object still image) including the position information of the object selected by the user and requests the information processing system 50 to create a follow viewpoint.

[0272] S317: The communication unit 51 of the information processing system 50 receives the request for creating a follow viewpoint, and the attention point setting reception unit 62 requests the follow viewpoint creation unit 63 to extract the object feature amount.

[0273] S318: The tracking viewpoint creation unit 63 extracts object features based on the object's position information in the object's still image. The object features may be SIFT features, edge shape, and color histogram, and are not limited to these.

[0274] S319: The tracking viewpoint creation unit 63 stores the object feature quantities in the point of interest storage unit 5005.

[0275] S320: Next, the point of interest setting reception unit 62 requests the tracking viewpoint creation unit 63 to create a tracking viewpoint using the object features.

[0276] S321: The tracking viewpoint creation unit 63 detects objects with object features for each frame of the recorded video and identifies the viewpoint (radial movement, polar angle, azimuth angle) of the objects in that frame. The tracking viewpoint information is a list of radial movement, polar angle, and azimuth angles arranged chronologically for each frame. Note that the identification of the viewpoint (radial movement, polar angle, azimuth angle) of the objects may be performed periodically, such as every few seconds, rather than frame by frame. Furthermore, the tracking viewpoint creation unit 63 may create the tracking viewpoint by tracking the objects using optical flow.

[0277] S322: The tracking viewpoint creation unit 63 stores the tracking viewpoint information in the point of interest storage unit 5005.

[0278] <<Screen for setting points of interest, screen for setting viewing viewpoint, screen for setting fixed points, screen for setting objects>> Figure 36 shows the focus setting screen 410 displayed by the communication terminal 30. The focus setting screen 410 is displayed when the user presses the focus setting button 392 on the home screen 390. The focus setting screen 410 displays the types of focus that the user can set. The focus setting screen 410 has the following buttons. The viewing viewpoint setting button 411 is used by the viewer to set the viewing viewpoint as a point of focus when viewing the recorded video. When the viewing viewpoint setting button 411 is pressed, the user is redirected to the viewing viewpoint setting screen 430. The fixed point setting button 412 is used to set a fixed point as the point of interest. When the fixed point setting button 412 is pressed, the screen transitions to the fixed point setting screen 420. The tracking viewpoint setting button 413 is a button that allows the user to set a viewpoint that tracks an arbitrary object as the point of focus. When the tracking viewpoint setting button 413 is pressed, the system transitions to the tracking settings screen 440.

[0279] Note that the types of points of interest are not limited to these, and other types of points of interest may be displayed on the points of interest setting screen 410.

[0280] Figure 37 shows the fixed point setting screen 420 displayed by the communication terminal 30. The fixed point setting screen 420 is a screen for setting one or more fixed points as points of interest. The user specifies a rectangular area 421 using a pointing device such as a mouse. The fixed points (radius, polar angle, azimuth angle) with respect to this rectangular area 421 as the viewpoint are stored in the point of interest storage unit 5005.

[0281] Figure 38 shows the viewing viewpoint setting screen 430 displayed by the communication terminal 30. The viewing viewpoint setting screen 430 is a screen for selecting the viewing viewpoint of the viewer who viewed the recorded video. The viewing viewpoint setting screen 430 displays, for example, a list 431 of viewing viewpoint names. Since the viewing viewpoint name includes the viewer's name, the user can easily determine whose viewpoint it is.

[0282] The user selects the viewing viewpoint name they wish to use as a reference using checkbox 432 and presses the select button 433. The user can select multiple viewing viewpoint names at once. The selected viewing viewpoint names are used as identification information, and the viewer information identified by these viewing viewpoint names is stored in the focus point memory unit 5005. Since the viewer views objects they deem important by changing their viewpoint in the wide-field image, the user can set objects they deem important as focus points.

[0283] Figure 39 shows the tracking settings screen 440 displayed by the communication terminal 30. The tracking settings screen 440 is a screen for the user to select any object to which they want to create a focus video by tracking it. Since the object to be tracked is often moving, the tracking settings screen 440 has a stop button 442. The stop button 442 switches between a stop button and a play button when pressed by the user. The tracking settings screen 440 also has a fast-forward button 443 (fast-forwards by a predetermined number of seconds) and a fast-back button 441 (fast-backs by a predetermined number of seconds). The user can operate these to stop the recorded video at a point where the desired object is visible.

[0284] When the desired object is found, the user specifies a rectangular area 445 using a pointing device such as a mouse. Since this rectangular area 445 is specified by location information, the tracking viewpoint creation unit 63 can extract the object's features and store them in the point of interest storage unit 5005. The tracking viewpoint creation unit 63 then detects objects with similar object features in each frame of the recorded video and stores the viewpoint of those objects as tracking viewpoint information in the point of interest storage unit 5005.

[0285] Alternatively, as shown in Figure 40, candidate objects may be displayed on the tracking settings screen 450. Figure 40 shows the tracking settings screen (part 2) 450. The tracking settings screen 450 (part 2) displays candidate objects 452 extracted from the recorded video in order of viewing duration (priority). Objects that the viewer deems important are more likely to be displayed on the tracking settings screen (part 2) 450 (displayed in order of viewing duration), so the user can set their focus to track important objects.

[0286] The user selects the object they want to track using the checkboxes and presses the OK button 453. This stores the object feature quantities of the selected object in the focus point storage unit 5005.

[0287] <Creating a hybrid video> Next, with reference to Figure 41, the process by which the information processing system 50 creates a hybrid video will be described. Figure 41 is a sequence diagram showing the process by which the communication terminal 30 and the information processing system 50 communicate and the information processing system 50 creates a hybrid video using the point of interest information.

[0288] S401: The user presses the video creation button 393 on the home screen 390.

[0289] S402: The reception unit 32 receives the press, and the communication unit 31 of the communication terminal 30 sends a video creation request for the recorded video selected in step S205 of Figure 32 to the information processing system 50.

[0290] S403: The communication unit 51 of the information processing system 50 receives a video creation request, and the storage / reading unit 59 obtains the point of interest information associated with the recorded video ID from the point of interest storage unit 5005.

[0291] S404: Next, the video creation unit 64 retrieves the recorded video associated with the recorded video ID from the recorded video storage unit 5004.

[0292] S405: The video creation unit 64 creates a composite video from the point of interest information and recorded video. The process of creating the composite video will be explained with reference to Figure 42.

[0293] S406: The video creation unit 64 saves the created hybrid video to the hybrid video information storage unit 5006.

[0294] Figure 42 is a flowchart illustrating the process by which the video creation unit 64 creates a hybrid video.

[0295] As described above, the video creation unit 64 obtained the point of interest information from the point of interest storage unit 5005 (S501).

[0296] The video creation unit 64 sequentially refers to the information on points of interest and determines whether or not there are any points of interest that have not yet been processed (S502).

[0297] If there are any unprocessed points of interest, the video creation unit 64 acquires fixed point, viewer viewpoint information, or tracking viewpoint information according to the type of point of interest (S503).

[0298] The video creation unit 64 creates a point of interest video based on fixed point, viewer viewpoint information, or tracking viewpoint information (S504). The point of interest video is a video of recorded footage where the fixed point, viewer viewpoint information, or tracking viewpoint information is within a predetermined area. Details of the creation of the point of interest video will be described later.

[0299] The video creation unit 64 creates a composite video by combining the videos of points of interest created based on each point of interest into a single screen (S505).

[0300] Next, we will explain how to create a video highlighting key points, referring to Figures 43 to 45. There are mainly two methods for creating a video highlighting key points.

[0301] (i) The video creation unit 64 extracts a planar image of a predetermined area from the recorded video based on a fixed point, viewing viewpoint, or tracking viewpoint. The video creation unit 64 repeats this process for each frame of the recorded video to create a point of interest video in which the planar image is used as a frame. In this method, the point of interest video only contains the field of view of the predetermined area, but the processing load during playback is low.

[0302] (ii) The video creation unit 64 incorporates the recorded video into the video viewer along with fixed point, viewing viewpoint information, or tracking viewpoint information, while maintaining the wide field of view of the recorded video. With this method, when playing back the mixed video, the user can change the viewpoint to an angle of view other than the predetermined area, but a video viewer is required that plays back the recorded video while automatically changing the viewpoint in the point of interest video.

[0303] Figure 43(a) is a diagram illustrating method (i). Figure 43(a) shows the method for creating a point of interest video in the case of a viewing viewpoint or a following viewpoint. The video creation unit 64 sets predetermined regions T at each time point of the recorded video 160 as frames 161 to 163 of the point of interest video. That is, the video creation unit 64 refers to the viewing viewpoint or the following viewpoint at each time point t1, t2, t3, ... which is the timing for creating frames 161 to 163. The video creation unit 64 determines the predetermined region T indicated by the viewing viewpoint or the following viewpoint at each time point and creates frames 161 to 163 by performing perspective projection transformation (cutting out) in the same way as during normal viewing. The video creation unit 64 constructs the point of interest video using these frames 161 to 163 as the frames of the point of interest video.

[0304] Typically, the number of viewpoints from the viewing viewpoint or the tracking viewpoint is less than the number of frames 161 to 163, so the video creation unit 64 may create frames 161 to 163 based on the timestamps of the viewing viewpoint or the tracking viewpoint. The video creation unit 64 may also create a point-of-interest video by interpolating the created frame 161 and the next created frame 162 (and similarly between frames 162 and 163) using image processing.

[0305] In the case of a fixed point, the only difference is that the viewpoint does not change at each time point; therefore, the method for creating the point-of-interest video can be the same as for viewer viewpoint information or follow-view viewpoint information.

[0306] Figure 43(b) illustrates method (ii). The video viewer is a program that automatically changes the viewpoint at a given viewpoint without the user having to manually manipulate the viewpoint, and displays a predetermined area T of the recorded video, which is a wide-field image. Therefore, as shown in Figure 43(b), if the information processing system 50 registers the recorded video, viewing viewpoints (one for each viewer), tracking viewpoints, and fixed points in the video viewer, the video creation unit 64 can create a video of points of interest. The screen generation unit 52 can also perform similar processing.

[0307] In the case of a web application, the video viewer is provided to the communication terminal 30 as JavaScript (registered trademark) code included in the web application, while in the case of a native application, the video viewer is included in the native application. The video viewer of a native application only needs to download viewing viewpoints (one for each viewer), tracking viewpoints, and fixed points from the information processing system 50.

[0308] Next, referring to Figure 44, a method for creating a hybrid video using point-of-interest videos will be explained. As shown in Figure 44(a), point-of-interest video 171 based on viewer A's viewing viewpoint, point-of-interest video 172 based on viewer B's viewing viewpoint, point-of-interest video 173 based on a tracking viewpoint, and point-of-interest video 174 based on a fixed point have been created. These point-of-interest videos may be created using either (i) or (ii). If created using method (ii), point-of-interest videos 171-174 will have an extent beyond the predetermined area. The user can arbitrarily change the viewpoint of each point-of-interest video.

[0309] The video creation unit 64 creates the hybrid video 175 by placing each frame of these point-of-interest videos 171 to 174 into each frame of the hybrid video 175 (Figure 44(b)).

[0310] In Figure 44(b), one video operation button 177 (play, stop, fast forward, rewind, etc.) is assigned to each of the highlight videos 171-174. When a hybrid video 175 is created in this way, the hybrid video becomes a single video, so when the user operates play, stop, fast forward, rewind, etc., the same operation is performed on all of the highlight videos 171-174. The user can play each of the highlight videos contained in the hybrid video 175 in parallel.

[0311] On the other hand, users may want to play individual highlight videos when playing a composite video. Regarding audio, if a composite video is created from only one recorded video, it may be constructed as a single video. However, if a composite video is created from multiple recorded videos, the audio may be mixed. Therefore, it is also beneficial for users to be able to play multiple highlight videos individually in a composite video.

[0312] Figure 44(c) shows an example of a hybrid video 176 in which multiple highlight videos 171-174 can be played individually by the user. In Figure 44(c), each of the highlight videos 171-174 has video operation buttons 177 for play, stop, fast forward, and rewind. The audio can also be controlled individually by the user, allowing them to play only the audio of a specific highlight video or adjust the volume of the audio of multiple highlight videos individually. Even with highlight videos 171-174 as shown in Figure 44(c), the user can still play each highlight video 171-174 in parallel.

[0313] As shown in Figures 44(b) and 44(c), the creation of the composite videos 175 and 176 is generally performed during playback of the composite video, and the creation of the composite videos 175 and 176 is a process closely related to the playback of the composite video. On the other hand, if the composite video 175 in Figure 44(b) has the point-of-interest videos 171 to 174 created by method (i), the video creation unit 64 can create the composite videos 175 and 176 in advance.

[0314] Furthermore, when the hybrid videos 175 and 176 in Figures 44(b) and 44(c) are played on the communication terminal 30, they become the hybrid video playback screen 460. The number of featured videos placed on the hybrid video playback screen 460 is not limited to four; there can be five or more, and there are no restrictions on the number of featured videos as long as a web application or native application can be displayed.

[0315] Furthermore, the hybrid videos 175 and 176 in Figures 44(b) and 44(c) may be created using either method (i) or (ii). Also, both web applications and native applications can play either of the hybrid videos 175 and 176 in Figures 44(b) and 44(c).

[0316] Next, we will explain the zoom in / out display of the highlight videos with reference to Figure 45. In the mixed videos 175 and 176 in Figures 44(b) and 44(c), the user can zoom in on any of the highlight videos 171 to 174 or minimize any highlight videos that they do not wish to view.

[0317] Figure 45(a) shows a hybrid video created when the user has minimized the remaining two focus videos, 171 and 172. Figure 45(b) shows a hybrid video created when the user has minimized the remaining three focus videos, 171, and 174. In this way, the user can enlarge and view only the focus videos that contain the viewpoint they want to see. In addition, the user may individually change the size or arrangement of focus videos 171-174.

[0318] Furthermore, in either case, if the highlight video is created using method (ii), the user can change the viewpoint of the highlight video at will.

[0319] <Playback of mixed video> Next, with reference to Figure 46, the process by which a user plays a hybrid video will be explained. Figure 46 is a sequence diagram showing the process by which a communication terminal 30 and an information processing system 50 communicate and the information processing system 50 provides a hybrid video to the communication terminal 30.

[0320] S601: When the communication terminal 30 enters a virtual room or a user logs in, the communication unit 31 of the communication terminal 30 can receive a list of mixed videos created by the user. The list of mixed videos received is not limited to the user's own; it may also include mixed videos from other users belonging to the same tenant as the user.

[0321] S602: The user selects the mixed video they want to view.

[0322] S603: The reception unit 32 receives the press, and the communication unit 31 of the communication terminal 30 requests the mixed video from the information processing system 50, specifying the mixed video ID.

[0323] S604: The communication unit 51 of the information processing system 50 receives a request for a mixed video, and the storage / reading unit 59 retrieves the mixed video associated with the mixed video ID from the mixed video information storage unit 5006.

[0324] S605: Next, the screen generation unit 52 generates a hybrid video playback screen 460 using the hybrid video. When a web application displays a hybrid video created by method (i), the screen generation unit 52 generates screen information that includes a video playback function and the hybrid video. When the web application displays a hybrid video created by method (ii), the screen generation unit 52 generates screen information for the video viewer that incorporates viewing viewpoints (one for each viewer), tracking viewpoints, and fixed points. • When a native app displays a hybrid video created using method (i), the screen generation unit 52 only needs to prepare the hybrid video. When a native app displays a hybrid video created using method (ii), the screen generation unit 52 only needs to provide viewing viewpoints (one for each viewer), a follow-up viewpoint, and a fixed point.

[0325] S606: The communication unit 51 of the information processing system 50 transmits the screen information of the mixed video playback screen 460 to the communication terminal 30.

[0326] S607: The communication unit 31 of the communication terminal 30 receives screen information of the hybrid video playback screen 460, and the display control unit 33 displays the hybrid video playback screen 460. The hybrid video playback screen 460 may be the same as in Figures 44(b)(c) and 45.

[0327] <Examples of communication system applications in telemedicine> Figure 47 illustrates an example of remote communication where communication system 1 is applied to telemedicine. Note that the explanation of Figure 47 primarily focuses on the differences from Figure 1. In Figure 47, base A is an operating room, but the processing flow from (1) to (6) is the same as in Figure 1. In Figure 47, the patient is placed on an operating table 355 and undergoes surgery performed by a medical professional such as a doctor. The medical professional (corresponding to the user) uses various surgical instruments 354, such as forceps and scalpels, to operate on the patient. The medical professional can also wear smart glasses 88 and transmit images of the surgical field to the communication network N. Furthermore, various cameras, such as an operating room camera 351, a surgical field camera 352, and an endoscope 353, are arranged in the operating room as imaging devices similar to the imaging device 10. These imaging devices may also have imaging functions to capture images for generating wide-field images. All imaging devices and smart glasses 88 in the operating room can be associated with a virtual room.

[0328] A main unit 356 is located in the operating room to monitor the patient's vital signs and the operating status of medical equipment. The main unit 356 corresponds to the communication terminal 30 in this embodiment. In addition to the functions shown in Figure 1, the communication terminal 30 (main unit 356) in the operating room may also have the function of receiving images from the endoscope 353 and the surgical field camera 352. The communication terminal 30 can display the received images, including wide-field images, on the display 306 and can transmit them to the information processing system 50 as images from the communication terminal 30's location. The operation panel 357 is an input interface that accepts various operations, and may allow medical personnel to operate equipment in the operating room via the operation panel 357. Furthermore, the endoscope 353, surgical field camera 351, and surgical field camera 352 may communicate directly with the information processing system 50 without going through the communication terminal 30. In this way, multiple imaging devices 10 can be associated with the same virtual room, so users at remote locations can request the acquisition of wide-field images capturing various scenes from the site at location A. For example, if you want to image the inside of a patient's body, you can send an imaging request to the imaging device corresponding to the endoscope 353, and if you want to image the entire operating room, you can send an imaging request to the imaging device corresponding to the operating room camera 351.

[0329] Furthermore, the communication terminal 30 may have the functionality of an electronic medical record system, or it may have the functionality to communicate with an electronic medical record system. The communication terminal 30 may display information from the electronic medical record along with the wide-field image on the display 306. Also, the storage 90 may be an electronic medical record system. In such a case, the wide-field image (and associated viewpoint information) acquired in response to the imaging request may be associated with the patient's electronic medical record and saved by the association processing unit 53. Also, the folders indicated by the storage location of the storage 90 may be classified by patient or surgery. Also, the virtual room information storage unit 5002 may store associated information indicating the patient and the details of the surgery. In this way, information related to the patient and surgery can always be displayed on the viewing screen of the communication terminal.

[0330] As an example, Figure 48 shows a hybrid video playback screen 460 when the imaging device 10 is used in a medical setting. In this hybrid video playback screen 460, videos 181-184 focusing on points of interest obtained from a medical perspective from the recorded video captured by the imaging device 10 in the medical setting are displayed. Therefore, in Figure 48, videos 181-184 focusing on points of interest are shown, with the doctor, main unit, surgical field, and medical instruments as fixed points, viewing points, or tracking points, representing the perspective of the medical setting.

[0331] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0332] For example, the configuration example shown in Figure 13 is divided according to its main function in order to facilitate understanding of the processing performed by the information processing system 50, the imaging device 10, and the communication terminal 30. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the information processing system 50, the imaging device 10, and the communication terminal 30 can be further divided into more processing units depending on the processing content. Furthermore, each processing unit can be divided to include even more processing.

[0333] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0334] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0335] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figures 25, 26, 32, and 33, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0336] <Additional notes regarding claims> [Claim 1] A communication system comprising an imaging device capable of capturing wide-field images with a wide field of view, and an information processing system that transmits video including the wide-field images captured by the imaging device to a communication terminal, The aforementioned information processing system is A recording unit that records the video footage of the wide-field image displayed on the communication terminal, A video creation unit creates a hybrid video in which multiple point-of-interest videos are arranged, each playing back a predetermined area of ​​the recorded video recorded in the recording unit at different points of interest. A communication unit transmits screen information of the hybrid video, in which multiple videos of points of interest are played in parallel, to a communication terminal. A communication system having [Claim 2] The recording unit records the recorded video in correspondence with the viewing viewpoint when the wide-field image is displayed on the communication terminal. The communication system according to claim 1, wherein the types of points of interest include the viewing viewpoint. [Claim 3] The aforementioned information processing system The system includes a focus point setting reception unit that receives settings for the type of focus point for creating the focus point video from the aforementioned communication terminal, The communication system according to claim 2, wherein the number of types of points of interest received by the point of interest setting reception unit is multiple, and the video creation unit creates multiple point of interest videos using different types of points of interest. [Claim 4] The communication system according to claim 3, wherein the types of points of interest are at least two of the following: a fixed point that fixes the viewpoint for the recorded video, the viewing viewpoint recorded by the recording unit, and a tracking viewpoint that follows an object included in the recorded video. [Claim 5] The communication unit transmits screen information that accepts any viewpoint of the recorded video to the communication terminal. The communication system according to claim 4, wherein the point of interest setting reception unit receives the arbitrary viewpoint received by the communication terminal as the fixed point for the recorded video. [Claim 6] The communication unit plays back the recorded video as a video and transmits screen information that accepts an arbitrary object to the communication terminal. The aforementioned information processing system is The system includes a tracking viewpoint creation unit that creates a tracking viewpoint in the recorded video that follows the object received by the communication terminal in the recorded video. The communication system according to claim 4, wherein the point of interest setting reception unit receives the tracking viewpoint created by the tracking viewpoint creation unit as the point of interest. [Claim 7] The communication unit transmits the viewer information of the recorded video to the communication terminal. The aforementioned information processing system is The communication system according to claim 4, wherein the communication terminal receives the viewing viewpoint associated with the viewing information it has received as the point of interest. [Claim 8] The communication system according to any one of claims 4 to 7, wherein the video creation unit creates the point of interest video at each of the fixed point, the viewing viewpoint, and the tracking viewpoint, and creates the mixed video in which multiple point of interest videos are arranged. [Claim 9] The communication system according to claim 8, wherein the video creation unit creates a video of planar images in which predetermined areas identified by the fixed point, the viewing viewpoint, and the tracking viewpoint are cut out from the recorded video, and creates a single hybrid video in which multiple videos of points of interest are played in parallel. [Claim 10] The communication system according to claim 8, wherein the video creation unit creates a hybrid video in which the videos of a predetermined area identified at a fixed point in the recorded video, a predetermined area identified at the viewing viewpoint of the recorded video, and a predetermined area identified at the tracking viewpoint of the recorded video are each arranged on a single screen as the video of the point of interest. [Claim 11] The communication system according to claim 9 or 10, wherein the hybrid video allows for individual video manipulation of any of the aforementioned point-of-interest videos. [Claim 12] The aforementioned video of points of interest is a communication system according to claim 9, in which the viewpoint can be changed at will. [Explanation of symbols]

[0337] 10 Imaging device 30 Communication terminals 50 Information Processing Systems [Prior art documents] [Patent Documents]

[0338] [Patent Document 1] Japanese Patent Publication No. 2021-022788

Claims

1. A communication system comprising an imaging device capable of capturing wide-field images with a wide field of view, and an information processing system that transmits video including the wide-field images captured by the imaging device to a communication terminal, The aforementioned information processing system is A recording unit that records the video footage of the wide-field image displayed on the communication terminal, A video creation unit creates a hybrid video in which multiple point-of-interest videos are arranged, each playing back a predetermined area of ​​the recorded video recorded in the recording unit at different points of interest. A communication unit transmits screen information of the hybrid video, in which multiple videos of points of interest are played in parallel, to the communication terminal. A communication system having

2. The recording unit records the recorded video in correspondence with the viewing viewpoint when the wide-field image is displayed on the communication terminal. The communication system according to claim 1, wherein the types of points of interest include the viewing viewpoint.

3. The aforementioned information processing system The system includes a focus point setting reception unit that receives settings for the type of focus point for creating the focus point video from the aforementioned communication terminal, The communication system according to claim 2, wherein the number of types of points of interest received by the point of interest setting reception unit is multiple, and the video creation unit creates multiple point of interest videos using different types of points of interest.

4. The communication system according to claim 3, wherein the types of points of interest are at least two of the following: a fixed point that fixes the viewpoint for the recorded video, the viewing viewpoint recorded by the recording unit, and a tracking viewpoint that follows an object included in the recorded video.

5. The communication unit transmits screen information that accepts any viewpoint of the recorded video to the communication terminal. The communication system according to claim 4, wherein the point of interest setting reception unit receives the arbitrary viewpoint received by the communication terminal as the fixed point for the recorded video.

6. The communication unit plays back the recorded video as a video and transmits screen information that accepts an arbitrary object to the communication terminal. The aforementioned information processing system is The system includes a tracking viewpoint creation unit that creates a tracking viewpoint in the recorded video that follows the object received by the communication terminal in the recorded video. The communication system according to claim 4, wherein the point of interest setting reception unit receives the tracking viewpoint created by the tracking viewpoint creation unit as the point of interest.

7. The communication unit transmits the viewer information of the recorded video to the communication terminal. The aforementioned information processing system is The communication system according to claim 4, wherein the communication terminal receives the viewing viewpoint associated with the viewing information it has received as the point of interest.

8. The communication system according to any one of claims 4 to 7, wherein the video creation unit creates the point of interest video at each of the fixed point, the viewing viewpoint, and the tracking viewpoint, and creates the mixed video in which multiple point of interest videos are arranged.

9. The communication system according to claim 8, wherein the video creation unit creates a video of planar images in which predetermined areas identified by the fixed point, the viewing viewpoint, and the tracking viewpoint are cut out from the recorded video, and creates a single hybrid video in which multiple videos of points of interest are played in parallel.

10. The communication system according to claim 8, wherein the video creation unit creates a hybrid video in which the videos of a predetermined area identified at a fixed point in the recorded video, a predetermined area identified at the viewing viewpoint of the recorded video, and a predetermined area identified at the tracking viewpoint of the recorded video are each arranged on a single screen as the video of the point of interest.

11. The communication system according to claim 9, wherein the hybrid video allows for individual video manipulation of any of the aforementioned point-of-interest videos.

12. The aforementioned video of points of interest is a communication system according to claim 9, in which the viewpoint can be changed at will.

13. An information processing system that transmits video, including a wide-field image captured by an imaging device capable of capturing a wide-field image with a wide field of view, to a communication terminal, A recording unit that records the video footage of the wide-field image displayed on the communication terminal, A video creation unit creates a hybrid video in which multiple point-of-interest videos are arranged, each playing back a predetermined area of ​​the recorded video recorded in the recording unit at different points of interest. A communication unit transmits screen information of the hybrid video, in which multiple videos of points of interest are played in parallel, to the communication terminal. An information processing system having

14. A video creation method performed by a communication system comprising an imaging device capable of capturing wide-field images with a wide field of view, and an information processing system that transmits video including the wide-field images captured by the imaging device to a communication terminal, The aforementioned information processing system is The steps include recording video of the wide-field image displayed on the communication terminal, The steps include creating a hybrid video in which multiple point-of-interest videos are arranged, each playing back a predetermined area of ​​the recorded video recorded at different points of interest, and A communication unit and a step that transmit screen information of the hybrid video in which multiple videos of points of interest are played in parallel to the communication terminal, A method for creating videos that include [specific features / features].

15. An information processing system that transmits video, including wide-field images captured by an imaging device capable of capturing wide-field images with a wide field of view, to a communication terminal. A recording unit that records the video footage of the wide-field image displayed on the communication terminal, A video creation unit creates a hybrid video in which multiple point-of-interest videos are arranged, each playing back a predetermined area of ​​the recorded video recorded in the recording unit at different points of interest. A communication unit transmits screen information of the hybrid video, in which multiple videos of points of interest are played in parallel, to the communication terminal. A program designed to function as such.

Images