Imaging apparatus, transmission method, and program

Abstract

To fix image data transmitted by an imaging apparatus.SOLUTION: The imaging apparatus comprises: an imaging processing unit which repetitively generates image frames constituting a moving image captured by the imaging apparatus; a communication unit which transmits to a communication terminal at another base the image frames generated repetitively by the imaging processing unit; and a reception unit which receives a fixation instruction of the image frames to be transmitted to the communication terminal. In a case where the reception unit receives the fixation instruction, the communication unit transmits the image frames to the communication terminal repetitively.SELECTED DRAWING: Figure 24

Description

[Technical field]

[0001] The present invention relates to an imaging device, a transmission method, and a program. [Background technology]

[0002] There is known a communication system that transmits at least images and audio from one location to one or more other locations in real time, and enables remote communication between users in remote locations using images and audio. As such images, there is known a wide-field-of-view image having a wide viewing angle captured in a wide imaging range including, for example, a 360-degree image (also called a celestial sphere image, omnidirectional image, or omnidirectional image) in which the entire 360-degree surroundings are captured as an imaging range that includes areas that cannot be confirmed with a normal angle of view. A user can view a part of the wide-field-of-view image from any virtual viewpoint by changing the virtual viewpoint of a part of the wide-field-of-view image displayed on the display screen of the communication terminal by operating the communication terminal.

[0003] There is known a technique for correcting an image when an imaging device is shaken (for example, see Patent Document 1). Patent Document 1 discloses a technique for performing optimal shake correction in a surveillance camera according to the degree of shaking of the surveillance camera body and its characteristics. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, there is a problem that the image data transmitted by the imaging device is not fixed. For example, when a user moves with an imaging device and delivers a wide-field image while stationary, it may take some time for an image frame with appropriate image quality to be transmitted immediately after the user stops due to the characteristics of image encoding and the available bandwidth of the network.

[0005] In view of the above-mentioned problems, the present invention provides a technique for fixing image data transmitted by an imaging device. [Means for solving the problem]

[0006] In consideration of the above problems, the present invention comprises an imaging processing unit that performs imaging processing to repeatedly generate image frames that constitute a moving image captured by an imaging device, a communication unit that transmits the image frames repeatedly generated by the imaging processing unit to a communication terminal at another base, and a reception unit that receives an instruction to fix the image frames transmitted to the communication terminal, and when the reception unit receives the fixation instruction, the communication unit repeatedly transmits the same image frame to the communication terminal. Effect of the Invention

[0007] According to the present invention, a technique for fixing image data transmitted by an imaging device can be provided. [Brief description of the drawings]

[0008] [Figure 1] 1A and 1B are diagrams illustrating an example of remote communication using a wide-field-of-view image. [Diagram 2] FIG. 1 is a schematic diagram illustrating an example of a configuration of a communication system. [Diagram 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal and an information processing system. [Diagram 5] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. [Figure 6] FIG. 1 is a conceptual diagram of an imaging device in use. [Figure 7] (a) is a hemispherical image (front) captured by an imaging device, (b) is a hemispherical image (back) captured by an imaging device, and (c) is an image represented by equirectangular projection. [Figure 8] (a) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (b) is a diagram showing a spherical image. [Figure 9] FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional solid sphere. [Figure 10]FIG. 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on a display. [Figure 11] 13 is a diagram showing the relationship between predetermined region information and an image of a predetermined region T. FIG. [Figure 12] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 13] FIG. 1 is a diagram illustrating an example of a functional configuration of a communication system. [Figure 14] 4 is a conceptual diagram showing image management information stored in an image management information storage unit. FIG. [Figure 15] 1A is a diagram showing an example of an entry screen, and FIG. 1B is a diagram showing an example of an image viewing screen displayed by a communication terminal when a user enters a virtual room. [Figure 16] 11 is an example of a sequence diagram illustrating a process in which a user (or a communication terminal) enters a virtual room. [Figure 17] FIG. 13 is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room. [Figure 18] FIG. 13 is a diagram showing an example of a virtual room association screen (part 2). [Figure 19] FIG. 13 is a diagram showing an example of a virtual room association screen (part 3). [Figure 20] 13 is a diagram showing an example of a wide-field image transmission start / stop dialogue displayed by the communication terminal. FIG. [Figure 21] 11 is an example of a sequence diagram showing a procedure in which a user registers an imaging device in a virtual room. [Figure 22] FIG. 13 is an example of a sequence diagram illustrating a flow in which a wide-field image is shared. [Diagram 23] FIG. 1 is a diagram illustrating a situation in which a user captures images of the surroundings with an imaging device and distributes the images through streaming. [Figure 24] 11 is an example of a sequence diagram illustrating a process of fixing and releasing an image frame in response to a button operation on an operation unit, and transmitting image data from an imaging device. FIG. [Diagram 25]11 is a flowchart illustrating an example of how the imaging device fixes and releases the fixation of an image frame in response to a button operation on an operation unit. [Figure 26] 13 is a diagram showing an example of an image frame selection screen displayed by the imaging device or a communication terminal connected to the imaging device. FIG. [Figure 27] 11 is a flowchart illustrating an example of locking and unlocking of the imaging device by button operations on an operation unit or a timer. [Figure 28] 13A and 13B are diagrams illustrating an example of an image frame displayed by a communication terminal while a fixed flag is ON. [Figure 29] 11 is an example of a sequence diagram illustrating a process in which an imaging device transmits image data in response to locking and unlocking of an image frame by a remote command. [Diagram 30] 11 is a flowchart illustrating an example of an image capture device locking and unlocking an image frame by remote control. [Diagram 31] 11 is an example of a sequence diagram illustrating a process in which the imaging device transmits image data in response to fixing of an image frame due to detection of an object. [Diagram 32] 11 is a flowchart illustrating an example of a process in which the imaging device fixes an image frame upon detection of an object. [Diagram 33] 11 is an example of a sequence diagram illustrating a process in which the imaging device transmits image data in response to fixing of an image frame upon detection of a stop of the imaging device; [Diagram 34] 11 is a flowchart illustrating an example of a process for fixing an image frame by the imaging device upon detection of a stop of the imaging device. [Diagram 35] FIG. 13 is an example of a flowchart in which the imaging device fixes the image frame upon detection of the stop of the imaging device, and the fixation flag is turned off in response to detection of a timer or the start of movement. [Diagram 36] FIG. 1 is a diagram illustrating an example of remote communication in which the communication system is applied to remote medical care. [Figure 37]FIG. 13 is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room in the case of remote medical care. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An information processing system and a transmission method performed in the information processing system will be described below as an example of an embodiment of the present invention.

[0010] <An example of remote communication> Fig. 1 is a diagram for explaining an example of remote communication using a wide-field-of-view image. In Fig. 1, three locations A to C are communicating via an information processing system 50. The number of locations is merely an example, and there may be two locations, or four or more locations. Remote communication refers to communicating with a person who is physically distant through images and sounds by utilizing IT tools.

[0011] As an example, the site A is a construction site. Sites B and C can be anywhere, for example, an office, or other place where wide-field images can be communicated. At the site A, an imaging device 10 is disposed that can capture a wide-angle image, such as a celestial sphere image, or a wide-angle image of 180 to 360 degrees in the horizontal direction. In the following, such a wide-angle image is simply referred to as a wide-field image. At the sites A to C, various communication terminals 30A to 30C that can view the wide-field image are disposed. In the following, any one of the communication terminals 30A to 30C is referred to as a "communication terminal 30."

[0012] At the construction site, various construction works are being carried out by workers at each location, and while capturing the entire construction site with a wide-field-of-view image, users at each of the bases A to C can arbitrarily change the viewpoint to monitor any construction or work that they want to focus on. The viewpoint is the center position or range of the entire wide-field-of-view image that is displayed on the display of the communication terminals 30A to 30C.

[0013] The imaging device 10 is attached to a tripod 86 or an arm 85 via a gimbal 87. A relay device (in FIG. 1, the communication terminal 30A also serves as the relay device) is installed at the construction site, and the communication terminal 30A transmits the wide-field-of-view image received from the imaging device 10 by wire or wirelessly to the information processing system 50. The communication terminal 30A can also be a terminal for viewing the wide-field-of-view image. A camera 9 is connected to the communication terminal 30A (or may be built-in), and an image with a normal angle of view captured by the camera 9 (which may be capable of capturing a celestial sphere image) can also be transmitted to the information processing system 50. In addition, a user A (which may be a worker, for example) can wear smart glasses 88, and an image with a normal angle of view captured by the smart glasses 88 (which may be capable of capturing a celestial sphere image) may be transmitted to the information processing system 50. The smart glasses 88 are an information terminal that displays information acquired via the Internet on a display while ensuring a field of view. The smart glasses 88 may be placed at any base.

[0014] At the site B, a PC (Personal Computer), a smartphone, or the like is disposed as an example of the communication terminal 30B. The communication terminal 30B may be any device capable of communicating with the information processing system 50 via the communication network N, and may also be a display device such as a tablet terminal, a PDA (Personal Digital Assistant), an electronic whiteboard, or a projector. A camera may be built into or connected to the communication terminal 30B.

[0015] At the site C, a PC, a smartphone, a VR goggles 89, etc. are placed as an example of the communication terminal 30C, and in FIG. 1, a camera 8 is built in or connected to the communication terminal 30C. The VR goggles 89 are information terminals that display an artificial world or a celestial sphere image on a computer according to the direction of the movement of the neck or body. The camera 8 may be a wide-angle or normal-angle camera. In addition, the communication terminal 30C may be any device that can communicate with the information processing system 50 via the communication network N, and may be a display device such as a tablet terminal, a PDA, an electronic whiteboard, or a projector. The VR goggles 89 may be placed at any site.

[0016] In this embodiment, the imaging device 10 and the communication terminal 30 are managed in a communication group called a virtual room. The imaging device 10 is associated with the virtual room, and the communication terminal 30 (the user who operates the communication terminal 30) can receive the wide-field image transmitted by the imaging device 10 by entering the virtual room, and the user can view the wide-field image. The smart glasses 88 and the VR goggles 89 can also be associated with the virtual room. The cameras 8 and 9 enter the virtual room together with the communication terminal 30.

[0017] Users A to C at locations A to C can arbitrarily change the viewpoint of the wide-field-of-view image. For this reason, it is possible that each of users A to C viewing the wide-field-of-view image in real time is looking at a different viewpoint, which may make communication difficult. Therefore, in this embodiment, the viewpoint of the communication terminal 30 at location A is shared with other communication terminals 30 at locations B and C. An overview of viewpoint sharing will be described below. For the sake of explanation, it is assumed below that the viewpoint of the user at location B is shared by the users at locations A and C.

[0018] (1) The communication terminals 30A to 30C share a wide-field-of-view image captured by the imaging device 10. When a user B requests imaging of a wide-field-of-view image while viewing the image from an arbitrary viewpoint on the communication terminal 30B, the communication terminal 30B transmits viewpoint information and an imaging request to the information processing system 50.

[0019] (2) In response to an image capture request, the information processing system 50 requests the image capture device 10 to capture an image (either a still image or a moving image) by specifying viewpoint information.

[0020] (3) The imaging device 10 captures an image in response to an imaging request, and stores the wide-field image and viewpoint information at a URL (in FIG. 1, the imaging device 10 is the storage 90) notified by the information processing system 50. The wide-field image stored in the storage 90 can be downloaded and displayed by any communication terminal 30.

[0021] (4) The information processing system 50 transmits the URL to the communication terminal 30B.

[0022] (5) Furthermore, the information processing system 50, either automatically or in response to a request from user B, transmits the URL to the communication terminals 30A and 30C that are currently in the same virtual room.

[0023] (6) The communication terminals 30A and 30C connect to the URL to receive the viewpoint information and the wide-field image, and display the wide-field image identified by the viewpoint information by setting the viewpoint to match the center of the image field. Note that the viewpoint does not need to be perfectly matched to the center, and the viewpoint may be set to be included in a range near the center of the image field.

[0024] The same applies when the viewpoint of a user at site A is shared by users at sites B and C, and when the viewpoint of a user at site C is shared by users at sites A and B.

[0025] As described above, in the communication system 1 of this embodiment, viewpoint information is shared among the locations even when a wide-field-of-view image is distributed, so that communication between users becomes easier.

[0026] It is also possible that the imaging device 10 transmits the wide-field image itself to the information processing system 50 in (3), and the information processing system 50 transmits the wide-field image to the communication terminals 30A to 30C in (4).

[0027] In FIG. 1, an example in which the imaging device 10 is placed at a construction site has been described, but the present embodiment can also be applied to VR education, event distribution, remote customer service, remote medical care, and the like. In VR education, the imaging device 10 is placed at a base at the site, such as a laboratory, and students can freely change the viewpoint from a remote base to view a blackboard, equipment, samples, experiment results, and the like. In event distribution, the imaging device 10 is placed at the location where the event is held at the site, and event participants such as spectators can freely change the viewpoint online from a remote base to view the state of the event site. The state of the event site includes images of subjects such as performers, contestants, presenters, products and exhibits explained at the event, images of materials explained at the event, images of the state of the event site, and the like. The event site may be indoors or outdoors, and includes venues for sports, concerts, plays, and the like. In remote customer service, for example, when applied to customer service at a travel agency, the imaging device 10 is placed at the travel destination at the site, and customers can freely change the viewpoint from a remote base to consider their itinerary. In remote medical care, the imaging device 10 is placed in a medical facility such as an operating room, and doctors, students, medical equipment personnel, etc. can change the viewpoint at will from a remote location and view the actions of doctors and nurses performing medical procedures at the medical facility, the placement of instruments, the patient's condition, vital signs, etc.

[0028] The locations where images are captured are not limited to these, and can be any space where there is a need for a user (viewer) at the viewing location to understand the situation at a remote location, such as a school, factory, warehouse, construction site, server room, or store.

[0029] <Terminology> A tenant is a company, organization, or the like that has entered into a contract to receive an image distribution service from a service provider (an information processing system in this embodiment). As an example, a user belongs to a tenant, but a user may subscribe to the service individually. In addition to users, imaging devices, virtual rooms, and the like are registered in a tenant.

[0030] A base is a place where activities are based. In this embodiment, a conference room will be described as an example of a base. A conference room is a room that is set up mainly for use in meetings. A conference is also called a meeting, a conference, a gathering, a get-together, a gathering, or the like.

[0031] The device refers to a device other than a general-purpose communication terminal 30 such as a PC or a smartphone, and is an imaging device or a viewing device for a wide-field image. In this embodiment, examples of the device include the imaging device 10, smart glasses 88, and VR goggles 89.

[0032] The viewpoint information is parameter information that specifies which specific area of ​​the wide-field-of-view image to display on the display. In this embodiment, the radius vector, polar angle, and azimuth angle corresponding to the center of the wide-field-of-view image displayed on the display are described as examples of viewpoint information, but the viewpoint information may be specified by other parameter information such as the coordinates of the diagonal vertices.

[0033] A wide-field-of-view image is an image having a wide field of view captured in a wide imaging range, including a 360-degree image (also called a celestial sphere image, omnidirectional image, or panoramic image) in which the entire surroundings are captured at 360 degrees.

[0034] Remote communication refers to communicating with people who are physically far away through images and audio by utilizing IT tools.

[0035] VR education is an educational tool that uses virtual reality (VR) to attract children's attention and deepen their understanding by creating an unrealistic space in front of the eyes and immersing the viewer in a realistic way. Remote customer service is a customer service where staff in a remote location introduce products to users via video conferencing on a tablet or display.

[0036] An image is something that visually represents an event, and image data is data that allows a computer to handle images. A video is an image projected on a display device, but is generally considered a sub-concept of video and other images.

[0037] An image with a normal angle of view is not a wide-field image, but in this embodiment, it will be described as an image that is not a wide-field image (planar image).

[0038] A communication group is a group of users to whom wide-field images are shared (distributed). In a normal space, a communication group is described as a virtual room, in the sense that wide-field images can be shared by each user when they enter the same room. "Virtual" means that it is realized by information processing via a network.

[0039] Users at each location communicate remotely with each other. Remote communication is a meeting held at a remote location. A meeting is when people get together for consultation or discussion. Meetings can take various forms, such as customer service, conferences, meetings, consultations, study sessions, classes, seminars, and presentations. Communication does not necessarily have to be two-way. Therefore, a virtual room may be called a virtual conference room.

[0040] <Example of communication system configuration> 2 is an example of a schematic diagram of the configuration of the communication system 1. The communication system 1 is a system for bidirectionally transmitting and receiving wide-field images and normal-angle images captured by an imaging device 10 between a plurality of bases. The system allows users at a certain base to view images by displaying the images distributed from the other bases. Note that as an example of a wide-field image, a celestial sphere image captured by the imaging device 10 is distributed. The communication system 1 allows, for example, a wide-field image captured at a specific base to be viewed at another base in a remote location.

[0041] As shown in FIG. 2, in the communication system 1, an imaging device 10, a communication terminal 30A, an information processing system 50, and communication terminals 30B and 30C, each of which is located at a plurality of bases (bases B and C), are communicatively connected to each other.

[0042] If the imaging device 10 can be directly connected to the communication network N, the communication terminal 30A is not necessary as a relay device (e.g., a router). In this case, the communication terminal 30A is connected to the communication network N without the imaging device 10. However, if the communication terminal 30A is placed at the site A, the communication terminal 30A also functions as a relay device, and the user A can view the wide-field image in the same way as the communication terminals 30B and 30C. Note that an imaging device 10 may be placed at a site other than the site A, and multiple imaging devices 10 may be placed at the site A.

[0043] Each communication terminal 30 and the information processing system 50 can communicate with each other via a communication network N. The communication network N is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), etc. Note that the communication network N may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0044] As described below, the imaging device 10 is a special digital camera capable of capturing a celestial sphere image by capturing an image of a subject, scenery, or the like and obtaining two original hemispherical images. The wide-field image obtained by the imaging device 10 may be a moving image or a still image, or may be both a moving image and a still image. The captured image may also be a moving image that includes sound together with the image or has multiple images.

[0045] The communication terminal 30 is a computer such as a PC used by a user at each base. The communication terminal 30 displays an image captured at the own base, a wide-field image (still image or video) distributed from another base, and an image with a normal angle of view. The communication terminal 30 acquires a wide-field image captured by the imaging device 10, for example, via a communication network N. In addition, software for performing image processing such as OpenGL ES is installed in the communication terminal 30, and an image display based on viewpoint information that specifies a partial area of ​​the wide-field image can be performed. Note that OpenGL ES is an example of software for performing image processing, and other software may be used. In addition, even if software for performing image processing is not installed in the imaging device 10, the imaging device 10 may perform image processing by software received from the outside, or may display an image by receiving a result of image processing performed by external software. In other words, the communication terminal 30 can display a predetermined partial area of ​​the wide-field image.

[0046] The communication terminal 30 can arbitrarily change the viewpoint relative to the display range of the wide-field image in response to user operation. The communication terminal 30 can change and display the field of view range (predetermined area) based on viewpoint information corresponding to the viewpoint after the movement by moving the virtual viewpoint in response to user operation input (including key input, drag, scroll, etc.) on a touch panel, directional buttons, mouse, keyboard, touch pad, etc. Furthermore, in the case where the communication terminal 30 is a communication terminal worn by a user such as VR goggles, for example, the communication terminal 30 may detect posture information of the communication terminal 30 changed in response to changes in the movement of the user wearing the goggles, and move the virtual viewpoint in response to the detected posture information, thereby changing and displaying the field of view range (predetermined area) based on viewpoint information corresponding to the viewpoint after the movement.

[0047] The communication terminal 30A distributes the wide-field image acquired from the imaging device 10 via a wired cable such as a USB (Universal Serial Bus) cable to the communication terminal 30 at another site via the information processing system 50. The connection between the imaging device 10 and the communication terminal 30A may be a wireless connection using short-range wireless communication or the like instead of a wired connection using a wired cable. A plurality of communication terminals 30A may be arranged at site A.

[0048] A user at site A may wear smart glasses 88, and the smart glasses 88 may connect to the communication network N. In this case, images captured by the smart glasses 88 are transmitted to the information processing system 50 via the communication network N, and the information processing system 50 can distribute the images to the communication terminals 30 at each site.

[0049] The communication terminal 30B is located at a location B where a user B is present, and the communication terminal 30C is located at a location C where a user C is present. A plurality of communication terminals 30B and 30C may be located at the locations B and C. The communication terminals 30B and 30C may be carried by the users B and C, respectively.

[0050] The communication terminals 30A to 30C at the locations A to C may be either terminals with built-in cameras or terminals that can be externally attached. The communication terminals 30A to 30C can distribute images of their own locations captured by their own cameras 8, 9, etc. to other locations. Any device may be installed at the locations A to C.

[0051] The arrangement of each terminal and device (communication terminal 30 and imaging device) and user shown in Fig. 2 is one example, and other examples may be used. Furthermore, communication terminal 30 is not limited to a PC, and may be, for example, a tablet terminal, a smartphone, a wearable device, a projector, an electronic whiteboard (a whiteboard having an electronic whiteboard function capable of mutual communication), an autonomous robot, or the like. Communication terminal 30 may be any computer on which a web browser or an application dedicated to an image distribution service runs.

[0052] Furthermore, if the imaging device 10 has a display, it may be configured to display images distributed from other locations.

[0053] The information processing system 50 has one or more information processing devices. The information processing system 50 manages and controls the communication between the imaging devices 10 and the communication terminals 30 between the various locations, and manages the wide-field images that are transmitted and received. The information processing system 50 provides a platform that can utilize the functions required to provide an image distribution service that distributes wide-field images. This platform may be made available to service providers, such as individuals or companies, who wish to provide image distribution services, by contract. Hereinafter, a service provider that uses the platform to provide an image distribution service to a user is referred to as a platform subscriber, in order to distinguish it from a tenant that receives the image distribution service.

[0054] For this reason, the information processing system 50 may publish an API (Application Programming Interface) as a platform, and the platform subscriber may use this API to provide various image distribution services. The platform subscriber only needs to develop software such as applications that mainly perform screens displayed by the communication terminal 30 and calls the API, and does not need to develop functions provided by the API, such as image distribution, from scratch.

[0055] The information processing system 50 may be constructed by a single computer, or may be constructed by multiple computers to which each unit (function or means) is divided and arbitrarily assigned. In addition, all or part of the functions of the information processing system 50 may be a server computer existing in a cloud environment, or may be a server computer existing in an on-premise environment.

[0056] The storage 90 is a storage device separate from the information processing system 50. The storage 90 may be an external storage (which may be a storage arranged on a cloud or may be a storage arranged on-premise), or may be a storage included in the information processing system 50.

[0057] <Hardware configuration example> Next, the hardware configuration of each device or terminal included in the image communication system according to this embodiment will be described with reference to Figures 3 and 4. Note that components may be added or deleted from the hardware configurations shown in Figures 3 and 4 as necessary.

[0058] <<Hardware configuration of the imaging device>> First, the hardware configuration of the imaging device 10 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the hardware configuration of the imaging device 10. In the following, the imaging device 10 is assumed to be an omnidirectional (all-directional) imaging device using two imaging elements, but the number of imaging elements may be two or more. In addition, the imaging device does not necessarily have to be a device dedicated to omnidirectional imaging, and an omnidirectional imaging unit may be attached to a normal digital camera, smartphone, or the like to have substantially the same functions as the imaging device 10.

[0059] As shown in FIG. 3, the imaging device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0060] Among these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter, when there is no need to distinguish between them, they will be referred to as lenses 102) each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to the lenses. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates a horizontal or vertical synchronization signal or a pixel clock for the image sensor, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set. Note that the configuration in which the imaging unit 101 includes two wide-angle lenses is merely an example, and the imaging unit 101 may include only one wide-angle lens, or may include three or more wide-angle lenses.

[0061] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the input / output I / F 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network I / F 121 are also connected to the bus 110.

[0062] The image processing unit 104 takes in image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create equirectangular projection image data, which will be described later.

[0063] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b, and send it to the CPU 111.

[0064] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function corresponding to video display on a display (for example, a display of an external terminal such as a smartphone that performs short-distance communication with the imaging device 10 using the short-distance communication circuit 117). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / min).

[0065] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via an I / F bus and performs a predetermined process on the sound data.

[0066] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104, and data of a processed equirectangular projection image.

[0067] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. A user operates the operation unit 115 to input various imaging modes, imaging conditions, and the like. Part of the operation unit 115 includes a distribution start button 115a, a fixing button 115b, and a fixing release button 115c. The distribution start button 115a is a button for causing the imaging device 10 to start distributing image data and sound data to another base. The fixing button 115b is a button for repeatedly distributing the same image frame. The fixing release button 115c is a button for releasing the pressing of the fixing button 115b and distributing image frames updated by imaging one after another. The fixing release button 115c and the fixing button 115b may be one alternate button.

[0068] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as necessary.

[0069] The short-range communication circuit 117 communicates with an external terminal (device) by a short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the imaging device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0070] The electronic compass 118 calculates the direction of the imaging device 10 from the earth's magnetism and outputs direction information. This direction information is an example of related information (metadata) according to Exif and is used for image processing such as image correction of the captured image. The related information also includes each data of the image capture date and time and the data capacity of the image data. The gyro sensor 119 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) accompanying the movement of the imaging device 10. The angle changes are an example of related information (metadata) according to Exif and are used for image processing such as image correction of the captured image. The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The imaging device 10 calculates the attitude (angle with respect to the direction of gravity) of its own device (imaging device 10) based on the acceleration detected by the acceleration sensor 120. By providing the acceleration sensor 120, the imaging device 10 improves the accuracy of image correction.

[0071] The network I / F 121 is an interface for performing data communication using a communication network N such as the Internet via a router or the like.

[0072] <<Hardware configuration of communication terminal>> 4 is a diagram showing an example of the hardware configuration of the communication terminal 30 and the information processing system 50. First, the communication terminal 30 will be described. The hardware configuration of the communication terminal 30 is indicated by a reference number in the 300 series. The communication terminal 30 is constructed by a computer, and includes a CPU 301, a ROM 302, a RAM 303, a HD (Hard Disk) 304, a HDD controller 305, a display 306, an external device connection I / F 308, a network I / F 309, a bus line 310, a keyboard 311, a pointing device 312, a DVD-RW (Digital Versatile Disk Rewritable) drive 314, a media I / F 316, an audio input / output I / F 317, a microphone 318, a speaker 319, a short-distance communication circuit 320, and a camera 321, as shown in FIG.

[0073] Among these, the CPU 301 controls the operation of the entire communication terminal 30. The ROM 302 stores programs such as an IPL used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data such as programs. The HDD controller 305 controls reading or writing of various data from or to the HD 304 under the control of the CPU 301. The display 306 displays various information such as a cursor, a menu, a window, a character, or an image. The display 306 is an example of a display unit. The display 306 may be a touch panel display equipped with an input means. The external device connection I / F 308 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory or a printer. The network I / F 309 is an interface for data communication using the communication network N. The bus line 310 is an address bus or a data bus for electrically connecting each component such as the CPU 301 shown in FIG. 4.

[0074] The keyboard 311 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. The input means may be not only the keyboard 311 and the pointing device 312, but also a touch panel, a voice input device, etc. The DVD-RW drive 314 controls reading or writing of various data from the DVD-RW 313 as an example of a removable recording medium. The DVD-RW 313 may be a DVD-R or a Blu-ray (registered trademark) Disc, etc. The media I / F 316 controls reading or writing (storing) of data from the recording medium 315 such as a flash memory. The microphone 318 is a type of built-in sound collecting means for inputting sound. The sound input / output I / F 317 is a circuit for processing input / output of sound signals between the microphone 318 and the speaker 319 under the control of the CPU 301. The short-range communication circuit 320 is a communication circuit for communicating with an external terminal (device) by short-range wireless communication technology such as NFC, Bluetooth (registered trademark), or Wi-Fi. The camera 321 is a type of built-in imaging means that captures an image of a subject and obtains image data. The microphone 318, the speaker 319, and the camera 321 may be external devices rather than built into the communication terminal 30.

[0075] <<Hardware configuration of information processing system>> As shown in Fig. 4, each hardware configuration of the information processing system 50 is indicated by a reference number in parentheses in the 500 range. The information processing system 50 is constructed by a computer, and as shown in Fig. 4, has the same configuration as the communication terminal 30, so a description of each hardware configuration will be omitted.

[0076] Each of the above programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the communication terminal 30 realizes the image display method according to the present invention by executing the program according to the present invention.

[0077] <Wide-field images and viewpoint information> A method for generating a wide-field image (panoramic image) will be described below with reference to FIGS.

[0078] First, the appearance of the imaging device 10 will be described with reference to Fig. 5. The imaging device 10 is a digital camera for obtaining captured images that are the basis of a spherical (360°) image. Fig. 5(a) is a left side view of the imaging device, Fig. 5(b) is a front view of the imaging device, and Fig. 5(c) is a plan view of the imaging device. This external view is merely one example of the imaging device 10, and other external views may be used.

[0079] As shown in FIG. 5(a), the imaging device 10 is sized so that a person can hold it in one hand, but this shape is merely an example and may be other shapes. Also, as shown in FIG. 5(a), FIG. 5(b), and FIG. 5(c), an imaging element 103a is provided on the front side (front side) and an imaging element 103b is provided on the back side (rear side) of the upper part of the imaging device 10. These imaging elements (image sensors) 103a and 103b are used in conjunction with optical members (for example, lenses 102a and 102b described later) capable of capturing hemispherical images (angle of view of 180° or more). Also, as shown in FIG. 5(b), an operation unit 115 such as a shutter button is provided on the surface opposite to the front side of the imaging device 10. As described above, the imaging device 10 may have only one imaging element, or may have three or more imaging elements.

[0080] Next, the usage of the imaging device 10 will be described with reference to Fig. 6. Fig. 6 is an image diagram of the imaging device in use. As shown in Fig. 6, the imaging device 10 is used, for example, to capture an image of a subject around the imaging device 10. In this case, two hemispherical images can be obtained by capturing images of the subject around the imaging device 10 using the imaging element 103a and the imaging element 103b shown in Fig. 5, respectively.

[0081] Next, an outline of the process of creating a celestial sphere image from an image captured by the imaging device 10 will be described with reference to Fig. 7 and Fig. 8. Fig. 7(a) is a diagram showing a hemispherical image (front side) captured by the imaging device, Fig. 7(b) is a diagram showing a hemispherical image (rear side) captured by the imaging device, and Fig. 7(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 8(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 8(b) is a diagram showing a celestial sphere image.

[0082] As shown in Fig. 7(a), the image obtained by the imaging element 103a becomes a hemispherical image (front side) curved by the lens 102a described below. Also, as shown in Fig. 7(b), the image obtained by the imaging element 103b becomes a hemispherical image (rear side) curved by the lens 102b described below. Then, the imaging device 10 synthesizes the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees to generate an equirectangular projection image EC as shown in Fig. 7(c).

[0083] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to paste an equirectangular projection image EC so as to cover the sphere as shown in FIG. 8(a) and create a celestial sphere image (celestial sphere panoramic image) CE as shown in FIG. 8(b). In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. OpenGL ES is merely an example of software that executes image processing, and the celestial sphere image CE may be created by other software. Also, the celestial sphere image CE may be a still image or a video.

[0084] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to humans. Therefore, the imaging device 10 can display a predetermined region T (hereinafter, referred to as a "predetermined region image") that is a part of the spherical image CE as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to humans. This will be described with reference to Figs. 9 and 10.

[0085] FIG. 9 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional sphere. FIG. 10(a) is a three-dimensional perspective view of FIG. 9, and FIG. 10(b) is a diagram showing the predetermined area image when displayed on a display. FIG. 10(a) shows the omnidirectional image CE shown in FIG. 9 as a three-dimensional sphere CS. If the omnidirectional image CE generated in this way is a sphere CS, as shown in FIG. 9, the virtual camera IC is located inside the omnidirectional image CE. The predetermined area T in the omnidirectional image CE is an imaging area of ​​the virtual camera IC, and is specified by predetermined area information indicating the imaging direction and the angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. In addition, the zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the omnidirectional image CE. The predetermined area image Q is an image of a predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 11).

[0086] Then, the predetermined area image Q shown in Fig. 10(a) is displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in Fig. 10(b). The image shown in Fig. 10(b) is a predetermined area image represented by the initial (default) predetermined area information. The following description will be given using the imaging direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be represented by the position coordinates (X, Y, Z) of the imaging area of ​​the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

[0087] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 11. FIG. 11 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 11, "ea" indicates an elevation angle, "aa" indicates an azimuth angle, and "α" indicates an angle of view. That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the imaging direction (ea, aa) becomes the center point CP(x, y) of the predetermined area T, which is the imaging area of ​​the virtual camera IC. As shown in FIG. 11, the center point CP(x, y) when the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC is α becomes the parameter ((x, y)) of the predetermined area information. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between any vertex of the predetermined region T and the center point CP(x, y) (2L is the diagonal). In FIG. 11, the trigonometric function generally expressed by the following (Equation 1) holds.

number

[0088] FIG. 12 is a diagram showing the relationship described in FIG. 11 by points in a three-dimensional Euclidean space by spherical coordinates. Here, the position coordinates when the center point CP shown in FIG. 11 is expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are a radius, a polar angle, and an azimuth angle, respectively. The radius r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to the center point CP, and is therefore equal to the distance f shown in FIG. 11. FIG. 12 is a diagram showing these relationships. Hereinafter, the position coordinates (r, θ, φ) of the virtual camera IC will be used as an example of viewpoint information for explanation. Note that the viewpoint information may be parameter information that can specify the predetermined area T (predetermined area image Q) displayed as an image of the imaging area of ​​the virtual camera IC on the predetermined display shown in FIG. 10 as described above, and also includes the coordinates of the diagonal vertices of the predetermined area T. In addition, the viewpoint information may be information indicating the angle of view α of the virtual camera IC described in FIG. 11 and information indicating the center point CP (x, y). Moreover, the viewpoint information includes not only position coordinate information based on spherical coordinates, but also position coordinate information based on Cartesian coordinates and a difference value of coordinates from the initially set (default) predetermined area information. Moreover, the viewpoint information may be information other than coordinate information such as an angle or distance as shown in FIG. 11. Moreover, although the center point of the predetermined area T is used as a reference in FIG. 11 and FIG. 12, the predetermined area T may be specified by parameter information based on one of the vertices of the predetermined area T. Note that, although the viewpoint information has been described above by taking the case where the wide-field image is a celestial sphere image as an example, in the case of other wide-field images, information that specifies the predetermined area T in the wide-field image becomes the viewpoint information.

[0089] <About the function> Next, the functional configuration of the communication system 1 according to the present embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the functional configuration of the communication system 1 according to the present embodiment. Note that Fig. 13 shows those terminals, devices, and servers shown in Fig. 1 that are related to the processes or operations described below.

[0090] <<Functional configuration of the imaging device>> First, the functional configuration of the imaging device 10 will be described with reference to Fig. 13. The imaging device 10 has a communication unit 11, a reception unit 12, an imaging processing unit 13, an analysis unit 14, a registration request unit 15, a connection unit 16, a storage processing unit 17, an image transmission control unit 18, a sound collection unit 19, an encoding unit 20, a control unit 21, an object detection unit 22, a timer management unit 23, a motion detection unit 24, and a storage / readout unit 29. Each of these units is a function or a means that is realized by any of the components shown in Fig. 3 operating according to an instruction from the CPU 111 according to a program loaded on the SRAM 113 or the DRAM 114. The imaging device 10 also has a storage unit 1000 constructed by the ROM 112 shown in Fig. 3, etc.

[0091] The communication unit 11 has a function of connecting to the communication network N using a wireless communication means such as Wi-Fi and transmitting / receiving various data or information to / from other devices. In this embodiment, a form in which the wide-field image acquired by the imaging processing unit 13 is transmitted to the information processing system 50 via the connection unit 16 will be mainly described. However, it is also possible for the communication unit 11 to transmit the wide-field image to the information processing system 50.

[0092] The reception unit 12 has a function of receiving operation input from a user to the imaging device 10. The reception unit 12 receives input such as power on / off and shutter button on / off (start or stop transmission of a wide-field image).

[0093] The imaging processing unit 13 captures an image of a subject, a landscape image, or the like, and acquires the captured image. The captured image acquired by the imaging processing unit 13 may be a moving image or a still image (or both), and may include sound together with the image. The imaging processing unit 13 also captures, for example, a two-dimensional code displayed on the display 306 of the communication terminal 30.

[0094] The analysis unit 14 analyzes the two-dimensional code captured and acquired by the imaging processing unit 13, and extracts information included in the two-dimensional code (a URL for registering the imaging device to a tenant, and a temporary ID and password).

[0095] The registration request unit 15 uses the information included in the two-dimensional code read by the analysis unit 14 to transmit a request to the information processing system 50 to register the imaging device 10 with the tenant of the information processing system 50 .

[0096] The connection unit 16 is realized by, for example, a short-distance communication circuit 117, and has a function of receiving power supply from the communication terminal 30A and performing data communication.

[0097] The storage processing unit 17 performs processing to store a wide-field image captured in response to an imaging request from an arbitrary location in a URL notified by the information processing system 50 (for example, the storage 90, etc.).

[0098] The image transmission control unit 18 is a function for controlling the transmission of wide-field images to the information processing system 50. For example, the image transmission control unit 18 transmits captured images acquired by the imaging processing unit 13 to the information processing system 50 periodically or in response to a user operation if the captured images are still images, or at a predetermined FPS (Frame Per Second) if the captured images are moving images. The image transmission control unit 18 also switches between the communication unit 11 and the connection unit 16.

[0099] The sound collection unit 19 collects surrounding sounds and performs PCM sampling to generate sound data.

[0100] The encoding unit 20 encodes the image data using an encoding method such as H.264, H.265, AV1, VVC (Versatile Video Coding), Xvid, Divx, or MPEG-4.

[0101] The reception unit 12 receives input to the imaging device from a user. The reception unit 12 receives input sent not only from a user who uses the imaging device, but also from a user who uses the communication terminal 30.

[0102] The control unit 21 stores image frames to suppress camera shake, controls a fixation flag, and controls transmission of image frames according to the state of the fixation flag, etc. Details will be described later.

[0103] The object detection unit 22 analyzes the image data and detects a predetermined object reflected in the image data. The type of object that the object detection unit 22 detects (the predetermined object) is set in advance.

[0104] The timer management unit 23 measures the time since a fixed flag, which will be described later, is set to ON using a timer, and detects that a certain period of time has elapsed.

[0105] The motion detection unit 24 detects whether the imaging device 10 is moving or stopped, that is, whether the user A is moving or not, based on a signal from the acceleration sensor 120 and the like.

[0106] The storage / readout unit 29 has a function of storing various data in the storage unit 1000 or reading out various data from the storage unit 1000. The storage unit 1000 also stores image data acquired by the imaging processing unit 13, an imaging device ID, and the like. The image data stored in the storage unit 1000 may be configured to be deleted when a predetermined time has elapsed since the image data was acquired by the imaging processing unit 13, or the data transmitted to the information processing system 50 may be deleted.

[0107] In addition, an application (also called a plug-in) for compatibility with the communication system 1 is installed in the imaging device 10. This application is not necessary for a commercially available imaging device 10, but is used when associating the imaging device 10 with a virtual room and accepting control from outside. Some of the functions shown in Fig. 13 (for example, the registration request unit 15) are provided by this application.

[0108] <<Functional configuration of communication terminal>> Next, the functional configuration of the communication terminal 30 will be described with reference to Fig. 13. The communication terminal 30 has a communication unit 31, a reception unit 32, a display control unit 33, an imaging unit 34, a storage / readout unit 35, and a connection unit 36. Each of these units is a function or a means for performing a function that is realized by operating any of the components shown in Fig. 4 according to an instruction from the CPU 301 in accordance with a program (which may be a web browser or a dedicated application) deployed on the RAM 303. The communication terminal 30 also has a storage unit 3000 constructed by the ROM 302 or the recording medium 315 shown in Fig. 4.

[0109] The communication unit 31 is realized by, for example, a network I / F 309, and has a function of connecting to a communication network N and transmitting / receiving various data or information to / from other devices.

[0110] The reception unit 32 is a function that receives various selections or operation inputs to the communication terminal 30. The display control unit 33 is a function that displays a wide-field image, an image with a normal angle of view, and various screens on the display 306 of the communication terminal 30. The display control unit 33 displays, for example, a two-dimensional code transmitted from the information processing system 50 on the display 306. The two-dimensional code is, for example, a QR code (registered trademark), DataMatrix (DataCode), MaxiCode, PDF417, or the like. The two-dimensional code may be a barcode.

[0111] The connection unit 36 ​​is realized by, for example, a short-range communication circuit 320, and has a function of supplying power to the image capture device 10 and performing data communication.

[0112] 4, and has the function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. An image management information storage unit 3001 is formed in the storage unit 3000. The image management information storage unit 3001 will be described in detail in the description of the information processing system 50.

[0113] <<Functional configuration of information processing system>> Next, the functional configuration of the information processing system 50 will be described. The information processing system 50 has a communication unit 51, a screen generation unit 52, an association processing unit 53, an image delivery unit 54, an authentication unit 55, a communication group management unit 56, a communication control unit 57, a connection management unit 58, a storage / reading unit 59, and an API management unit 60. Each of these units is a function or a means for performing a function that is realized by operating any of the components shown in Fig. 4 by an instruction from the CPU 501 according to a program loaded on the RAM 503. The information processing system also has a storage unit 5000 constructed by the ROM 502, HD 504, or recording medium 515 shown in Fig. 4.

[0114] The communication unit 51 has a function of transmitting and receiving various data or information to and from other devices via the communication network N.

[0115] The screen generation unit 52 generates screen information to be displayed by the communication terminal 30. When the communication terminal 30 executes a Web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), and the like. When the communication terminal 30 executes a native application, the screen information is held by the communication terminal 30, and the information to be displayed is transmitted in XML or the like. The screen generation unit 52 generates screen information on which the wide-field image and the like delivered by the image delivery unit 54 are arranged.

[0116] The association processing unit 53 performs control regarding sharing of viewpoint information of the wide-field image. When the association processing unit 53 receives an imaging request together with viewpoint information from the communication terminal 30, the association processing unit 53 performs processing to associate the wide-field image acquired by requesting imaging from the imaging device 10 with the viewpoint information. Furthermore, the association processing unit 53 stores the associated wide-field image and viewpoint information in the image management information storage unit 5001. The association processing unit 53 also transmits a URL to the communication terminal 30 as information indicating a storage location where the associated wide-field image and viewpoint information are stored. Note that the information processing system 50 does not need to simultaneously receive the viewpoint information and the imaging request from the communication terminal 30, and may perform processing to associate them after receiving them separately. Also, the URL is an example of information indicating a storage location, and may be in another format such as a URI.

[0117] The image delivery unit 54 delivers a wide-field image transmitted by the imaging device 10 associated with the same virtual room to the communication terminal 30 operated by a user who is in the virtual room. Images with a normal angle of view captured by the camera of the communication terminal 30 or the connected cameras 8 and 9 are also delivered in the same manner.

[0118] The authentication unit 55 has a function of authenticating the requester based on an authentication request received by the communication unit 51. The authentication unit 55 authenticates the user based on, for example, whether or not authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches authentication information stored in advance. Note that the authentication information may be biometric authentication information such as an IC card number, face, fingerprint, or voiceprint. The authentication unit 55 may also perform authentication using an external authentication system or an authentication method such as OAuth.

[0119] The communication group management unit 56 manages the entry of the communication terminal 30 and the user into the virtual room, the association of devices, etc. When the authentication unit 55 is successful, the communication group management unit 56 registers the user ID and the IP address of the communication terminal 30 in the virtual room information storage unit 5002, and associates the imaging device 10 with the virtual room.

[0120] The communication control unit 57 manages the start, establishment, and end of communication with the imaging device 10 associated with each virtual room. The communication control unit 57 also manages the start, establishment, and end of communication for delivering wide-field images and audio in response to the communication terminal 30 entering or leaving a virtual room.

[0121] The connection management unit manages the communications (connections) established between the communication terminal 30 and the imaging device 10 and the information processing system 50 in association with the virtual room.

[0122] The API management unit 60 manages the API used when the platform contractor provides an image delivery service of a wide-field image. When using the API, the platform contractor may separately develop software that calls the API. The developed software may run on a server or on a client such as a communication terminal. Any function that the information processing system 50 has, such as the image delivery unit 54, the association processing unit 53, the communication control unit 57, etc., can be provided as an API. In addition, it is also possible to provide a function added to the information processing system 50 later as an API. Whether or not to provide it as an API is determined by a communication terminal operated by the platform provider accessing the information processing system 50 and accepting the public setting of the API, so that the API management unit 60 can control the API based on the public setting. In addition, the API management unit 60 may perform an authentication process to confirm whether the software that requests the API call is software developed by a legitimate platform contractor. The authentication process can be confirmed by comparing the information stored in the platform contractor information storage unit with the information transmitted from the requesting software. As a specific example of the authentication process, the information processing system 50 receives from the requesting software an application ID issued in advance by the API management unit 60 for software developed by a platform contractor, and if the API management unit 60 determines that the application ID is stored in the platform contractor information storage unit, the API management unit 60 performs control to permit the provision of the API as legitimate software. On the other hand, if the software cannot be determined to be legitimate, the API management unit 60 performs control not to permit the provision of the API. Note that the application ID is an example of authentication information for determining legitimacy, and the API management unit 60 may confirm the legitimacy of the requesting source using authentication information such as an access token, ticket, security key, password, or PIN code issued in advance by the API management unit 60 of the information processing system or an external system.In this embodiment, the form of using the functions of the information processing system 50 as an API is not described, but the processing flow may be the same except that software such as an application developed by the platform subscriber uses the functions of the information processing system 50 via judgment in the API management unit 60.

[0123] The storage / reading unit 59 has a function of storing various data in the storage unit 5000 or reading various data from the storage unit 5000. The storage unit 5000 also includes an image management information storage unit 5001, a virtual room information storage unit 5002, and a tenant information storage unit 5003.

[0124] "Image management information storage unit 5001" Fig. 14(a) is a conceptual diagram showing image management information stored in image management information storage unit 5001. Image management information such as that shown in Fig. 14 is stored in image management information storage unit 5001. Image management information is information for managing wide-field images captured in response to an imaging request, and when a user transmits an imaging request from communication terminal 30, one record of image management information is generated. Each item contained in the image management information will be described.

[0125] The data ID is identification information for identifying the wide-field image. The data ID is assigned by the information processing system 50. ID is an abbreviation for Identification and means an identifier or identification information. The ID refers to a name, a code, a character string, a number, or a combination of one or more of these that is used to uniquely distinguish a specific object from multiple objects.

[0126] The data name is the name of the wide-field image set by the user of the communication terminal 30. The data name can be set by the user, but may also be set automatically.

[0127] The imaging date and time information is information for identifying the imaging date and time of the wide-field-of-view image, such as the date and time when the user inputs an imaging request to the communication terminal 30, or the date and time when the imaging device 10 captures the wide-field-of-view image. The imaging date and time information may be replaced with a time stamp of the wide-field-of-view image.

[0128] The image capturer information is identification information (which may be a user name) of the user who inputs an image capture request to the communication terminal 30. Since the user inputs an image capture request to the communication terminal 30 while in a virtual room, the user registered in the image capturer information is identified by authentication to the information processing system 50 or the virtual room. The image capturer information is transmitted to the information processing system 50 together with the image capture request.

[0129] The imaging device information is identification information (imaging device ID) of the imaging device 10 that captured the wide-field image. The imaging device ID is assigned by the information processing system 50 and shared with the imaging device 10, but information unique to the imaging device 10, such as a MAC address or serial number, may also be used. The imaging device ID is transmitted to the information processing system 50 together with the wide-field image.

[0130] The predetermined area information of the photographer is viewpoint information consisting of a radius vector, a polar angle, and an azimuth angle. The viewpoint information indicates the central coordinates of the wide-field-of-view image being displayed by the communication terminal 30. The viewpoint information is transmitted from the communication terminal 30 that requests imaging. Note that the viewpoint information may include information specifying the width and height of the display range in addition to the radius vector, polar angle, and azimuth angle. Alternatively, the viewpoint information may be only the width and height of the display range.

[0131] The virtual room ID at the time of imaging is identification information of the virtual room with which the imaging device 10 is associated.

[0132] - Data storage location information is the URL or file path where the wide-field image is saved.

[0133] Fig. 14(b) is also a conceptual diagram showing image management information. In Fig. 14(b), wide-field images having the same virtual room ID at the time of capture are stored. Image management information may be classified by virtual room.

[0134] <Entering a virtual room from a communication device> Next, the process of user B entering a virtual room will be described with reference to Fig. 15 and Fig. 16. It is assumed that the imaging device 10 has already been associated with the virtual room, and the communication terminal 30A has already transmitted a wide-field image and an image with a normal angle of view to the information processing system 50 (association of the imaging device 10 with the virtual room will be described in Fig. 18 and subsequent figures). In the following description, there is no particular distinction between user B entering a virtual room and communication terminal 30B operated by user B entering a virtual room.

[0135] FIG. 15 shows an example of a screen displayed by communication terminal 30B when user B enters a virtual room. FIG. 15(a) is an example of an entrance screen 200. More specifically, prior to the display of entrance screen 200, user B logs in to information processing system 50. By logging in, the tenant to which user B belongs is identified. Virtual rooms are associated with tenants. User B displays a list of virtual rooms associated with the tenants on communication terminal 30B (see FIG. 17), and selects the virtual room to enter from the list. FIG. 15(a) is the entrance screen 200 for the virtual room selected by user B in this manner.

[0136] Alternatively, the creator of the virtual room may request the information processing system 50 to issue a URL corresponding to the virtual room, and may send this URL by email or the like to user B. When user B presses the URL displayed on communication terminal 30B, communication terminal 30B displays entry screen 200 of FIG.

[0137] The entry screen 200 has a virtual room name 201, a participant name input field 202, and an enter button 203. The virtual room name 201 is the same as that stored in the virtual room information storage unit 5002. The participant name input field 202 may be a nickname or other name for user B. Since the user name of user B is identified when user B logs in, this user name may be displayed automatically. The enter button 203 is a button for user B to request entry into the virtual room.

[0138] Note that authentication for entering the virtual room may be required at the time of entry, separately from logging in to the tenant.

[0139] FIG. 15(b) shows an image viewing screen 210 displayed by the communication terminal 30B when the user B enters the virtual room. In the image viewing screen 210 of FIG. 15(b), the imaging device 10 has already started distributing a wide-field-of-view image, and the communication terminal 30A has already started distributing an image with a normal field of view. Therefore, the image viewing screen 210 has a first image field 211 and a second image field 212. The wide-field-of-view image is displayed in the first image field 211, and the image with a normal field of view is displayed in the second image field 212. If there are three or more locations transmitting images, the image viewing screen 210 is divided according to the number of locations from which the images are transmitted.

[0140] A wide-field-of-view image mark 213 is displayed in the first image field 211. The wide-field-of-view image mark 213 is set by the screen generation unit 52 of the information processing system 50 when it determines that the image to be displayed in the first image field 211 is a wide-field-of-view image. The wide-field-of-view image mark 213 may be determined and displayed by the communication terminal 30B. By looking at the wide-field-of-view image mark 213, user B knows that the viewpoint can be changed. In addition, a device name 214 (transmitted from the imaging device 10 together with the wide-field-of-view image) is displayed in the first image field 211. The device name 214 is set by user A as described below (see FIG. 19).

[0141] A participant name 215 is displayed in the second image field 212. As the participant name 215, the participant name of a user who has already entered the room (in this case, since user A has already entered the room, "AAA" entered by user A in the participant name input field 202) is displayed in the participant name input field 202.

[0142] FIG. 16 is a sequence diagram illustrating a process in which user B (or communication terminal 30B) enters a virtual room.

[0143] S1: First, user B at site B performs an operation to display a list screen of virtual rooms. When the reception unit 32 receives the operation to display the list screen, the display control unit 33 of the communication terminal 30B causes the display 306 to display a selection screen.

[0144] S2: When the user B selects a selection button for a certain virtual room, the reception unit 32 of the communication terminal 30B receives the selection of the virtual room. The display control unit 33 of the communication terminal 30B causes the display 306 to display the entry screen 200.

[0145] S3: User B enters the necessary information and presses the enter button 203. When the reception unit 32 receives the press, the communication unit 31 of the communication terminal 30B transmits a request to enter the virtual room to the information processing system 50. This entry request includes the virtual room ID indicating the virtual room selected in step S2, the user ID of the logged-in user B, and the IP address of the communication terminal 30B that is the request source terminal.

[0146] S4: As a result, the communication unit 51 of the information processing system 50 receives the entry request. The communication group management unit 56 registers the logged-in user ID and IP address in the virtual room information specified by the virtual room ID in the virtual room information storage unit 5002.

[0147] S5: Then, the communication unit 51 of the information processing system 50 transmits a "Already entered" message to the communication terminal 30B. As a result, the communication unit 31 of the communication terminal 30B receives the "Already entered" message.

[0148] <Associating imaging devices with rooms> FIG. 17 shows an example of a virtual room association screen (part 1) 260 for associating the imaging device 10 with a virtual room. The screen configuration may be the same for the VR goggles 89 and the smart glasses 88. The virtual room association screen (part 1) 260 has a list 261 of virtual rooms. The list 261 of virtual rooms displays individual virtual room fields 262 to 264 based on the virtual rooms created in the tenant. Each of the individual virtual room fields 262 to 264 has a link issue button 265, an entry button 266, a setting button 267, and a virtual room name 268. The link issue button 265 is a button for issuing a link (URL for invitation) and a passcode to the virtual room. The entry button 266 is a button for user A to enter the virtual room. The setting button 267 is a button for associating the imaging device 10 with the virtual room. The virtual room name 268 is the same as that stored in the virtual room information storage unit 5002. Therefore, user A presses set button 267. By pressing set button 267, communication terminal 30A displays virtual room association screen (part 2) 270.

[0149] Furthermore, if a device has already been associated with a virtual room, the name 269 of the device is displayed in the individual virtual room field (individual virtual room field 264 in the figure).

[0150] 18 shows an example of virtual room matching screen (part 2) 270. Note that virtual room matching screen (part 2) 270 is displayed as a pop-up on virtual room matching screen (part 1) 260. A screen transition from virtual room matching screen (part 1) 260 to virtual room matching screen (part 2) 270 does not go through information processing system 50, but a screen transition that goes through information processing system 50 is also possible.

[0151] The virtual room association screen (part 2) 270 has a name 271 (not yet registered and therefore not registered in the figure) of the image capture device 10 currently (already) associated with the virtual room, a connection button 272, and a storage button 273. The connection button 272 is a button for displaying a list of devices registered in the tenant. The storage button 273 is a button for displaying a list of storages 90 in which the image capture device 10 associated with the virtual room stores wide-field images. Pressing the connection button 272 causes the communication terminal 30A to display the virtual room association screen (part 3).

[0152] The communication terminal 30A sends a virtual room ID to the information processing system 50 and obtains the names of devices (including device IDs, etc.) registered in the tenant in which the virtual room is generated, and the names of devices (including device IDs, etc.) associated with the virtual room.

[0153] 19 shows an example of a virtual room association screen (part 3) 280. The virtual room association screen (part 3) 280 has a name 281 of the imaging device 10 currently (already) associated with the virtual room, an addable device list 282, and a save button 283. User A selects a device to be associated with the virtual room from the addable device list 282, and presses the save button 283. This associates the device with the virtual room (the imaging device ID is registered in the virtual room information storage unit 5002).

[0154] <Wide-field image transmission start process for imaging device> In the above manner, devices such as the image capture device 10 are associated with the virtual room, but user A needs to operate the device to start transmitting images.

[0155] For the VR goggles 89 and smart glasses 88, user A turns on and off the transmission of images by operating the device itself. This is because currently, a dedicated application is not running in the communication system 1 for the VR goggles 89 and smart glasses 88. If a dedicated application runs in the communication system 1 for the VR goggles 89 and smart glasses 88 as well, user A can remotely turn on and off the transmission of images.

[0156] In the case of the imaging device 10, if the application is enabled, user A can enter the virtual room and turn on and off the transmission of wide-field images from the menu.

[0157] FIG. 20 is an example of a wide-field-of-view image transmission start / stop dialogue 290 displayed by the communication terminal 30A. The wide-field-of-view image transmission start / stop dialogue 290 is displayed as a pop-up on the image viewing screen 210. It is assumed that the user A operates the communication terminal 30A to enter a virtual room to which the imaging device 10 is associated. The wide-field-of-view image transmission start / stop dialogue 290 displays a name 292 of the imaging device 10 associated with this virtual room. A toggle button 291 is displayed near the name 292, and the user A can operate the toggle button 291 to set the transmission of the wide-field-of-view image by the imaging device 10 to ON or OFF. Note that the method of setting ON or OFF using the toggle button is one example, and it is sufficient as long as it can be set according to the input of the user operation. For example, the user A may set it by selecting a radio button or a predetermined icon, operating a menu, or the like. In addition, the transmission of the wide-field-of-view image may be automatically started after the imaging device 10 enters the room without requiring a user operation. In addition, certain conditions, such as the date and time, the number of users who have entered, or the participation of a specific user, may be determined in advance, and the transmission of the wide-field image may begin when it is determined that the conditions are met.

[0158] The communications terminal 30A transmits the state of the toggle button 291 to the information processing system 50. The information processing system 50 transmits a transmission start request or a transmission stop request according to the state of the toggle button 291 to the imaging device 10.

[0159] Fig. 20(a) shows a state in which the toggle button 291 is off. Therefore, no wide-field image is displayed in Fig. 20(a). On the other hand, in Fig. 20(a), when the communication terminal 30A enters the room, an image with a normal angle of view captured by the camera 9 of the communication terminal 30A has already been shared and is displayed on the image viewing screen 210.

[0160] 20(b) shows a state in which the toggle button 291 is on. When the toggle button 291 is on, the information processing system 50 transmits an on request to the imaging device 10, and the imaging device 10 starts transmitting a wide-field image. As a result, two images are shared in one virtual room, and the image viewing screen 210 is divided into two.

[0161] <<How to register an imaging device in a virtual room>> Next, a procedure for registering the imaging device 10 in the virtual room described in the screen transition will be described with reference to Fig. 21. Fig. 21 is an example of a sequence diagram showing a procedure for user A to register the imaging device 10 in the virtual room.

[0162] S11: First, user A connects the communication terminal 30A to the information processing system 50, inputs authentication information (user ID, password, etc.), and requests to log in to the tenant to which the user belongs. The reception unit 32 of the communication terminal 30A receives the operation.

[0163] S12: The communication unit 31 of the communication terminal 30A designates authentication information and transmits a login request to the information processing system 50. The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 55 performs authentication. Here, it is assumed that the authentication is successful.

[0164] S13: In response to a user operation, the screen generation unit 52 of the information processing system 50 generates the device registration screen 220, and the communication unit 51 transmits screen information of the device registration screen 220 to the communication terminal 30A.

[0165] S14: The communication unit 31 of the communication terminal 30A receives the screen information of the device registration screen 220, and the display control unit 33 displays the device registration screen 220. The user A selects the type of device (assuming that the imaging device 10 is selected here), and then inputs the name and description of the imaging device 10. The reception unit 32 receives the input.

[0166] S15: The communication unit 31 of the communication terminal 30A transmits a request for a two-dimensional code to the information processing system 50, specifying the name and description input by user A.

[0167] S16: The communication unit 51 of the information processing system 50 receives the request for the two-dimensional code. The communication group management unit 56 generates a URL (a connection destination for registration) in association with the name and description, and generates a two-dimensional code including the URL, a temporary ID, and a password. The communication unit 51 of the information processing system 50 transmits the two-dimensional code to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the two-dimensional code, and the display control unit 33 displays the two-dimensional code.

[0168] S17: Next, the user A operates the imaging device 10 that he / she wishes to associate with the virtual room to capture an image of the two-dimensional code. The reception unit 12 of the imaging device 10 receives the operation.

[0169] S18: The imaging processing unit 13 of the imaging device 10 generates image data by performing imaging processing including the two-dimensional code, and the analysis unit 14 analyzes the image data to extract the URL, the temporary ID, and the password. As a result, the registration request unit 15 connects to the URL via the connection unit 16, and transmits a registration request for the imaging device 10 to the information processing system 50 by specifying the temporary ID and password.

[0170] S19: The communication unit 51 of the information processing system 50 receives the temporary ID and password, and determines whether or not they match the temporary ID and password associated with the URL to which the authentication unit 55 is connected. Here, it is assumed that the temporary ID and password match.

[0171] S20: Since registration of the imaging device 10 is requested, the communication group management unit 56 of the information processing system 50 generates an imaging device ID and registers it in the tenant where the user A logs in. Note that a name and a description are associated with the imaging device ID.

[0172] S21: The communication unit 51 of the information processing system 50 transmits the imaging device ID to the imaging device 10. The connection unit 16 of the imaging device 10 receives the imaging device ID and stores it in the storage unit 1000.

[0173] S22: The communication terminal 30A is notified of the completion of registration, and as a result, user A starts associating the imaging device 10 with the virtual room. User A causes the communication terminal 30A to display the virtual room association screen (part 1) 260, and selects the virtual room to which the imaging device 10 registered in the tenant is to be associated. The reception unit 32 of the communication terminal 30A receives the selection.

[0174] S23: Next, User A displays the virtual room association screen (part 2) 270 on the communication terminal 30A and presses “Add Device.” The reception unit 32 of the communication terminal 30A receives the press.

[0175] S24: The communication unit 31 of the communication terminal 30A requests the information processing system 50 to transmit the devices registered with the tenant and the devices associated with the virtual room ID selected in step S23.

[0176] S25: The communication unit 51 of the information processing system 50 receives a request for the devices registered with the tenant and the devices associated with the virtual room ID, and the screen generation unit 52 generates a virtual room matching screen (third) 280 including the device IDs of the devices registered with the tenant and the devices associated with the virtual room ID. The communication unit 51 of the information processing system 50 transmits screen information of the virtual room matching screen (third) 280 to the communication terminal 30A.

[0177] S26: The communication unit 31 of the communication terminal 30A receives the screen information of the virtual room matching screen (part 3) 280, and the display control unit 33 displays the virtual room matching screen (part 3) 280. The user A selects the imaging device 10 to be matched with the virtual room. The reception unit 32 of the communication terminal 30A receives the selection, and the imaging device ID is identified.

[0178] S27: The communication unit 31 of the communication terminal 30A transmits an association request to the information processing system 50, specifying the virtual room ID selected in step S23 and the imaging device ID selected in step S27.

[0179] S28: The communication unit 51 of the information processing system 50 receives the association request, and the communication group management unit 56 registers the imaging device 10 in the virtual room. That is, the communication group management unit 56 registers the imaging device ID in the virtual room information storage unit 5002.

[0180] S29: Now that the imaging device ID has been associated with the virtual room, the communication unit 51 of the information processing system 50 transmits the virtual room ID, name, and description to the imaging device 10. The information processing system 50 may use a push notification, or the imaging device 10 may transmit the information by polling. The connection unit 16 of the imaging device 10 receives the virtual room ID, name, and description, and stores them in the storage unit 1000. As a result, when the imaging device 10 transmits a wide-field image, it can attach the imaging device ID, virtual room ID, name, description, etc.

[0181] S30: The communication terminal 30A is notified of the completion of the association, and as a result, the user A turns on the toggle button 291 of the imaging device 10 associated with the virtual room on the image viewing screen 210. The reception unit 32 of the communication terminal 30A receives the on command.

[0182] S31: The communication unit 31 of the communication terminal 30A specifies the imaging device ID and transmits a request to start transmitting a wide-field-of-view image to the information processing system 50. Note that user A may directly operate a button on the imaging device 10 to start transmitting a wide-field-of-view image. Note that user A's operation may also cause the communication unit 31 of the communication terminal 30A to transmit a request to stop transmission to the information processing system 50.

[0183] S32: The communication unit 51 of the information processing system 50 receives the transmission start request and requests the imaging device 10 identified by the imaging device ID to start transmission. The information processing system 50 may use push notification, or the imaging device 10 may use polling. The connection unit 16 of the imaging device 10 receives the transmission start request, and the imaging processing unit 13 starts imaging. The image transmission control unit 18 repeats transmission of the wide-field image via the connection unit 16 at a determined FPS or an FPS according to the bandwidth. Therefore, the communication terminal 30 that has entered the virtual room can display the situation at site A on the image viewing screen 210 in real time.

[0184] <Distribution of wide-field images, etc.> With reference to Fig. 22, a flow in which a wide-field image and an image with a normal angle of view are shared will be described. Fig. 22 is an example of a sequence diagram illustrating a flow in which a wide-field image is shared. In Fig. 22, communication terminals 30A and 30B have entered the virtual room. Furthermore, communication terminal 30A has a camera 9 with a normal angle of view, which is shared with communication terminal 30B. Images captured by smart glasses 88 associated with the virtual room, instead of the camera 9 of communication terminal 30A, may be shared.

[0185] S41: The imaging unit 34 of the communication terminal 30A repeatedly captures images, and the communication unit 31 designates the virtual room ID of the currently entered room and repeatedly transmits the images and audio to the information processing system 50.

[0186] S42, S43: When the communication unit 51 of the information processing system 50 receives the image and sound, the image distribution unit 54 acquires the IP addresses of the communication terminals 30A, 30B that have entered the virtual room from the virtual room information storage unit 5002, and transmits the image and sound via the communication unit 51. Note that in Fig. 22, the communication unit 31 of the communication terminal 30A receives and displays an image with a normal angle of view from the information processing system 50, but the image with a normal angle of view captured by the imaging unit 34 may be displayed without receiving it.

[0187] S44: Next, in response to a transmission start request made by the imaging device 10 turning on the toggle button 291, the imaging processing unit 13 repeatedly captures wide-field images, and the image transmission control unit 18 specifies the virtual room ID, imaging device ID, name, and description via the connection unit 16, and repeatedly transmits the wide-field images and audio to the information processing system 50.

[0188] S45, S46: When the communication unit 51 of the information processing system 50 receives the wide-field image and audio, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A, 30B that have entered the virtual room from the virtual room information storage unit 5002, and transmits the wide-field image and audio via the communication unit 51.

[0189] S47: Next, a communication terminal 30C equipped with a camera 9 newly enters the virtual room.

[0190] S48: The communication unit 31 of the communication terminal 30C transmits to the information processing system 50 an image and sound with a normal angle of view.

[0191] S49-S51: The communication unit 51 of the information processing system 50 receives images and audio with a normal angle of view from the communication terminal 30C, obtains the IP addresses of the communication terminals 30A-30C that have entered the virtual room from the virtual room information storage unit 5002, and the image distribution unit 54 transmits the images and audio with a normal angle of view.

[0192] S52: Furthermore, the communication unit 51 of the information processing system 50 also transmits the wide-field image and audio to the communication terminal 30C that has entered the same virtual room.

[0193] In this way, users A and B who enter the same virtual room can share the wide-field image captured by the imaging device 10 associated with the virtual room. Note that the order of transmission of the images shown in Fig. 22 is an example, and the wide-field image may be shared first, or the normal angle of view image may be shared first.

[0194] A supplementary explanation about the smart glasses 88 and the VR goggles 89. The smart glasses 88 have a camera with a normal angle of view and a display function. An image with a normal angle of view captured by a camera held by the smart glasses 88 is distributed in the same manner as the cameras 8 and 9. The display function held by the smart glasses 88 is flat like a normal display, so a part of a wide-field image is displayed at a viewpoint specified by the user. The VR goggles 89 have a display function (may further have a camera with a normal angle of view). The display function held by the smart glasses 88 projects a wide-field image with a viewpoint determined by the direction of the user's head, so a part of the wide-field image is displayed at a viewpoint according to the direction of the user's head. While viewing a wide-field image with the smart glasses 88 or the VR goggles 89, the user can transmit an imaging request specifying the viewpoint information being viewed to the information processing system 50.

[0195] <Situations in which camera shake may occur> FIG. 23 is a diagram showing a schematic diagram of a situation in which user A captures images of the surroundings with the imaging device 10 and distributes the images through streaming. In FIG. 23, the imaging location is a construction site, but this is merely an example. User A moves around the construction site while capturing images with the power turned on. The distribution start button 115a described below is turned on, and the imaging device 10 transmits image data during the movement to the communication terminals 30B and 30C. In order for user A to show the contents of the electronic whiteboard 601 to users B and C, he stops in front of the electronic whiteboard 601 and points the imaging device 10 toward the electronic whiteboard 601 (since it is a wide-field image, there should be no obstacle between the electronic whiteboard 601 and the imaging device 10).

[0196] Although user A is still imaging the electronic whiteboard 601, slight vibrations and movements may occur in the imaging device 10. During this time, the imaging device 10 encodes the image frames using a predetermined encoding method such as H.264. A typical encoding method generates a predicted image from an image immediately before or after an image at a certain moment, detects the difference with the actual image at that time, and encodes the image. For this reason, if user A moves a lot, the difference also becomes large, and if high resolution is required for characters, it takes time for the image to stabilize. In addition, in a network environment with a low available bandwidth, the communication speed decreases, so it takes time for the image to become fine enough to distinguish details.

[0197] Users B and C of communication terminals 30B and 30C that received the image data try to read information from the unstable image, but have difficulty reading it until the image becomes stable or high definition. In this embodiment, this phenomenon may be simply called "blurring." In addition, in this embodiment, a technique for suppressing blurring due to encoding or bandwidth is described, but the technique of this embodiment is also suitably applied to suppress so-called camera shake. Camera shake refers to the shaking of the hand holding the camera, which causes the image to become blurred.

[0198] Therefore, in this embodiment, the transmitting side transmits a fixed image frame as described below, thereby suppressing blurring on the receiving side (viewer side). User A can distribute the intended image frame by fixing it without having to keep a fixed posture, and the communication terminals 30B and 30C of users B and C can instantly display the image frame intended by the imager. This improves the user's recognition of the image. Note that an image frame refers to each still image that constitutes a moving image captured by the imaging device 10.

[0199] <Fixing and unlocking the image frame by operating the buttons on the operation unit> Next, a process flow in which the imaging device 10 fixes and releases the image data when the user A operates a button on the operation unit 115 will be described with reference to Fig. 24. Fig. 24 is a sequence diagram illustrating the process of fixing and releasing the image frame in response to a button operation on the operation unit 115, and the imaging device 10 transmitting the image data.

[0200] S101: User A who wishes to distribute image data and sound data presses the distribution start button 115a. User A may be moving or stationary. The reception unit 12 receives the press, and the imaging device 10 starts distributing the image data and sound data. Note that the button operation on the imaging device 10 may be substituted with a voice operation, or if the imaging device 10 has a touch panel, a touch operation on a soft key may be used.

[0201] S102: The reception unit 12 notifies the control unit 21 of the start of distribution. In response to this, the control unit 21 starts distribution of image data and sound data. Note that the sound data is omitted in FIG. 24 for simplicity.

[0202] During distribution, the control unit 21 repeats the following process for each frame. The cases where the fixation flag is ON and OFF are explained separately. The fixation flag is a flag that controls whether or not image data is fixed. When the fixation flag is ON, the communication unit 11 repeatedly transmits the same still image (image frame). When the fixation flag is OFF, the communication unit 11 successively transmits image frames that are repeatedly updated by capturing images.

[0203] S103: The fixed flag is OFF in the initial state immediately after the distribution start button 115a is pressed. The fixed flag may be ON in the initial state immediately after the distribution start button 115a is pressed. The control unit 21 transmits, via the communication unit 11, the coded frames obtained by coding the image frames captured by the imaging processing unit 13 in the coding unit 20. The control unit 21 instructs the imaging processing unit 13 to capture images and acquires the image frames.

[0204] S104: The control unit 21 requests the encoding unit 20 to encode the image frame and acquires the encoded frame. The encoded frame is encoded using a codec method such as H.264, Xvid, Divx, or MPEG-4.

[0205] Next, the case where the fixed flag is OFF will be described.

[0206] S105: If the fixation flag is ON, the control unit 21 acquires the image frame stored in the storage unit 1000.

[0207] S106: The control unit 21 requests the encoding unit 20 to encode the image frame and obtains the encoded frame. The encoding process may be the same as when the fixed flag is OFF.

[0208] S107: In common with whether the fixed flag is ON or OFF, the control unit 21 notifies the communication unit 11 of a transmission instruction together with the encoded frame. When the fixed flag is OFF, the communication unit 11 transmits the encoded frame that is repeatedly generated to the communication terminal 30 in real time. "Real time" means completing the processing within the estimated maximum delay time. When the fixed flag is ON, the control unit 21 notifies the communication unit 11 of a transmission instruction together with the same encoded frame.

[0209] S108: As a common process whether the fixed flag is ON or OFF, the communication unit 11 transmits the encoded frame to the information processing system 50. By performing these processes for each frame, the image data is transmitted.

[0210] When the communication unit 11 transmits the coded frame to the information processing system immediately after the fixed flag is turned ON, it is preferable to forcibly generate and transmit an intra frame. For example, in H.264, an I frame (intra frame), a P frame (predictive frame), and a B frame (bidirectional frame) are transmitted according to the magnitude of change in the image frame. An I frame is a frame that has information on all of the frame. If the image frame transmitted when the fixed flag is turned ON is a P frame or a B frame, it takes time to transmit the entire image frame to the communication terminals 30B and 30C. Therefore, when the communication unit 11 transmits the coded frame to the information processing system immediately after the fixed flag is turned ON, it is possible to transmit the entire image frame to the communication terminals 30B and 30C in a shorter time by forcibly generating and transmitting an intra frame.

[0211] When the fixed flag is ON, the control unit 21 may transmit the image frame stored in the storage unit 1000 only once, rather than repeatedly. In this case, depending on the specifications of the receiving communication terminals 30B, 30C, the last received image frame can be displayed, which has the effect of reducing the load on the imaging device 10. However, in the case of only one transmission, the image frame is not transmitted to users who join the room after transmission while the fixed flag is ON. For this reason, it is preferable for the control unit 21 to repeatedly transmit the image frame stored in the storage unit 1000.

[0212] The process when the user presses the fix button 115b and the release button 115c will be described.

[0213] S109: User A presses the fix button 115b to suppress camera shake. The reception unit 12 receives the pressing of the fix button 115b. As a result, the image frame to be transmitted to the communication terminal 30 is fixed.

[0214] S110: The reception unit 12 notifies the control unit 21 of an instruction to fix the image frame.

[0215] S111: The control unit 21 sets the fixation flag to ON in response to the notification of the fixation instruction.

[0216] S112, S113: In response to the notification of the fixation instruction, the control unit 21 stores the captured image frame in the storage unit 1000 when the fixation flag is ON. The captured image frame may be the last image frame captured at the time when the fixation flag is ON, but it does not necessarily have to be the last image frame. While the fixation flag is ON, the image frame actually captured by the imaging processing unit 13 is transmitted to the communication terminal 30, although it is a still image, so that the users B and C can understand that the imaging device 10 is communicating and the minimum situation of the base where the imaging device 10 is located. However, the image frame transmitted to the communication terminal 30 while the fixation flag is ON may be one that is not easily affected by the characteristics of the video encoding and the available bandwidth of the network. For example, the image frame transmitted to the communication terminal 30 while the fixation flag is ON may be one in which a message is displayed, or a preset animation, etc. A display example is shown in FIG. 28.

[0217] S114: Next, user A assumes that the quality of the image frames has stabilized due to voice instructions from users B and C and the passage of time, and presses the release button 115c. The reception unit 12 accepts the pressing of the release button 115c. As a result, the image frames transmitted to the communication terminals 30B and 30C of users B and C are no longer fixed.

[0218] S115: The reception unit 12 notifies the control unit 21 of an instruction to release the fixation of the image frame.

[0219] S116: In response to the notification of the fixation release instruction, the control unit 21 sets the fixation flag to OFF. The image frame is no longer fixed, and normal image distribution is resumed.

[0220] 24 are functions that the imaging device 10 has. However, as shown in FIG. 1, since the imaging device 10 can be connected to a communication terminal 30A, the functions of the reception unit 12, control unit 21, imaging processing unit 13, encoding unit 20, storage unit 1000, and communication unit 11 may be distributed and held by the imaging device 10 and the communication terminal 30.

[0221] Fig. 25 is a flow chart showing how the imaging device 10 fixes and releases the fixation of an image frame in response to a button operation on the operation unit 115. Note that Fig. 25 is a flow chart showing the sequence diagram of Fig. 24.

[0222] The reception unit 12 receives a button pressed on the operation unit 115 operated by the user A (S201). The reception unit 12 determines the type of the pressed button (S202). If the type of the pressed button is other than the distribution start button 115a, the process returns to step S201.

[0223] If the pressed button is the distribution start button 115a, the control unit 21 judges the state of the fixed flag (S203).

[0224] If the fixation flag is ON, the control unit 21 acquires an image frame stored in the storage unit 1000 (S204).

[0225] If the fixation flag is OFF, the control unit 21 acquires an image frame from the imaging processing unit 13 (S205).

[0226] The control unit 21 encodes the image frame (S206), and the communication unit 11 transmits the encoded frame to the information processing system 50 (S207).

[0227] It is assumed that the reception unit 12 receives a button press on the operation unit 115 from the user A again (S208). The reception unit 12 determines the type of the button pressed on the operation unit 115 (S209).

[0228] If the pressed button is the release button 115c, the control unit 21 turns off the fixation flag (S210).

[0229] If the pressed button is the fixed button 115b, the control unit 21 stores the captured image frame in the storage unit 1000 (S211), and turns on the fixed flag (S212).

[0230] If the pressed button is neither the fix button 115b nor the release button 115c, the process returns to step S203.

[0231] In this way, after distribution is started by pressing a button, if the fixed flag is OFF, the imaging device 10 distributes image data by repeatedly encoding and transmitting successively updated image frames. At that time, if a button on the operation unit 115 is pressed and it is the fixed button, the image frame is saved and the fixed flag is set ON.

[0232] When the fixation flag is ON, the communication unit 11 fixes the image frame by transmitting the image frame stored in the storage unit 1000, instead of the image frame that is updated one after another. At that time, when the fixation release button 115c is pressed, the fixation flag is turned OFF, and the fixation of the image frame is released.

[0233] <<User A's selection of image frames to be sent while the fixed flag is ON>> The image frame stored in the storage unit 1000 by pressing the lock button 115b may already be blurred due to the timing of the encoding process, etc. Since the imaging device 10 transmits image frames with less blur to the communication terminals 30B and 30C, it is preferable that the user A can select the image frame to transmit from multiple image frames.

[0234] Fig. 26 shows an image frame selection screen 610 displayed by the imaging device 10 or the communication terminal 30A connected to the imaging device 10. Three image frames 612 to 614 are stored on the image frame selection screen 610 in Fig. 26. The text in the image frames 612 and 614 is unclear due to blurring. User A taps on any image frame (for example, image frame 613) to select it, and presses the selection button 615. As a result, the selected image frame becomes the image frame to be repeatedly transmitted.

[0235] The image frame selection screen 610 is displayed immediately after the control unit 21 stores several image frames in the storage unit 1000 in step S113. The several image frames may be, for example, a plurality of image frames captured at different times. It is expected that there will be image frames with little blur among the image frames captured at different times.

[0236] <<Unfixing without pressing the unfix button 115c>> There is a possibility that user A may press fix button 115b, but forget or be late in pressing release button 115c, so it is preferable to have a function that automatically turns off the fix flag using a timer.

[0237] Fig. 27 is a flowchart showing how the imaging device 10 fixes and releases the fixation of an image frame in response to a button operation on the operation unit 115 or a timer. The explanation of Fig. 27 will mainly focus on the differences from Fig. 25. The processing up to step S212 may be the same as that of Fig. 25.

[0238] When the control unit 21 turns the fixed flag ON in step S211, the timer management unit 23 starts the timer (S213). If the timer is up (Yes in S214), the control unit 21 turns the fixed flag OFF (S215).

[0239] On the other hand, if neither the fix button 115b nor the release button 115c is pressed, the timer management unit 23 judges whether the timer has timed out (S214). If the timer has not timed out (No in S214), the process returns to step S203.

[0240] By this process, even if user A forgets or is late in pressing the release button 115c, delivery of the image frame can be started.

[0241] <<Image frame displayed by the communication terminal while the fixed flag is ON>> In Figures 24 and 25, it has been explained that while the fixed flag is ON, the communication terminal 30 transmits the image frame stored in the memory unit 1000 repeatedly or only once, but other image frames may also be transmitted.

[0242] Fig. 28 shows an example of a message displayed by the communication terminal 30B while the fixed flag is ON. Fig. 28(a) is an example of a message 611 displayed by the communication terminal 30. The message 611 in Fig. 28(a) is displayed on the image viewing screen 210 displayed by the communication terminal 30B when user B enters the virtual room, as shown in Fig. 15(b).

[0243] The message 611 may be, for example, "Transmission will begin when the image stabilizes. Please wait a moment." Users B and C know that the image data will be delivered soon, so they can wait with peace of mind.

[0244] FIG. 28(b) is an example of an animation displayed by the communication terminal 30. As described above, when the fixation is automatically released after a certain time has passed (time-up), the animation may display the remaining time until the fixation is released. This animation is made up of a number of still images, and shows the gradual countdown of the time until the fixation is released. The control unit 21 switches between the still images in FIG. 28(b), which are prepared in advance, according to the count value of a timer. The imaging device 10 transmits an image frame including the count value of the timer to the communication terminal 30.

[0245] Users B and C know the specific time until the image data is delivered. For example, if the waiting time is 5 seconds, 5 image frames are transmitted the number of times according to the FPS (Frame Per Rate), but since the difference between the image frames is small, the imaging device 10 can transmit image frames with little blur.

[0246] The imaging device 10 may combine the stored image frames, messages, and animations and transmit them to the communication terminals 30B and 30C of users B and C as follows. - Picture frame that contains both a picture frame and a message Image frames that contain both image frames and animations -Image frames and image frames with messages and animations -Image frames that contain both messages and animations

[0247] <Freezing and unlocking image frames via remote commands> Next, referring to FIG. 29, a process flow in which the imaging device 10 fixes and releases image data when the imaging device 10 receives a fixing or release of an image frame from a remote location will be described. FIG. 29 is a sequence diagram for describing a process until the imaging device 10 transmits image data in response to fixing and releasing of an image frame by a remote command. A remote command is a command transmitted from a communication terminal 30 at a location other than the location where the imaging device 10 is located. The remote command may be transmitted from the same location (location A). One of the remote commands is a fixing instruction and a fixing release instruction. Note that the explanation of FIG. 29 will mainly explain the differences from FIG. 24. Steps S121 to S128 in FIG. 29 may be the same as S101 to S108 in FIG. 24.

[0248] S129: User B or user C issues a fixation instruction on the viewer's communication terminal 30B or 30C. The communication unit 11 receives the fixation instruction command. When User B or user C issues an unfixation instruction on the viewer's communication terminal 30B or 30C, the communication unit 11 receives the unfixation command.

[0249] S130: The communication unit 11 notifies the reception unit 12 of the fixation instruction command or the fixation release command.

[0250] S131: When the communication unit 11 receives a fixation instruction command, the reception unit 12 notifies the control unit 21 of an instruction to fix the image frame.

[0251] S132: The control unit 21 sets the fixation flag to ON in response to the notification of the fixation instruction.

[0252] S133 and S134: Furthermore, the control unit 21 stores in the storage unit 1000 the image frame that has been captured when the fixation flag is ON in response to the notification of the fixation instruction.

[0253] S135: When the communication unit 11 receives the fixation release command, the reception unit 12 notifies the control unit 21 of an instruction to release the fixation of the image frame.

[0254] S136: In response to the notification of the release fixation instruction, the control unit 21 sets the fixation flag to OFF. The image frame is no longer fixed, and normal image distribution is resumed.

[0255] In this way, the imaging device 10 accepts a fixation instruction or a fixation release instruction from a remote base, so that it is possible to fix and release the image frame from a remote base. When the image data is blurred, the users B and C at the remote locations can fix and release the image frame.

[0256] It is preferable that the imaging device 10 performs exclusive control so that user C cannot cancel the fixation even if user B has instructed to fix. The control unit 21 stores the identification information of user B or communication terminal 30B who instructed the fixation, and turns the fixation flag OFF only when the fixation is canceled by user B or communication terminal 30B. Furthermore, while the control unit 21 is controlling the fixation flag to ON in response to a fixation instruction from user B, the control unit 21 will not accept a fixation instruction even if it is transmitted from communication terminal 30C operated by user C (for example, the control unit 21 transmits an error message to communication terminal 30C).

[0257] Fig. 30 is a flowchart showing how the imaging device 10 fixes and releases the fixation of an image frame by remote control. Fig. 30 is a flowchart showing the sequence diagram of Fig. 24. The explanation of Fig. 30 will mainly focus on the differences from Fig. 25. Steps S221 to S227 in Fig. 30 may be the same as S201 to S207 in Fig. 25.

[0258] In step S227, the communication unit 11 transmits the encoded frame to the communication terminal 30. Next, the reception unit 12 receives a remote command (fixing instruction or fixing release instruction) from the communication terminal 30 via the communication unit 11 (S228).

[0259] The reception unit 12 determines the type of the received remote command (S229). If the type of the remote command is an unfix command, the control unit 21 turns off the fix flag (S230).

[0260] If the type of remote command is a fixation instruction command, the control unit 21 stores the captured image frame in the storage unit (S231), and turns on the fixation flag (S232).

[0261] If the type of the remote command is neither a fix command nor a fix release command, the process returns to step S223.

[0262] Note that even if the fixed flag is turned ON by a remote command, the fixed flag may be automatically turned OFF by a timer. In this case, the processing is the same as steps S213 to S214 in FIG.

[0263] <Fixing and Unfixing Image Frames with Object Detection> Next, with reference to Fig. 31, a process flow in which the imaging device 10 fixes image data when the imaging device 10 detects a predetermined object will be described. As described in Fig. 23, there are cases where the user A stops moving in order to take a picture of a specific object such as the electronic whiteboard 601. That is, the user A stops when he wants the imaging device 10 to capture an image of the predetermined object. In such a case, if the imaging device 10 detects the predetermined object and automatically turns on the fixation flag, the user A does not need to press the fixation button 115b.

[0264] The object varies depending on what the user A wants to photograph. As described above, at a construction site, it may be an electronic whiteboard 601, but it may also be construction machinery, etc. Furthermore, when the user A photographs an operating room, the object may be a main unit 356 or an internal organ, which will be described later. It is assumed that the objects detected by the imaging device 10 are set in advance.

[0265] Fig. 31 is a sequence diagram for explaining the process up to when the imaging device 10 transmits image data in response to fixing of the image frame due to detection of an object. Note that the explanation of Fig. 31 will mainly explain the differences from Fig. 24. Steps S141 to S148 in Fig. 31 may be the same as S101 to S108 in Fig. 24.

[0266] S149: The control unit 21 requests the object detection unit 22 to detect an object, and obtains the object detection result from the object detection unit 22.

[0267] The object detected here is a predetermined object that has been learned in advance by machine learning. Machine learning is a technology for making a computer acquire a learning ability like a human, and refers to a technology in which a computer autonomously generates an algorithm required for judgment such as data identification from training data that is previously input, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. Any learning method for machine learning is acceptable.

[0268] For example, convolutional neural networks are known as algorithms for object recognition such as the object detection of this embodiment. A person in charge prepares in advance a large amount of image data containing identification information of objects to be detected and objects, and constructs a convolutional neural network. In this way, the object detection unit 22 outputs identification information of objects appearing in the input image frames.

[0269] In addition, when the imaging device 10 is a spherical (omnidirectional) imaging device 10, the angle of view is wide, so that an object may appear in the image frame before the user A approaches the object. In this case, the fixation flag may be turned ON far in front of the object, and the image frame may be fixed. Therefore, it is preferable that the object detection unit 22 determines that an object has been detected when the object occupies a certain number of pixels or more.

[0270] S150: If an object is detected, the control unit 21 notifies the receiving unit 12 of the object detection result including the fact that the object has been detected.

[0271] S151: Since an object has been detected, the reception unit 12 notifies the control unit 21 of an instruction to fix the image frame.

[0272] S152: The control unit 21 sets the fixation flag to ON in response to the notification of the fixation instruction.

[0273] S153, S154: Furthermore, the control unit 21 stores in the storage unit 1000 an image frame that has been captured when the fixation flag is ON in response to detection of an object.

[0274] S155: Next, the user A presses the release button 115c. The reception unit 12 receives the pressing of the release button 115c. As a result, the image frame transmitted to the communication terminal 30 is no longer fixed.

[0275] S156: The reception unit 12 notifies the control unit 21 of the release of the image frame fixation.

[0276] S157: In response to the notification of release of fixation, the control unit 21 sets the fixation flag to OFF. The image frame is no longer fixed, and normal image distribution is resumed.

[0277] In this way, the imaging device 10 accepts a fixation instruction when a specified object is detected, making it possible to fix the image frame without user A pressing the fixation button 115b or users B and C sending a remote command.

[0278] Fig. 32 is a flowchart showing how the imaging device 10 fixes an image frame by detecting an object. Fig. 32 is a flowchart showing the sequence diagram of Fig. 31. The explanation of Fig. 32 will mainly focus on the differences from Fig. 25. Steps S241 to S247 in Fig. 32 may be similar to S201 to S207 in Fig. 25.

[0279] In step S247, the communication unit 11 transmits the encoded frame to the communication terminal 30. Next, the object detection unit 22 attempts to detect a predetermined object (S248).

[0280] The control unit 21 determines whether the object detection unit 22 has detected a predetermined object (S249). If a predetermined object has been detected, the control unit 21 notifies the reception unit 12 of that effect, and in response to a fixation instruction from the reception unit 12, stores the already captured image frame in the storage unit 1000 (S250), and turns on the fixation flag (S251).

[0281] Next, the reception unit 12 receives a button press on the operation unit 115 operated by the user A (S252). The reception unit 12 determines the type of the pressed button (S253). If the type of the pressed button is other than the release button 115c, the process returns to step S243.

[0282] If the pressed button is the release button 115c, the control unit 21 turns off the fixation flag (S254).

[0283] Note that even if the fixation flag is turned ON due to detection of a specific object, if the specific object is not detected for a certain period of time, a timer will run out and the fixation flag will be automatically turned OFF. If the image frame is automatically fixed due to object detection, it is difficult for user A to notice that the image frame has been fixed, so it is particularly effective to have the fixation flag automatically turned OFF by a timer.

[0284] The fixation flag may be automatically turned off when a specific object is detected. If user A photographs the object for a sufficient period of time until the image quality stabilizes, the image quality will often remain stable even after the fixation is released, and the viewer can view an image frame that is not fixed.

[0285] <Freezing image frames by detecting stoppage> Next, referring to Fig. 33, a process flow in which the imaging device 10 fixes image data when it detects that the imaging device 10 has stopped will be described. As described in Fig. 23, there are cases where user A stops to take a picture of a specific object such as the electronic whiteboard 601. In other words, when user A stops, it is possible that he or she wants to reduce blur. If the imaging device 10 automatically turns on the fixation flag by detecting that the imaging device 10 has stopped, user A does not need to press the fixation button 115b. It is assumed that when user A stops, the imaging device 10 also stops, and when the imaging device 10 stops, user A also stops.

[0286] Fig. 33 is a sequence diagram for explaining the process up to when the imaging device 10 transmits image data in response to fixing of the image frame due to detection of stoppage of the imaging device 10. Note that the explanation of Fig. 33 will mainly explain the differences from Fig. 24. Steps S161 to S168 in Fig. 33 may be similar to S101 to S108 in Fig. 24.

[0287] S169: The control unit 21 requests the motion detection unit 24 to detect the motion of the imaging device 10, and obtains the motion detection result from the motion detection unit 24. In this embodiment, since it is desired to detect whether the imaging device 10 is moving or stopped, the motion detection unit 24 monitors the signals of the acceleration sensor 120 and the gyro sensor 119 and judges whether the imaging device 10 is stopped. For example, since the acceleration can be converted into a speed by integrating the acceleration, the motion detection unit 24 integrates the acceleration and judges whether the speed has become almost zero. The gyro sensor 119 may be the same as the acceleration sensor 120. In addition, the motion detection unit 24 may judge whether the imaging device 10 is moving or stopped by a model that machine-learns the correspondence between the signals of the acceleration sensor 120 and the gyro sensor 119 and the state of moving or stopping.

[0288] S170: If a stop is detected, the control unit 21 notifies the receiving unit 12 of the motion detection result including the fact that a stop has been detected.

[0289] S171: Since the imaging device 10 has stopped, the reception unit 12 notifies the control unit 21 of an instruction to fix the image frame.

[0290] S172: The control unit 21 sets the fixation flag to ON in response to the notification of the fixation instruction.

[0291] S173, S174: Furthermore, in response to detection of the stoppage, the control unit 21 stores in the storage unit the image frame that has been captured when the fixation flag is ON.

[0292] S175: Next, the user A presses the release button 115c. The reception unit 12 receives the pressing of the release button 115c. As a result, the image frame transmitted to the communication terminal 30 is no longer fixed.

[0293] S176: The reception unit 12 notifies the control unit 21 of the release of the image frame fixation.

[0294] S177: In response to the notification of release of fixation, the control unit 21 sets the fixation flag to OFF. The image frame is no longer fixed, and normal image distribution is resumed.

[0295] In this way, since the imaging device 10 accepts a fixation instruction when it detects that its own device has stopped, it becomes possible to fix the image frame without user A having to press the fixation button 115b or users B and C having to send a remote command.

[0296] Fig. 34 is a flowchart showing how the imaging device 10 fixes the image frame upon detection of stoppage. Fig. 34 is a flowchart showing the sequence diagram of Fig. 33. The explanation of Fig. 34 will mainly focus on the differences from Fig. 25. Steps S261 to S267 in Fig. 34 may be similar to S201 to S207 in Fig. 25.

[0297] In step S267, the communication unit 11 transmits the encoded frame to the communication terminal 30. Next, the motion detection unit 24 detects whether the imaging device 10 is moving or stationary (S268).

[0298] The control unit 21 determines whether or not the motion detection unit 24 has detected a stop of the imaging device 10 (S269). If a stop is detected, the control unit 21 notifies the reception unit 12, stores the image frames that have already been captured in response to a fixation instruction from the reception unit 12 in the storage unit 1000 (S270), and turns on the fixation flag (S271).

[0299] Next, the reception unit 12 receives a button pressed on the operation unit 115 operated by the user A (S273). The reception unit 12 determines the type of the pressed button (S274).

[0300] If the pressed button is the release button 115c, the control unit 21 turns off the fixation flag (S275).

[0301] If the pressed button is other than the release button 115c (No in S274), the process returns to step S263.

[0302] Therefore, when user A stops the imaging device 10 (when user A stops), the image frame can be automatically fixed.

[0303] Note that even if the fixation flag is turned ON due to user A stopping the imaging device 10, the fixation flag may be automatically turned OFF when the timer times out. If the image frame is automatically fixed due to the stopping of the imaging device 10, user A will not easily notice this, so it is particularly effective to have the fixation flag automatically turned OFF by the timer.

[0304] Furthermore, when user A resumes movement, the fixation flag may be automatically turned OFF. Even if user A does not press the release fixation button 115c, the fixation flag can be turned OFF by user A resuming movement.

[0305] Fig. 35 is a flow chart showing a case where the imaging device 10 fixes the image frame upon detection of the stop of the imaging device 10, and the fixation flag is turned OFF in response to detection of the timer or the start of movement. In the explanation of Fig. 35, differences from Fig. 34 will be mainly explained.

[0306] In step S271, when the control unit 21 turns on the fixed flag, the control unit 21 starts a timer (S272).

[0307] After that, if it is determined in step S274 that the type of the pressed button is other than the release button 115c, the timer management unit 23 determines whether the timer has timed out (S276). If the timer has timed out, the timer management unit 23 turns off the fixation flag (S275).

[0308] If the timer has not timed out, the control unit 21 determines whether or not the resumption of movement has been detected (S277). If the resumption of movement has been detected (Yes in S277), the control unit 21 turns off the fixation flag (S275).

[0309] If no motion is detected (No in S277), the process returns to step S263. In this way, the imaging device 10 can automatically fix the image frame when it detects that its own device has stopped, and can automatically release the fixation of the image frame when it detects that its own device has resumed movement or when the timer expires.

[0310] <Application of communication systems in remote medical care> An example of application of a communication system in remote medical care will be described below. FIG. 36 is a diagram for explaining an example of remote communication in which the communication system 1 is applied to remote medical care. In the explanation of FIG. 36, the difference from FIG. 1 will be mainly explained. The site A in FIG. 36 is an operating room, but the process flow of (1) to (6) may be the same as that in FIG. 1. In FIG. 36, a patient is placed on an operating table 355 and undergoes surgery by a medical professional such as a doctor. The medical professional (corresponding to a user) operates on the patient using various surgical tools 354 such as forceps and a scalpel. In addition, the medical professional can wear smart glasses 88 and can transmit an image of the medical professional's surgical field to the communication network N. In addition, various cameras such as an operating room camera 351, an operating field camera 352, and an endoscope 353 are arranged in the operating room. All the cameras in the operating room and the smart glasses 88 are associated with the virtual room.

[0311] In the operating room, a main unit 356 is disposed to monitor the vital signs of the patient and the operating state of the medical equipment. The main unit 356 corresponds to the communication terminal 30 of this embodiment. The communication terminal 30 (main unit 356) in the operating room may also have a function of receiving images from the endoscope 353 and the surgical field camera 352 in addition to the functions shown in FIG. 1. The communication terminal 30 can display the received images on the display 306, and can transmit the images to the information processing system 50 as images of the base of the communication terminal 30. The operation panel 357 is an input interface that accepts various operations, and may allow medical personnel to operate the equipment in the operating room via the operation panel 357. In addition, the endoscope 353, the surgical field camera 351, and the surgical field camera 352 may communicate with the information processing system 50 directly without going through the communication terminal 30.

[0312] The communication terminal 30 may also have a function of an electronic medical record system, or may have a function of communicating with the electronic medical record system. The communication terminal 30 may display electronic medical record information on the display 306.

[0313] 37 is a diagram showing an example of a virtual room association screen (part 1) 360 for associating the image capture device 10 with a virtual room in the case of remote medical care. In the explanation of FIG. 37, differences from FIG. 17 will be mainly explained.

[0314] In the case of remote medical care, a list of virtual rooms 361 associated with, for example, surgeries or examinations performed remotely is displayed on the virtual room association screen (part 1) 360. Medical cameras 362 including omnidirectional cameras are associated with the virtual rooms. The medical cameras 362 include endoscopes, surgical field cameras used to capture images of the surgical field in the operating room, cameras that capture microscopic images, and the like.

[0315] <Major Effects> As described above, the imaging device 10 of this embodiment transmits a fixed image frame on the transmitting side, thereby suppressing blurring on the receiving side (viewer side). User A on the transmitting side can deliver a fixed image frame as intended without having to keep a fixed posture, and users B and C on the receiving side can instantly display the image frame as intended by the imager. This improves the user's recognition of the image.

[0316] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0317] For example, the configuration example in Fig. 13 and the like is divided according to main functions in order to facilitate understanding of the processing by the information processing system 50, the imaging device 10, and the communication terminal 30. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the information processing system 50, the imaging device 10, and the communication terminal 30 can be divided into more processing units according to the processing contents. Also, one processing unit can be divided so as to include more processes.

[0318] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0319] Additionally, the devices described in the examples are merely one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the information processing system 50 includes a plurality of computing devices, such as a server cluster. The computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processes disclosed herein.

[0320] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figures 25, 26, 32, and 33, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Also, the information processing system 50 may be integrated into one server device, or may be divided into multiple devices.

[0321] <Additional Notes> [Claim 1] an imaging processing unit that performs imaging processing to repeatedly generate image frames that constitute a moving image captured by an imaging device; a communication unit that transmits the image frames repeatedly generated by the imaging processing unit to a communication terminal at another base; a reception unit that receives an instruction to fix the image frame transmitted to the communication terminal, When the reception unit receives the fixation instruction, the communication unit repeatedly transmits the same image frame to the communication terminal. [Appendix 2] 2. The imaging device according to claim 1, wherein when the reception unit receives a fixation release instruction to release the fixation of the image frame, the communication unit transmits the image frame repeatedly generated by the imaging processing unit to the communication terminal. [Appendix 3] a fixation button for receiving the fixation instruction, 2. The imaging device according to claim 1, wherein the reception unit receives the fixation instruction by pressing the fixation button. [Appendix 4] a release button for receiving the release instruction, 3. The imaging device according to claim 2, wherein the reception unit receives the release instruction by pressing the release button. [Appendix 5] The imaging device according to claim 1, wherein the reception unit receives the fixing instruction by a remote command transmitted from the communication terminal. [Appendix 6] 3. The imaging device according to claim 2, wherein the reception unit receives the fixation release instruction by a remote command transmitted from the communication terminal. [Appendix 7] an object detection unit that detects a predetermined object from the image frame generated by the imaging processing unit; 2. The imaging device according to claim 1, wherein the reception unit receives the fixation instruction when the object detection unit detects the predetermined object. [Appendix 8] The imaging device according to claim 7, wherein the specified object is an electronic whiteboard. [Appendix 9] a motion detection unit that detects whether the imaging device is moving or stopped; 2. The imaging device according to claim 1, wherein the reception unit receives the fixation instruction when the motion detection unit detects that the imaging device has stopped. [Appendix 10] the reception unit receives a fixation release instruction to release the fixation of the image frame when the motion detection unit detects that the imaging device has started to move; 10. The imaging device according to claim 9, wherein the communication unit transmits the image frames repeatedly generated by the imaging processing unit to the communication terminal. [Appendix 11] a timer management unit that starts a timer when the reception unit receives the fixing instruction, The imaging device according to any one of claims 1 to 10, wherein, when the timer times out, the communication unit starts transmitting the image frames repeatedly generated by the imaging processing unit to the communication terminal. [Appendix 12] a control unit that stores the image frame generated by the imaging processing unit when the reception unit receives the fixation instruction, 12. The imaging device according to any one of claims 1 to 11, wherein when the reception unit receives the fixation instruction, the communication unit repeatedly transmits the image frame stored by the control unit to the communication terminal. [Appendix 13] 2. The imaging device according to claim 1, wherein when the reception unit receives the fixing instruction, the communication unit transmits the image frame having a preset message to the communication terminal. [Appendix 14] 12. The imaging device according to claim 11, wherein when the reception unit receives the fixation instruction, the communication unit transmits to the communication terminal the image frame having a count value of the timer that counts the time until transmission of the image frame repeatedly generated by the imaging processing unit is started. [Appendix 15] 13. The imaging device according to claim 12, wherein immediately after the reception unit receives the fixation instruction, the communication unit transmits an intra-frame of the image frame generated by the imaging processing unit to the communication terminal. [Appendix 16] When the reception unit receives the fixation instruction, the control unit stores a plurality of image frames generated by the imaging processing unit, a display control unit that displays the plurality of image frames; 13. The imaging device according to claim 12, wherein the communication unit repeatedly transmits the image frame selected by the user from among the plurality of image frames to the communication terminal. [Explanation of symbols]

[0322] 10. Imaging device 30 Communication terminals 50 Information Processing Systems [Prior art documents] [Patent documents]

[0323] [Patent Document 1] JP 2001-024934 A

Claims

1. an imaging processing unit that performs imaging processing to repeatedly generate image frames that constitute a moving image captured by an imaging device; a communication unit that transmits the image frames repeatedly generated by the imaging processing unit to a communication terminal at another base; a reception unit that receives an instruction to fix the image frame transmitted to the communication terminal, When the reception unit receives the fixation instruction, the communication unit repeatedly transmits the same image frame to the communication terminal.

2. The imaging device according to claim 1 , wherein when the reception unit receives a fixation release instruction to release the fixation of the image frame, the communication unit transmits the image frame repeatedly generated by the imaging processing unit to the communication terminal.

3. a fixation button for receiving the fixation instruction, The imaging device according to claim 1 , wherein the reception unit receives the fixation instruction by pressing the fixation button.

4. a release button for receiving the release instruction, The imaging device according to claim 2 , wherein the reception unit receives the release instruction by pressing the release button.

5. The imaging device according to claim 1 , wherein the reception unit receives the fixing instruction by a remote command transmitted from the communication terminal.

6. The imaging device according to claim 2 , wherein the reception unit receives the fixation release instruction by a remote command transmitted from the communication terminal.

7. an object detection unit that detects a predetermined object from the image frame generated by the imaging processing unit; The imaging device according to claim 1 , wherein the reception unit receives the fixation instruction when the object detection unit detects the predetermined object.

8. 8. The imaging device according to claim 7, wherein the predetermined object is an electronic whiteboard.

9. a motion detection unit that detects whether the imaging device is moving or stopped; The imaging device according to claim 1 , wherein the reception section receives the fixation instruction when the motion detection section detects that the imaging device has stopped.

10. the reception unit receives a fixation release instruction to release the fixation of the image frame when the motion detection unit detects that the imaging device has started to move; The imaging device according to claim 9 , wherein the communication section transmits the image frames repeatedly generated by the imaging processing section to the communication terminal.

11. a timer management unit that starts a timer when the reception unit receives the fixing instruction, The imaging device according to any one of claims 1 to 10, wherein when the timer times out, the communication unit starts transmitting the image frames repeatedly generated by the imaging processing unit to the communication terminal.

12. a control unit that stores the image frame generated by the imaging processing unit when the reception unit receives the fixation instruction, The imaging device according to claim 1 , wherein when the reception unit receives the fixation instruction, the communication unit repeatedly transmits the image frame stored by the control unit to the communication terminal.

13. The imaging device according to claim 1 , wherein when the reception unit receives the fixing instruction, the communication unit transmits the image frame having a preset message to the communication terminal.

14. The imaging device of claim 11, wherein when the reception unit receives the fixation instruction, the communication unit transmits to the communication terminal the image frame having a count value of the timer that counts the time until transmission of the image frame repeatedly generated by the imaging processing unit begins.

15. The imaging device according to claim 12 , wherein the communication section transmits an intra-frame of the image frame generated by the imaging processing section to the communication terminal immediately after the reception section receives the fixation instruction.

16. When the reception unit receives the fixation instruction, the control unit stores a plurality of image frames generated by the imaging processing unit, a display control unit that displays the plurality of image frames; The imaging device according to claim 12 , wherein the communication unit repeatedly transmits the image frame selected by the user from among the plurality of image frames to the communication terminal.

17. An imaging process for repeatedly generating image frames constituting a moving image captured by an imaging device; A process of transmitting the image frames repeatedly generated to a communication terminal at another location; A process of receiving an instruction to fix the image frame transmitted to the communication terminal; a process of repeatedly transmitting the same image frame to the communication terminal when the fixation instruction is received; How to send.

18. an imaging processing unit that performs imaging processing to repeatedly generate image frames that constitute a moving image captured by an imaging device; a communication unit that transmits the image frames repeatedly generated by the imaging processing unit to a communication terminal at another base; a reception unit that receives an instruction to fix the image frame transmitted to the communication terminal; The communication unit is a program for repeatedly transmitting the same image frame to the communication terminal when the reception unit receives the fixation instruction.

Images