Communication terminal, display method, program, and communication management system

Abstract

To provide a technique for enabling a user to instruct processing of a video through chat.SOLUTION: There is provided a communication terminal capable of communicating with a communication management system having a chat function and receiving video data from an imaging device via a network. The communication terminal includes: a chat input reception unit configured to receive an input of a message for chat; a communication unit configured to transmit the message to the communication management system and receive information on the video data based on the message and the video data from the communication management system; and a video display unit configured to display the video data processed by using the information on the video data.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present invention relates to a communication terminal, a display method, a program, and a communication management system. [Background technology]

[0002] A communication system is known in which at least images and audio are transmitted in real time from one location to one or more other locations, enabling remote communication using images and audio between users in remote locations. Wide-field-of-view images with a wide viewing angle are known, including a 360-degree image (also called a celestial sphere image, omnidirectional image, or panoramic image) that captures the entire 360-degree surroundings as an imaging range that includes areas that cannot be seen with a normal angle of view. A user can view a portion of the wide-field-of-view image from any virtual viewpoint by operating a communication terminal to change the virtual viewpoint of a portion of the wide-field-of-view image displayed on the display screen of the communication terminal.

[0003] Another well-known feature is a chat function in which a chatbot automatically responds to messages, including user questions. A chatbot is software that communicates via text or voice instead of humans. Chatbots are widely used for user support and other purposes.

[0004] Regarding chatbots, a technology is known that displays information related to a user's question on VR goggles (see, for example, Patent Document 1). Patent Document 1 discloses a technology that displays information related to a user's question superimposed on the display of the VR goggles. Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technology, it was difficult for users to instruct video editing via chat. For example, when editing image data transmitted via remote communication, the user had to perform additional tasks separate from the remote communication, making it difficult to concentrate on the meeting.

[0006] In view of the above-mentioned problems, the present invention provides a technique that allows a user to instruct video processing via chat. [Means for solving the problem]

[0007] In view of the above problems, the present invention provides a communication terminal that has a chat function and can communicate via a network with a communication management system that receives video data from an imaging device, and includes a chat input acceptance unit that accepts input of chat messages, a communication unit that transmits the messages to the communication management system and receives information about the video data based on the messages and the video data from the communication management system, and a video display unit that displays the video data that has been processed using the information about the video data. [Effects of the Invention]

[0008] The present invention can provide a technique that allows a user to instruct video editing via chat. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 10 is a diagram illustrating an example of remote communication using a wide-field image. [Figure 2] FIG. 1 is a schematic diagram illustrating an example of a configuration of a communication system. [Figure 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus. [Figure 4] FIG. 1 is a diagram illustrating an example of a hardware configuration of a communication terminal and an information processing system. [Figure 5] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. 1C is a plan view of the imaging device. [Figure 6] FIG. 1 is a conceptual diagram of the imaging device in use. [Figure 7] (a) is a hemispherical image (front) captured by an imaging device, (b) is a hemispherical image (back) captured by an imaging device, and (c) is an image represented by equirectangular projection. [Figure 8] (a) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (b) is a diagram showing a spherical image. [Figure 9] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. [Figure 10] 6(a) is a three-dimensional perspective view of FIG. 5, and FIG. 6(b) is a diagram showing a state in which an image of a predetermined area is displayed on a display. [Figure 11] 10 is a diagram showing the relationship between predetermined area information and an image of a predetermined area T. FIG. [Figure 12] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 13] FIG. 1 is a diagram illustrating an example of a functional configuration of a communication system. [Figure 14] 10 is a conceptual diagram showing image management information stored in an image management information storage unit. FIG. [Figure 15] FIG. 2 is a conceptual diagram showing virtual room information stored in a virtual room information storage unit. [Figure 16] 3 is a diagram for explaining in detail the processes performed by a chat question analysis unit, a video analysis unit, and an audio analysis unit. FIG. [Figure 17] 1A is a diagram showing an example of an entry screen, and FIG. 1B is a diagram showing an example of an image viewing screen displayed by a communication terminal when a user enters a virtual room. [Figure 18] 10 is an example of a sequence diagram illustrating a process in which a user (or a communication terminal) enters a virtual room. [Figure 19] FIG. 10 is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room. [Figure 20] FIG. 10 is a diagram showing an example of a virtual room association screen (part 2). [Figure 21]FIG. 10 is a diagram showing an example of a virtual room association screen (part 3). [Figure 22] FIG. 10 is a diagram illustrating an example of a wide-field image transmission start / stop dialogue displayed by the communication terminal. [Figure 23] FIG. 10 is a sequence diagram illustrating an example of a procedure in which a user registers an imaging device in a virtual room. [Figure 24] FIG. 10 is an example of a sequence diagram illustrating a flow in which a wide-field image is shared. [Figure 25] 10A and 10B are diagrams illustrating a process of masking a person using a chat screen displayed by a communication terminal. [Figure 26] 10A and 10B are diagrams illustrating a process of adjusting the viewpoint to a person using a chat screen displayed by a communication terminal. [Figure 27] 10A and 10B are diagrams illustrating a process of displaying materials using a chat screen displayed by a communication terminal. [Figure 28] 10A and 10B are diagrams illustrating a process of sounding an alarm using a chat screen displayed on a communication terminal. [Figure 29] 10A and 10B are diagrams illustrating a process of removing noise from voice data using a chat screen displayed by a communication terminal. [Figure 30] FIG. 10 is a diagram illustrating an example of a control mode. [Figure 31] FIG. 10 is a diagram illustrating an example of control data in JSON format. [Figure 32] FIG. 10 is a diagram illustrating an example of person coordinates when the control data is person masking. [Figure 33] 10 is a flowchart illustrating an example of a process in which a communication management system sets a control mode and control data in response to a message from a communication terminal. FIG. [Figure 34] FIG. 10 is a flowchart illustrating an example of a process flow in which the communication management system creates control data. [Figure 35] FIG. 10 is a flowchart illustrating an example of a process in which a system administrator adds or deletes a video analysis process or a voice analysis process to or from the communication management system. [Figure 36]FIG. 10 is a flowchart illustrating an example of a process executed by a communication terminal. [Figure 37] FIG. 1 is a diagram illustrating an example of remote communication in which the communication system is applied to telemedicine. [Figure 38] FIG. 10 is a diagram showing an example of a virtual room association screen (part 1) for associating an imaging device with a virtual room in the case of remote medical care. DETAILED DESCRIPTION OF THE INVENTION

[0010] An information processing system and a display method performed in the information processing system will be described below as an example of an embodiment of the present invention.

[0011] <An example of remote communication> FIG. 1 is a diagram illustrating an example of remote communication using wide-field images. In FIG. 1, three locations A to C are communicating via a communication management system 50. The number of locations is merely an example, and the number may be two, four, or more. Remote communication refers to communicating with a person who is physically distant through images and audio by utilizing IT tools.

[0012] As an example, site A is a construction site. Sites B and C can be anywhere, such as an office, as long as they are locations where wide-field images can be transmitted. Site A is provided with an imaging device 10 that can capture wide-angle images, such as omnidirectional images, or images with a horizontal angle of 180 to 360 degrees. Hereinafter, such wide-angle images will be simply referred to as wide-field images. Sites A to C are provided with various communication terminals 30A to 30C that can view wide-field images. Hereinafter, any of communication terminals 30A to 30C will be referred to as "communication terminal 30."

[0013] At a construction site, various construction works are being carried out by workers at each location, and while the entire construction site is captured using wide-field images, users at each of bases A to C can freely change the viewpoint to monitor any construction or work that they want to focus on. The viewpoint is the center position or range of the entire wide-field image that is displayed on the display of communication terminals 30A to 30C.

[0014] The imaging device 10 is attached to a tripod 86 or to an arm 85 via a gimbal 87. A relay device (in FIG. 1, the communication terminal 30A also serves as the relay device) is installed at the construction site, and the communication terminal 30A receives wide-field-of-view images from the imaging device 10 via a wired or wireless connection and transmits them to the communication management system 50. The communication terminal 30A can also serve as a terminal for viewing wide-field-of-view images. An imaging device 9 is connected to the communication terminal 30A (or may be built-in), and images with a normal angle of view (which may be capable of capturing omnidirectional images) captured by the imaging device 9 can also be transmitted to the communication management system 50. In addition, a user A (who may be a worker, for example) can wear smart glasses 88, and images with a normal angle of view (which may be capable of capturing omnidirectional images) captured by the smart glasses 88 may be transmitted to the communication management system 50. The smart glasses 88 are an information terminal that displays information acquired via the Internet on a display while maintaining a field of view. The smart glasses 88 may be installed at any base.

[0015] At site B, a PC (Personal Computer), a smartphone, or the like is placed as an example of a communication terminal 30B. The communication terminal 30B may be any device that can communicate with the communication management system 50 via the communication network N, and may also be a display device such as a tablet terminal, a PDA (Personal Digital Assistant), an electronic whiteboard, or a projector. A camera may be built into or connected to the communication terminal 30B.

[0016] At the site C, a PC, a smartphone, a VR goggles 89, or the like is placed as an example of a communication terminal 30C, and in FIG. 1 , an imaging device 8 is built into or connected to the communication terminal 30C. The VR goggles 89 are information terminals that display an artificial world or a celestial sphere image on a computer in accordance with the direction of movement of the neck or body. The imaging device 8 may be for a wide angle or a normal angle of view. Furthermore, the communication terminal 30C may be any device that can communicate with the communication management system 50 via the communication network N, and may be a display device such as a tablet terminal, a PDA, an electronic whiteboard, or a projector. The VR goggles 89 may be placed at any site.

[0017] In this embodiment, the imaging device 10 and the communication terminal 30 are managed as a communication group called a virtual room. The imaging device 10 is associated with the virtual room, and the communication terminal 30 (the user operating the communication terminal 30) can receive the wide-field image transmitted by the imaging device 10 and view the wide-field image by entering the virtual room. Smart glasses 88 and VR goggles 89 can also be associated with the virtual room. The imaging devices 8 and 9 enter the virtual room together with the communication terminal 30.

[0018] Users A to C at locations A to C can arbitrarily change the viewpoint of the wide-field image. As a result, users A to C viewing the wide-field image in real time may be viewing different viewpoints, which may make communication difficult. Therefore, in this embodiment, the viewpoint of the communication terminal 30 at location A is shared with other communication terminals 30 at locations B and C. An overview of viewpoint sharing will be explained below. For the sake of explanation, it is assumed below that the viewpoint of the user at location B is shared by users at locations A and C.

[0019] (1) Communication terminals 30A to 30C share a wide-field-of-view image (an example of a first wide-field-of-view image) captured by imaging device 10. When user B requests imaging of a wide-field-of-view image while viewing the image from an arbitrary viewpoint on communication terminal 30B, communication terminal 30B (an example of a first communication terminal) transmits viewpoint information and an imaging request to communication management system 50.

[0020] (2) In response to the imaging request, the communication management system 50 specifies viewpoint information and requests the imaging device 10 to capture an image (either a still image or a moving image).

[0021] (3) The imaging device 10 captures an image in response to an imaging request, and saves the wide-field image (an example of a second wide-field image) and viewpoint information in a URL (an example of destination information; in FIG. 1, the imaging device 10 is in storage 90) notified by the communication management system 50. The wide-field image saved in storage 90 can be downloaded and displayed by any communication terminal 30.

[0022] (4) The communication management system 50 transmits the URL to the communication terminal 30B.

[0023] (5) Furthermore, the communication management system 50 automatically or in response to a request from user B transmits the URL to the communication terminals 30A and 30C (an example of a second communication terminal) that are currently in the same virtual room.

[0024] (6) The communication terminals 30A and 30C connect to the URL to receive the viewpoint information and the wide-field image, and then display the wide-field image identified by the viewpoint information by setting the viewpoint of the image to coincide with the center of the image field. Note that the viewpoint does not need to be perfectly centered, and the viewpoint may be set to be within a range near the center of the image field.

[0025] The same applies when the viewpoint of a user at site A is shared by users at sites B and C, or when the viewpoint of a user at site C is shared by users at sites A and B.

[0026] As described above, in the communication system 1 of this embodiment, viewpoint information is shared among the locations even when a wide-field-of-view image is distributed, facilitating communication between users.

[0027] In (3), the imaging device 10 can transmit the wide-field image itself to the communication management system 50, and in (4), the communication management system 50 can transmit the wide-field image to the communication terminals 30A to 30C.

[0028] While FIG. 1 illustrates an example in which the imaging device 10 is installed at a construction site, this embodiment can also be applied to VR education, event distribution, remote customer service, remote medical care, and the like. In VR education, the imaging device 10 is installed at an on-site base such as a laboratory, allowing students to freely change their viewpoint from a remote base and view the blackboard, equipment, samples, experiment results, and the like. In event distribution, the imaging device 10 is installed at the on-site event venue, allowing event participants such as spectators to freely change their viewpoint online from a remote base and view the venue. The venue includes footage of subjects such as performers, contestants, presenters, products and exhibits explained at the event, footage of materials explained at the event, and footage of the venue's condition. The event venue may be indoors or outdoors, including venues for sports, concerts, plays, and the like. In remote customer service, for example, when applied to travel agency customer service, the imaging device 10 is installed at the travel destination, allowing customers to freely change their viewpoint from a remote base and plan their itinerary. In telemedicine, the imaging device 10 is placed in a medical setting such as an operating room, and doctors, students, medical equipment personnel, etc. can change their viewpoint at will from a remote location and view the actions of doctors and nurses performing medical procedures at the medical setting, the placement of instruments, the patient's condition, vital signs, etc.

[0029] The locations where images are captured are not limited to these, but can also be schools, factories, warehouses, construction sites, server rooms, or stores, or any other space where a user (viewer) at the viewing location needs to understand the situation at a remote location.

[0030] <Terminology> A tenant is a company, organization, or the like that has entered into a contract to receive an image distribution service from a service provider (an information processing system in this embodiment). As an example, a user belongs to a tenant, but an individual user may also subscribe to the service. In addition to users, imaging devices, virtual rooms, and the like are registered in a tenant.

[0031] A base refers to a location that is the base of activities. In this embodiment, a conference room is used as an example of a base. A conference room is a room that is set up primarily for use in meetings. A conference is also called a get-together, a meeting, a consultation, a gathering, a get-together, a get-together, or the like.

[0032] The device refers to a device other than a general-purpose communication terminal 30 such as a PC or a smartphone, and is an imaging device or a device for viewing wide-field images. In this embodiment, examples of the device include the imaging device 10, smart glasses 88, and VR goggles 89.

[0033] The viewpoint information is parameter information that specifies which specific area of ​​the wide-field-of-view image to display on the display. In this embodiment, the radius vector, polar angle, and azimuth angle corresponding to the center of the wide-field-of-view image displayed on the display are described as examples of viewpoint information, but the viewpoint information may be specified by other parameter information such as the coordinates of the diagonal vertices.

[0034] A wide-field-of-view image is an image with a wide field of view captured over a wide imaging range, including a 360-degree image (also called a spherical image, omnidirectional image, or panoramic image) capturing the entire 360-degree surroundings.

[0035] An image with a normal angle of view is an image that is not a wide-field image, but in this embodiment, it will be described as an image that is not a wide-field image (planar image).

[0036] A communication group is a group of users who share (distribute) wide-field images. In a normal space, a communication group is described as a virtual room, in the sense that wide-field images can be shared by each user when they are in the same room. "Virtual" means that it is realized by information processing via a network.

[0037] Users at each location communicate remotely with each other. Remote communication means communicating with people who are physically far away through images and audio using IT tools. A meeting is when people get together for consultations, discussions, etc. Meetings can take various forms, such as customer service, conferences, rallies, meetings, study sessions, classes, seminars, and presentations. Communication does not necessarily have to be two-way. Therefore, a virtual room can also be called a virtual conference room.

[0038] VR education is an educational tool that uses virtual reality (VR), which creates an unrealistic space in front of the eyes and immerses the viewer in a realistic way, to attract children's attention and deepen their understanding.Remote customer service refers to customer service in which staff in remote locations use video conferencing to introduce products to users via tablets or displays.

[0039] An image is a visual representation of an event. A video is an image projected onto a display device, and can be a sub-concept of an image.

[0040] Information related to video data is information indicating how to process the video data. Information related to audio data is information indicating how to process the audio data. Information related to audio output is an instruction to cause the communication terminal to output audio. In this embodiment, these will be described using the term control data.

[0041] <Example of communication system configuration> FIG. 2 is an example of a schematic diagram of the configuration of communication system 1. Communication system 1 is a system for bidirectionally transmitting and receiving wide-field images and normal-angle images captured by imaging device 10 between multiple locations. This system allows users at other locations to view images distributed from one location by displaying the images at the other locations. Note that as an example of a wide-field image, a spherical image captured by imaging device 10 is distributed. Communication system 1, for example, allows a wide-field image captured at a specific location to be viewed at another location in a remote location.

[0042] As shown in FIG. 2, in the communication system 1, an imaging device 10, a communication terminal 30A, a communication management system 50, and communication terminals 30B and 30C located at each of multiple locations (locations B and C) are communicatively connected to each other.

[0043] If the imaging device 10 can be directly connected to the communication network N, the communication terminal 30A is not required as a relay device (e.g., a router). In this case, the communication terminal 30A is connected to the communication network N without the imaging device 10. However, if the communication terminal 30A is located at the base A, the communication terminal 30A also functions as a relay device, and user A can view the wide-field image in the same way as communication terminals 30B and 30C. Note that an imaging device 10 may be located at a base other than the base A, or multiple imaging devices 10 may be located at the base A.

[0044] Each communication terminal 30 and the communication management system 50 can communicate via a communication network N. The communication network N is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), etc. Note that the communication network N may include not only wired communication networks but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0045] As will be described later, imaging device 10 is a special digital camera capable of capturing a celestial sphere image by capturing an image of a subject, scenery, or the like and obtaining two original hemispherical images. The wide-field image obtained by imaging device 10 may be a moving image, a still image, or both a moving image and a still image. The captured image may also be a video including audio along with the image.

[0046] The communication terminal 30 is a computer such as a PC used by users at each location. The communication terminal 30 displays images captured at the local location, wide-field images (still images or videos) distributed from other locations, and images with a normal angle of view. The communication terminal 30 acquires wide-field images captured by the imaging device 10, for example, via a communication network N. Software for performing image processing, such as OpenGL ES, is installed on the communication terminal 30, enabling image display based on viewpoint information that identifies a partial region of the wide-field image. Note that OpenGL ES is an example of software for performing image processing, and other software may also be used. Even if the communication terminal 30 does not have software for performing image processing installed, it may perform image processing using software received from an external source, or may display images by receiving the results of image processing performed by external software. In other words, the communication terminal 30 is capable of displaying a predetermined partial region of the wide-field image.

[0047] Communication terminal 30 can arbitrarily change the viewpoint relative to the display range of the wide-field image in response to user operation. Communication terminal 30 can change and display the field of view range (predetermined area) based on viewpoint information corresponding to the viewpoint after the movement by moving the virtual viewpoint in response to user operation input (including key input, dragging, scrolling, etc.) on a touch panel, directional buttons, mouse, keyboard, touchpad, etc. Furthermore, if communication terminal 30 is a communication terminal worn by a user, such as VR goggles, it may detect posture information of communication terminal 30 changed in response to changes in the movement of the wearing user, and move the virtual viewpoint in response to the detected posture information, thereby changing and displaying the field of view range (predetermined area) based on viewpoint information corresponding to the viewpoint after the movement.

[0048] The communication terminal 30A distributes wide-field images acquired from the imaging device 10 via a wired cable such as a USB (Universal Serial Bus) cable to communication terminals 30 at other locations via the communication management system 50. The connection between the imaging device 10 and the communication terminal 30A may be a wireless connection using short-range wireless communication or the like, rather than a wired connection using a wired cable. Multiple communication terminals 30A may be installed at location A.

[0049] A user at site A may wear smart glasses 88, and the smart glasses 88 may connect to the communication network N. In this case, images captured by the smart glasses 88 are transmitted to the communication management system 50 via the communication network N, and the communication management system 50 can distribute the images to the communication terminals 30 at each site.

[0050] Communication terminal 30B is located at location B where user B is present, and communication terminal 30C is located at location C where user C is present. A plurality of communication terminals 30B and 30C may be located at locations B and C. Note that communication terminal 30B and communication terminal 30C may be carried by user B and user C, respectively.

[0051] The communication terminals 30A to 30C at the locations A to C may be terminals that have built-in imaging devices 8, 9 or terminals that can be externally attached. The communication terminals 30A to 30C can distribute images of their own locations captured by their own imaging devices 8, 9, etc. to other locations. Any devices may be installed at the locations A to C.

[0052] The arrangement of each terminal and device (communication terminal 30 and imaging device) and user shown in Figure 2 is one example, and other examples may be used. Furthermore, communication terminal 30 is not limited to a PC, and may be, for example, a tablet terminal, a smartphone, a wearable device, a projector, an electronic whiteboard (a whiteboard with an electronic blackboard function that allows intercommunication), an autonomous robot, or the like. Communication terminal 30 may be any computer that runs a web browser or an application dedicated to an image distribution service.

[0053] Furthermore, if the imaging device 10 has a display, it may be configured to display images distributed from other locations.

[0054] The communication management system 50 has one or more information processing devices. The communication management system 50 manages and controls communications between the imaging devices 10 and communication terminals 30 at each location, and manages the wide-field images sent and received. The communication management system 50 provides a platform that allows users to utilize the functions necessary to provide an image distribution service that distributes wide-field images. This platform may be made available to service providers, such as individuals or companies, who wish to provide image distribution services, through a contract. Hereinafter, to distinguish from tenants who receive image distribution services, service providers who use the platform to provide image distribution services to users will be referred to as platform subscribers.

[0055] For this reason, the communication management system 50 may publish an API (Application Programming Interface) as a platform, so that platform subscribers can use this API to provide various image distribution services. Platform subscribers only need to develop software such as applications that mainly perform screens displayed by the communication terminals 30 and call APIs, and do not need to develop functions provided by APIs such as image distribution from scratch.

[0056] The communication management system 50 may be constructed by a single computer, or may be constructed by multiple computers to which each unit (function or means) is divided and arbitrarily assigned. Furthermore, all or part of the functions of the communication management system 50 may be implemented by a server computer in a cloud environment or a server computer in an on-premise environment.

[0057] The storage 90 is a storage device separate from the communication management system 50. The storage 90 may be external storage (which may be storage located on the cloud or on-premise), or may be storage included in the communication management system 50.

[0058] <Hardware configuration example> Next, the hardware configuration of each device or terminal included in the image communication system according to this embodiment will be described with reference to Figures 3 and 4. Note that components may be added or deleted from the hardware configurations shown in Figures 3 and 4 as needed.

[0059] <<Hardware configuration of imaging device>> First, the hardware configuration of the imaging device 10 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the hardware configuration of the imaging device 10. In the following, the imaging device 10 is assumed to be an omnidirectional (omnidirectional) imaging device using two imaging elements, but the number of imaging elements may be two or more. Furthermore, the imaging device does not necessarily have to be a device dedicated to omnidirectional imaging; an omnidirectional imaging unit may be attached to a regular digital camera, smartphone, or the like to have substantially the same functions as the imaging device 10.

[0060] As shown in FIG. 3, the imaging device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0061] Of these, imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) each having a field angle of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each lens. Imaging elements 103a and 103b include image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors that convert optical images captured by lenses 102a and 102b into electrical image data for output, a timing generation circuit that generates horizontal and vertical synchronization signals and pixel clocks for the image sensors, and a group of registers that set various commands or parameters required for the operation of the imaging elements. Note that the configuration in which imaging unit 101 includes two wide-angle lenses is merely an example, and imaging unit 101 may include only one, three, or more wide-angle lenses.

[0062] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to a CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a network I / F 121, and the like.

[0063] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create equirectangular projection image data, which will be described later.

[0064] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0065] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function for displaying moving images on a display (for example, a display of an external terminal such as a smartphone that performs short-range communication with the imaging device 10 using the short-range communication circuit 117). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0066] As will be described later, the imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b. Although the imaging device 10 is not provided with a display unit in this embodiment, a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 receives the sound data output from the microphone 108 via an I / F bus and performs predetermined processing on the sound data.

[0067] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of processed equirectangular projection images.

[0068] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. The user operates the operation unit 115 to input various imaging modes, imaging conditions, etc.

[0069] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as needed.

[0070] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the imaging device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0071] The electronic compass 118 calculates the orientation of the imaging device 10 from the Earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) conforming to the Exif standard and is used for image processing such as image correction of the captured image. The related information also includes data such as the image capture date and time and the data size of the image data. The gyro sensor 119 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) associated with the movement of the imaging device 10. The angle changes are an example of related information (metadata) conforming to the Exif standard and are used for image processing such as image correction of the captured image. The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The imaging device 10 calculates the attitude (angle with respect to the direction of gravity) of its own device (the imaging device 10) based on the acceleration detected by the acceleration sensor 120. The imaging device 10 has the acceleration sensor 120, which improves the accuracy of image correction.

[0072] The network I / F 121 is an interface for performing data communication using a communication network N such as the Internet via a router or the like.

[0073] <<Hardware configuration of communication terminal>> 4 is a diagram showing an example of the hardware configuration of the communication terminal 30 and the communication management system 50. First, the communication terminal 30 will be described. The hardware configuration of the communication terminal 30 is indicated by reference numerals in the 300 series. The communication terminal 30 is constructed by a computer, and as shown in FIG. 4, includes a CPU 301, a ROM 302, a RAM 303, a hard disk (HD) 304, an HDD controller 305, a display 306, an external device connection I / F 308, a network I / F 309, a bus line 310, a keyboard 311, a pointing device 312, a DVD-RW (Digital Versatile Disk Rewritable) drive 314, a media I / F 316, an audio input / output I / F 317, a microphone 318, a speaker 319, a short-range communication circuit 320, and a camera 321.

[0074] Of these, the CPU 301 controls the overall operation of the communication terminal 30. The ROM 302 stores programs, such as an IPL, used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data, such as programs. The HDD controller 305 controls the reading and writing of various data from and to the HD 304 under the control of the CPU 301. The display 306 displays various information, such as a cursor, menus, windows, characters, or images. The display 306 is an example of a display unit. The display 306 may also be a touch panel display equipped with input means. The external device connection I / F 308 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory or a printer. The network I / F 309 is an interface for data communication using the communication network N. The bus line 310 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 301, shown in FIG. 4.

[0075] The keyboard 311 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. Note that the input means may be not only the keyboard 311 and the pointing device 312, but also a touch panel, a voice input device, etc. The DVD-RW drive 314 controls reading and writing of various data from and to a DVD-RW 313, which is an example of a removable recording medium. Note that the DVD-RW 313 may be a DVD-R or a Blu-ray (registered trademark) Disc. The media I / F 316 controls reading and writing (storing) of data from and to a recording medium 315 such as a flash memory. The microphone 318 is a type of built-in sound collection means for inputting audio. The audio input / output I / F 317 is a circuit that processes input and output of audio signals between the microphone 318 and a speaker 319 under the control of the CPU 301. The short-range communication circuit 320 is a communication circuit for communicating with an external terminal (device) by short-range wireless communication technology such as NFC, Bluetooth (registered trademark), or Wi-Fi. The camera 321 is a type of built-in imaging means that captures an image of a subject and obtains image data. Note that the microphone 318, speaker 319, and camera 321 do not have to be built into the communication terminal 30, but may be external devices.

[0076] <<Hardware configuration of information processing system>> As shown in Fig. 4, each piece of hardware configuration in the communication management system 50 is indicated by a reference number in the 500 range in parentheses. The communication management system 50 is constructed using a computer, and as shown in Fig. 4, has the same configuration as the communication terminal 30, so a description of each piece of hardware configuration will be omitted.

[0077] Each of the above programs may be recorded as an installable or executable file on a computer-readable recording medium and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the communication terminal 30 realizes the display method according to the present invention by executing the program according to the present invention.

[0078] <Wide-field images and viewpoint information> A method for generating a wide-field image (spherical image) will be described below with reference to Figs. 5 to 12. First, the appearance of the imaging device 10 will be described with reference to Fig. 5. The imaging device 10 is a digital camera for obtaining captured images that are the basis for a spherical (360°) image. Fig. 5(a) is a left side view of the imaging device, Fig. 5(b) is a front view of the imaging device, and Fig. 5(c) is a plan view of the imaging device. This external view is merely one example of the imaging device 10, and other external views may also be used.

[0079] As shown in FIG. 5(a), the imaging device 10 is sized to be held in one hand, but this shape is merely an example and other shapes may be used. As shown in FIGS. 5(a), 5(b), and 5(c), the imaging device 10 has an imaging element 103a on the front side (front side) and an imaging element 103b on the rear side (rear side) at the top of the imaging device 10. These imaging elements (image sensors) 103a and 103b are used in conjunction with optical components (e.g., lenses 102a and 102b, described below) capable of capturing hemispherical images (with a field angle of 180° or more). As shown in FIG. 5(b), an operation unit 115, such as a shutter button, is provided on the surface of the imaging device 10 opposite the front side. As described above, the imaging device 10 may have only one imaging element, or three or more imaging elements.

[0080] Next, the usage of the imaging device 10 will be described with reference to Fig. 6. Fig. 6 is an image diagram of the imaging device in use. As shown in Fig. 6, the imaging device 10 is used, for example, to capture an image of a subject around the imaging device 10. In this case, two hemispherical images can be obtained by capturing images of the subject around the imaging device 10 using the imaging element 103a and the imaging element 103b shown in Fig. 5, respectively.

[0081] Next, an outline of the process of creating a celestial sphere image from an image captured by the imaging device 10 will be described with reference to Fig. 7 and Fig. 8. Fig. 7(a) is a diagram showing a hemispherical image (front side) captured by the imaging device, Fig. 7(b) is a diagram showing a hemispherical image (rear side) captured by the imaging device, and Fig. 7(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 8(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 8(b) is a diagram showing a celestial sphere image.

[0082] As shown in Fig. 7(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a lens 102a (described later). Also, as shown in Fig. 7(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a lens 102b (described later). Then, the image capturing device 10 combines the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees to create an equirectangular projection image EC as shown in Fig. 7(c).

[0083] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to apply an equirectangular projection image EC to cover the spherical surface as shown in FIG. 8(a) and create a celestial sphere image (celestial sphere panoramic image) CE as shown in FIG. 8(b). In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. OpenGL ES is merely an example of software that performs image processing, and the celestial sphere image CE may be created using other software. Furthermore, the celestial sphere image CE may be a still image or a video.

[0084] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people. Therefore, the imaging device 10 displays a predetermined region T (hereinafter referred to as a "predetermined region image") that is a part of the spherical image CE as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to FIGS. 9 and 10.

[0085] FIG. 9 is a diagram showing the positions of a virtual camera and a predetermined region when a celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user viewing a celestial sphere image CE displayed as a three-dimensional sphere. FIG. 10(a) is a three-dimensional perspective view of FIG. 9, and FIG. 10(b) is a diagram showing the predetermined region image when displayed on a display. FIG. 10(a) shows the celestial sphere image CE shown in FIG. 9 as a three-dimensional sphere CS. If the celestial sphere image CE generated in this manner is a three-dimensional sphere CS, the virtual camera IC is located inside the celestial sphere image CE as shown in FIG. 9. The predetermined region T in the celestial sphere image CE is an imaging region of the virtual camera IC, and is specified by predetermined region information that indicates the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE. Zooming of the predetermined region T can also be achieved by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined area image Q is an image of a predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE (see FIG. 11).

[0086] The predetermined area image Q shown in FIG. 10(a) is then displayed on a predetermined display as an image of the imaging area of ​​the virtual camera IC, as shown in FIG. 10(b). The image shown in FIG. 10(b) is a predetermined area image represented by the initial (default) predetermined area information. The following description will be given using the imaging direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be represented by position coordinates (X, Y, Z) of the imaging area of ​​the virtual camera IC, which is the predetermined area T, instead of the angle of view α and distance f.

[0087] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 11. FIG. 11 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 11, "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the angle of view (Angle). That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC, indicated by the imaging direction (ea, aa), becomes the center point CP(x, y) of the predetermined area T, which is the imaging area of ​​the virtual camera IC. As shown in FIG. 11, when the diagonal angle of view of the predetermined area T, represented by the angle of view α of the virtual camera IC, is α, the center point CP(x, y) becomes the parameter ((x, y)) of the predetermined area information. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between any vertex of the predetermined region T and the center point CP(x, y) (2L is the diagonal). In FIG. 11, the trigonometric function shown in the following (Equation 1) generally holds.

[0088]

number

[0089] FIG. 12 is a diagram illustrating the relationship described in FIG. 11 using points in a three-dimensional Euclidean space based on spherical coordinates. Here, the position coordinates of the center point CP shown in FIG. 11 when expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius vector, polar angle, and azimuth angle, respectively. The radius vector r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to the center point CP, and is therefore equal to the distance f shown in FIG. 11. FIG. 12 is a diagram illustrating these relationships. Hereinafter, the position coordinates (r, θ, φ) of the virtual camera IC will be used as an example of viewpoint information. Note that the viewpoint information may be parameter information that can identify the predetermined area T (predetermined area image Q) displayed as an image of the imaging area of ​​the virtual camera IC on the predetermined display shown in FIG. 10 as described above, and also includes coordinates of the diagonal vertices of the predetermined area T. Furthermore, information indicating the angle of view α of the virtual camera IC and information indicating the center point CP(x, y) described in FIG. 11 may be the viewpoint information. Moreover, the viewpoint information includes not only position coordinate information based on spherical coordinates, but also position coordinate information based on Cartesian coordinates and a difference value of coordinates from initially set (default) predetermined area information. Moreover, the viewpoint information may be information other than coordinate information, such as angle or distance, as shown in FIG. 11. Moreover, although the center point of the predetermined area T is used as a reference in FIGS. 11 and 12, the predetermined area T may also be specified by parameter information based on one of the vertices of the predetermined area T. Note that, although the viewpoint information has been described above as an example in which the wide-field image is a celestial sphere image, in the case of other wide-field images, information that specifies the predetermined area T in that wide-field image becomes the viewpoint information.

[0090] <About the function> Next, the functional configuration of the communication system 1 according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the functional configuration of the communication system 1 according to this embodiment. Note that Fig. 13 shows the communication terminals and servers shown in Fig. 1 that are related to the processes or operations described below.

[0091] <<Functional configuration of the imaging device>> First, the functional configuration of the imaging device 10 will be described with reference to Fig. 13. The imaging device 10 has a communication unit 11, a reception unit 12, an imaging processing unit 13, an analysis unit 14, a registration request unit 15, a connection unit 16, a storage processing unit 17, an image transmission control unit 18, and a storage / readout unit 19. Each of these units is a function or a means for performing the function, which is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 111 in accordance with a program loaded on the SRAM 113 or DRAM 114. The imaging device 10 also has a storage unit 1000 constructed using the ROM 112 shown in Fig. 3, etc.

[0092] Communication unit 11 has a function of connecting to communication network N using wireless communication means such as Wi-Fi and transmitting and receiving various data or information to and from other devices. In this embodiment, a form in which wide-field images acquired by imaging processing unit 13 are transmitted to communication management system 50 via connection unit 16 will be mainly described. However, communication unit 11 can also transmit wide-field images to communication management system 50.

[0093] The reception unit 12 has a function of receiving operation input from the user to the imaging device 10. The reception unit 12 receives input such as power on / off and shutter button on / off (start or stop transmission of a wide-field image).

[0094] The imaging processing unit 13 captures an image of a subject, a landscape image, or the like, and acquires the captured image. The captured image acquired by the imaging processing unit 13 may be a moving image or a still image (or both), and may include audio along with the image. The imaging processing unit 13 also captures, for example, a two-dimensional code displayed on the display 306 of the communication terminal 30.

[0095] The analysis unit 14 analyzes the two-dimensional code captured and acquired by the imaging processing unit 13, and extracts information contained in the two-dimensional code (a URL for registering the imaging device to a tenant, a temporary ID and a password).

[0096] The registration request unit 15 uses the information included in the two-dimensional code read by the analysis unit 14 to send a request to the communication management system 50 to register the imaging device 10 with the tenant of the communication management system 50.

[0097] The connection unit 16 is realized by, for example, a short-distance communication circuit 117, and has the function of receiving power supply from the communication terminal 30A and performing data communication.

[0098] The storage processing unit 17 performs processing to store a wide-field image captured in response to an imaging request from an arbitrary location in a URL (for example, storage 90, etc.) notified by the communication management system 50.

[0099] The image transmission control unit 18 has a function of controlling the transmission of wide-field images to the communication management system 50. For example, the image transmission control unit 18 transmits captured images acquired by the imaging processing unit 13 to the communication management system 50 periodically or in response to a user operation if the captured images are still images, or at a predetermined FPS (frames per second) if the captured images are moving images. The image transmission control unit 18 also switches between the communication unit 11 and the connection unit 16.

[0100] The storage / readout unit 19 has a function of storing various data in the storage unit 1000 or reading out various data from the storage unit 1000. The storage unit 1000 also stores captured image data, an imaging device ID, and the like acquired by the imaging processing unit 13. The captured image data stored in the storage unit 1000 may be configured to be deleted when a predetermined time has elapsed since the image data was acquired by the imaging processing unit 13, or the data transmitted to the communication management system 50 may be deleted.

[0101] An application (also called a plug-in) for compatibility with the communication system 1 is installed in the imaging device 10. This application is not necessary for commercially available imaging devices 10, but is used when associating the imaging device 10 with a virtual room and accepting external control. Some of the functions shown in Fig. 13 (for example, the registration request unit 15) are provided by this application.

[0102] <<Functional configuration of communication terminal>> Next, the functional configuration of communication terminal 30 will be described with reference to Fig. 13. Communication terminal 30 has communication unit 31, reception unit 32, display control unit 33, imaging unit 34, storage / readout unit 35, connection unit 36, chat input reception unit 37, control data analysis unit 38, video display unit 39, audio playback unit 40, and chat display unit 41. Each of these units is a function or a means for performing the function, which is realized when any of the components shown in Fig. 4 operates in response to an instruction from CPU 301 in accordance with a program (which may be a web browser or a dedicated application) deployed on RAM 303. Communication terminal 30 also has storage unit 3000 constructed by ROM 302 or recording medium 315 shown in Fig. 4.

[0103] The communication unit 31 is realized by, for example, a network I / F 309, and has a function of connecting to a communication network N and transmitting and receiving various data or information to and from other devices.

[0104] The reception unit 32 has a function of receiving various selections or operation inputs to the communication terminal 30. The display control unit 33 has a function of displaying a wide-field image, an image with a normal angle of view, and various screens on the display 306 of the communication terminal 30. The display control unit 33 displays, for example, a two-dimensional code transmitted from the communication management system 50 on the display 306. The two-dimensional code is, for example, a QR code (registered trademark), DataMatrix (DataCode), MaxiCode, PDF417, or the like. The two-dimensional code may also be a barcode.

[0105] The connection unit 36 ​​is realized by, for example, a short-distance communication circuit 320, and has the function of supplying power to the image capture device 10 and performing data communication.

[0106] The chat input receiving unit 37 receives input of, for example, a question for the chatbot from the user. The input may be made using a keyboard or by voice input.

[0107] The control data analysis unit 38 analyzes the control data transmitted from the communication management system 50, and requests the video display unit 39 or the audio playback unit 40 to process the video data or audio data in accordance with the control data.

[0108] The audio playback unit 40 processes the audio data in response to a request from the control data analysis unit 38, and outputs the processed audio frames from a speaker. The video display unit 39 processes the video data in response to a request from the control data analysis unit 38, and displays the processed video data on the display 506.

[0109] The chat display unit 41 displays the response text from the chatbot received from the communication management system 50 on the display 506.

[0110] The storage / readout unit 35 is executed by instructions from the CPU 301 shown in Fig. 4, and has the function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. An image management information storage unit 3001 is formed in the storage unit 3000. The image management information storage unit 3001 will be described in detail in the description of the communication management system 50.

[0111] <<Functional configuration of information processing system>> Next, the functional configuration of the communication management system 50 will be described. The communication management system 50 includes a communication unit 51, a screen generation unit 52, an association processing unit 53, an image distribution unit 54, an authentication unit 55, a communication group management unit 56, a communication control unit 57, a connection management unit 58, a memory / readout unit 59, an API management unit 60, a chat question response unit 61, a chat question analysis unit 62, a video analysis unit 63, and an audio analysis unit 64. Each of these units is a function or a means for performing the function, which is realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 501 in accordance with a program loaded on the RAM 503. The information processing system also includes storage units 5000 and 5100, which are constructed using the ROM 502, HD 504, or recording medium 515 shown in FIG. 4.

[0112] The communication unit 51 has a function of transmitting and receiving various data or information to and from other devices via the communication network N.

[0113] The screen generation unit 52 generates screen information to be displayed by the communication terminal 30. When the communication terminal 30 executes a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), etc. When the communication terminal 30 executes a native application, the screen information is held by the communication terminal 30, and the information to be displayed is transmitted in XML, etc. The screen generation unit 52 generates screen information on which the wide-field image, etc. delivered by the image delivery unit 54 is arranged.

[0114] The association processing unit 53 controls the sharing of viewpoint information of the wide-field image. When the association processing unit 53 receives an imaging request together with viewpoint information from the communication terminal 30, it performs processing to associate the acquired wide-field image with the viewpoint information by requesting imaging from the imaging device 10. Furthermore, the association processing unit 53 stores the associated wide-field image and viewpoint information in the image management information storage unit 5001. The association processing unit 53 also transmits a URL to the communication terminal 30 as information indicating a storage location where the associated wide-field image and viewpoint information are stored. Note that the communication management system 50 does not need to receive the viewpoint information and the imaging request from the communication terminal 30 simultaneously; they may be received separately and then associated. The URL is an example of information indicating a storage location, and may be in another format, such as a URI.

[0115] The image distribution unit 54 distributes wide-field images transmitted by the imaging device 10 associated with the same virtual room to the communication terminal 30 operated by the user currently in the virtual room. Images with a normal angle of view captured by the imaging device of the communication terminal 30 or the connected imaging devices 8 and 9 are also distributed in the same manner.

[0116] The authentication unit 55 has a function of authenticating the requester based on an authentication request received by the communication unit 51. The authentication unit 55 authenticates the user, for example, by checking whether authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches pre-stored authentication information. Note that the authentication information may be biometric authentication information such as an IC card number, face, fingerprint, or voiceprint. The authentication unit 55 may also perform authentication using an external authentication system or an authentication method such as OAuth.

[0117] The communication group management unit 56 manages the entry of the communication terminal 30 and the user into the virtual room, the association of devices, etc. If the authentication unit 55 is successful, the communication group management unit 56 registers the user ID and the IP address of the communication terminal 30 in the virtual room information storage unit 5002, and associates the imaging device 10 with the virtual room.

[0118] The communication control unit 57 manages the start, establishment, and termination of communication with the imaging device 10 associated with each virtual room. The communication control unit 57 also manages the start, establishment, and termination of communication for delivering wide-field images and audio in response to the communication terminal 30 entering or leaving the virtual room.

[0119] The connection management unit 58 manages the communications (connections) established between the communication terminal 30 and the imaging device 10 and the communication management system 50 in association with the virtual rooms.

[0120] The API management unit 60 manages the API used by the platform contractor when providing an image distribution service for wide-field images. To use the API, the platform contractor may separately develop software that calls the API. The developed software may run on a server or on a client such as a communication terminal. Functions included in the communication management system 50, such as the image distribution unit 54, the association processing unit 53, and the communication control unit 57, can be provided as APIs. Functions added to the communication management system 50 later can also be provided as APIs. Whether or not a function is provided as an API can be determined by a communication terminal operated by the platform provider accessing the communication management system 50 and accepting API disclosure settings, allowing the API management unit 60 to control the API based on the disclosure settings. The API management unit 60 may also perform authentication processing to verify whether the software requesting the API call is software developed by a legitimate platform contractor. The authentication processing can be performed by comparing information stored in the platform contractor information storage unit with information sent from the requesting software. As a specific example of authentication processing, the communication management system 50 receives from the requesting software an app ID previously issued by the API management unit 60 for software developed by a platform contractor, and if the API management unit 60 determines that the app ID is stored in the platform contractor information storage unit, the API management unit 60 performs control to permit provision of the API as legitimate software. On the other hand, if the software cannot be determined to be legitimate, the API management unit 60 performs control to not permit provision of the API. Note that the app ID is an example of authentication information for determining legitimacy, and the API management unit 60 may also confirm the legitimacy of the requester using authentication information such as an access token, ticket, security key, password, or PIN code previously issued by the API management unit 60 of the information processing system or an external system.In this embodiment, we will not explain how to use the functions provided by the communication management system 50 as an API, but the processing flow may be the same except that software such as an application developed by a platform subscriber uses the functions provided by the communication management system 50 through judgment by the API management unit 60.

[0121] The chat question analysis unit 62 analyzes the message from the communication terminal 30, extracts requests related to video, audio, or materials, and calls the video analysis unit 63 or the audio analysis unit 64. If there is a request related to materials, the chat question analysis unit 62 refers to the material storage unit 5102 and acquires the material URL. The chat question analysis unit 62 also creates control data based on the analysis results by the video analysis unit 63 or the audio analysis unit 64 and the material URL, and stores the control data in the control data storage unit 5101.

[0122] The video analysis unit 63 acquires the latest wide-field image (hereinafter sometimes referred to as the latest video frame) stored in the video frame storage unit 5103, analyzes the latest video frame, and returns the analysis result to the chat question analysis unit 62. Processing based on the analysis result (which is included in the control data) is performed by the communication terminal 30.

[0123] The voice analysis unit 64 acquires the latest voice data (hereinafter sometimes referred to as the latest voice frame) stored in the voice frame storage unit 5104, analyzes the latest voice frame, and returns the analysis result to the chat question analysis unit 62. Processing based on the analysis result (which is included in the control data) is performed by the communication terminal 30.

[0124] The chat question answering unit 61 creates a response to a question using an existing method. The chat question answering unit 61 also acquires control data from the control data storage unit 5101 and passes it to the communication unit 51 together with the response. The chat question answering unit 61 and the chat question analysis unit 62 may use a large-scale language model to create a response or analyze a request. A large-scale language model is a language model composed of a neural network with many parameters, and is a generative model trained by self-supervised learning or semi-supervised learning using a huge amount of unlabeled text. Generative AI using large-scale language models is known. The large-scale language model may be owned by the communication management system 50, or may be used by the communication management system 50 calling an API provided by the large-scale language model.

[0125] The storage / readout unit 59 has the function of storing various data in the storage units 5000 and 5100, or reading out various data from the storage units 5000 and 5100. The storage unit 5000 is also formed with an image management information storage unit 5001 and a virtual room information storage unit 5002. The storage unit 5100 is formed with a control data storage unit 5101, a material storage unit 5102, a video frame storage unit 5103, and an audio frame storage unit 5104.

[0126] "Image management information storage unit 5001" Fig. 14(a) is a conceptual diagram showing image management information stored in image management information storage unit 5001. Image management information such as that shown in Fig. 14 is stored in image management information storage unit 5001. Image management information is information for managing wide-field images captured in response to an imaging request, and when a user sends an imaging request from communication terminal 30, one record of image management information is generated. Each item contained in the image management information will be described below.

[0127] The data ID is identification information that identifies the wide-field image. The data ID is assigned by the communication management system 50. ID is an abbreviation for Identification and means identifier or identification information. An ID refers to a name, code, character string, number, or a combination of one or more of these that is used to uniquely distinguish a specific object from multiple objects.

[0128] The data name is the name of the wide-field image set by the user of the communication terminal 30. The data name can be set by the user, but may also be set automatically.

[0129] The imaging date and time information is information for identifying the date and time when the wide-field image was captured, such as the date and time when the user input an imaging request to the communication terminal 30, or the date and time when the imaging device 10 captured the wide-field image. The imaging date and time information may be replaced with a timestamp of the wide-field image.

[0130] The photographer information is identification information (which may be a user name) of the user who inputs the imaging request to the communication terminal 30. The user inputs the imaging request to the communication terminal 30 while in the virtual room, so the user registered in the photographer information is identified by authentication to the communication management system 50 or the virtual room. The photographer information is sent to the communication management system 50 together with the imaging request.

[0131] The imaging device information is identification information (imaging device ID) of the imaging device 10 that captured the wide-field image. The imaging device ID is assigned by the communication management system 50 and shared with the imaging device 10, but information unique to the imaging device 10, such as a MAC address or serial number, may also be used. The imaging device ID is transmitted to the communication management system 50 together with the wide-field image.

[0132] The predetermined area information of the photographer is viewpoint information consisting of a radius vector, a polar angle, and an azimuth angle. The viewpoint information indicates the center coordinate of the wide-field image being displayed by the communication terminal 30. The viewpoint information is transmitted from the communication terminal 30 that requests imaging. Note that the viewpoint information may include information specifying the width and height of the display range in addition to the radius vector, polar angle, and azimuth angle. Alternatively, the viewpoint information may be only the width and height of the display range.

[0133] The virtual room ID at the time of image capture is identification information of the virtual room with which the image capture device 10 is associated.

[0134] Data storage location information is the URL or file path where the wide-field image is saved.

[0135] 14(b) is also a conceptual diagram showing image management information. In FIG. 14(b), wide-field images with the same virtual room ID at the time of capture are stored. Image management information may be classified by virtual room.

[0136] "Virtual room information storage unit 5002" FIG. 15 is a conceptual diagram showing virtual room information stored in virtual room information storage unit 5002. Virtual room information such as that shown in FIG. 15 is stored in virtual room information storage unit 5002. The virtual room information is information related to a virtual room and is held for each virtual room. Each item contained in the virtual room information will be described below. Note that the virtual rooms are registered with tenants.

[0137] The virtual room ID is identification information for identifying a virtual room. In this embodiment, the user can create any virtual room.

[0138] The virtual room name is a name that allows users to distinguish between virtual rooms, and can be set by the user at their discretion.

[0139] The imaging device information is identification information (imaging device ID) of the imaging device 10 associated with the virtual room. The currently active user is the user ID of the user currently in the virtual room. This user is a user who can view wide-field images. The entry method will be described later. The user ID is also associated with the IP address of the communication terminal 30 operated by the user.

[0140] "Control data storage unit," "material storage unit," "video frame storage unit," and "audio frame storage unit" The control data storage unit 5101 stores the control data created by the chat question analysis unit 62. The control data is updated by the chat question analysis unit 62 until the user sends a request to end this control mode, so the latest control data is stored in the control data storage unit 5101.

[0141] The material storage unit 5102 stores various materials. The materials may be prepared within a company, such as a specific database, knowledge base, or tacit knowledge. Alternatively, the materials may simply be various files stored in folders or directories. In addition to materials managed by the communication management system 50, the material storage unit 5102 may also store data on the Internet or a commercial (e.g., paid) database.

[0142] The video frame storage unit 5103 always stores the latest video frame. A video frame is one piece of image data that constitutes a moving image. The communication unit 51 overwrites the video frame storage unit 5103 with the video frame received from the imaging device 10. The audio frame storage unit 5104 always stores the latest audio frame. The audio frame may be audio data for one video frame, audio data from the last silent period to the next silent period (provided that the upper limit time is not exceeded), or audio data for a certain period of time. The communication unit 51 overwrites the video frame storage unit 5103 with the video frame received from the imaging device 10.

[0143] <Addition to the block diagram in Figure 13> In the block diagram of Figure 13, we will briefly explain the processing flow from when a user sends a question to the chatbot to when the response is fed back to the video and audio being viewed.

[0144] 1. First, the user inputs a message, which is a question for the chatbot, into the communication terminal 30. The chat input receiving unit 37 receives the input of this message. The communication unit 31 of the communication terminal 30 transmits the message to the communication management system 50.

[0145] 2. The communication unit 51 of the communication management system receives the message, and the chat question analysis unit 62 analyzes the message and performs request extraction processing related to video, audio, or materials. In the case of a request related to video or audio, the chat question analysis unit 62 calls the video analysis unit 63 or audio analysis unit 64. The video analysis unit 63 and audio analysis unit 64 perform processing related to the request while referring to the video frame storage unit 5103 or audio frame storage unit 5104, and return the analysis results to the chat question analysis unit 62.

[0146] 3. In the case of a request for materials, the chat question analysis unit 62 acquires the material URL from the material storage unit 5102.

[0147] 4. The chat question analysis unit 62 creates control data based on the analysis results and the document URL, and stores the control data in the control data storage unit 5101.

[0148] 5. The chat question response unit 61 acquires the control data from the control data storage unit 5101 and passes the control data and the response sentence to the communication unit 51.

[0149] 6. The communication unit 51 transmits the control data and the response text to the communication terminal 30 together with the video frame and the audio frame.

[0150] 7. The communication unit 31 of the communication terminal 30 receives the control data, the response message, the video frame, and the audio frame. The control data analysis unit 38 analyzes the control data and instructs the content of the analysis to the video display unit 39 or the audio playback unit 40. The video display unit 39 or the audio playback unit 40 processes the video frame and the audio frame and displays or plays them.

[0151] 8. The chat display unit 41 displays the response. The control data continues to be sent to the communication terminal 30 unless the user sends an instruction to stop the control to the communication management system 50. That is, the control data is sent together with the video frames and audio frames. The instruction to stop from the user is received by the chat input receiving unit 37.

[0152] 16 is a diagram for explaining in detail the processing performed by the chat question analysis unit 62, the video analysis unit 63, and the audio analysis unit 64. Below, it will be explained in detail how the chat question analysis unit 62 analyzes the message that is the question, and how the video analysis unit 63 and the audio analysis unit 64 analyze the video frames and the audio frames.

[0153] 1. First, the chat question analysis unit 62 breaks down the input question sentence into words and performs syntactic analysis using general natural language processing techniques (word and syntactic analysis 62a). Next, the chat question analysis unit 62 performs video-related request extraction processing 62b, audio-related request extraction processing 62c, or document-related request extraction processing 62d on the question sentence. All request extractions may be performed in a single message. Video-related requests include person masking processing, person tracking processing, etc. Audio-related requests include abnormal sound detection processing, noise canceling processing, etc.

[0154] 2. After extracting the request, if there is a matching process registered in the video analysis unit 63 or the audio analysis unit 64, the chat question analysis unit 62 calls that process. The video analysis unit 63 performs the requested analysis for the person masking process 63a, person tracking process 63b, and other processes 63c on the latest video frame in the video frame storage unit 5103. The person masking process 63a is a process that reduces the visibility of people appearing in the image data. Specifically, the person masking process 63a may be a mask such as a solid black mask, a non-display (deletion of the person area), a blur, or a reduction in resolution. The person tracking process 63b is a process that continuously enlarges and displays the area in which the specified person appears.

[0155] Here, analysis means identifying the coordinates of the person in the case of person masking processing 63a or person tracking processing 63b, and the processing itself is performed by the communication terminal 30. Therefore, the analysis result is the coordinates of the person, etc. Other processing 63c may be any processing that processes the video. For example, a process in which the user changes the viewpoint in chat is conceivable. For example, if the user types "Show me the opposite direction" in chat, the communication management system 50 transmits control data including viewpoint information in the opposite direction to the current direction to the communication terminal 30.

[0156] The audio analysis unit 64 performs the requested processes, namely, abnormal sound detection process 64a, noise canceling process 64b, and other processes 64c, on the latest audio frame in the audio frame storage unit 5104. The abnormal sound detection process 64a is, for example, a process of converting audio data into a frequency spectrum and monitoring the intensity of specific frequencies, or a process of determining the presence or absence of ions using a pre-trained model for detecting abnormal sounds. The noise canceling process 64b may be an existing noise removal process. The analysis results include whether an abnormal sound was detected, noise-canceled audio data, etc.

[0157] The processes registered in the video analysis unit 63 and the audio analysis unit 64 can be customized by the user, and processes can be added by registering a process definition file defined by the user in the communication management system 50. The process definition file includes algorithms, source code, etc.

[0158] <Entering the virtual room from a communication device> Next, the process of user B entering a virtual room will be described with reference to Figures 17 and 18. It is assumed that image capture device 10 has already been associated with the virtual room, and communication terminal 30A has already transmitted a wide-field image and a normal-angle image to communication management system 50 (the association of image capture device 10 with the virtual room will be described in Figure 18 and subsequent figures). In the following description, there is no particular distinction between user B entering a virtual room and communication terminal 30B operated by user B entering a virtual room.

[0159] FIG. 17 shows an example of a screen displayed by communication terminal 30B when user B enters a virtual room. FIG. 17(a) is an example of an entry screen 200. To add, before the entry screen 200 is displayed, user B logs in to the communication management system 50. By logging in, the tenant to which user B belongs is identified. Virtual rooms are associated with tenants. User B displays a list of virtual rooms associated with the tenants on communication terminal 30B (see FIG. 19) and selects the virtual room to enter from the list. FIG. 17(a) is the entry screen 200 for the virtual room selected by user B in this way.

[0160] Alternatively, the creator of the virtual room may request the communication management system 50 to issue a URL corresponding to the virtual room, and then send this URL by email or the like to user B. When user B clicks on the URL displayed on communication terminal 30B, communication terminal 30B displays room entry screen 200 shown in FIG. 17(a).

[0161] The entry screen 200 has a virtual room name 201, a participant name input field 202, and an enter button 203. The virtual room name 201 is the same as that stored in the virtual room information storage unit 5002. The participant name input field 202 may be a nickname or other name for user B. Since the user name of user B is identified when user B logs in, this user name may be displayed automatically. The enter button 203 is a button that user B clicks to request entry into the virtual room.

[0162] Note that authentication for entering the virtual room may be required at the time of entry, separately from logging in to the tenant.

[0163] FIG. 17(b) shows an image viewing screen 210 displayed by communication terminal 30B when user B enters the virtual room. On image viewing screen 210 in FIG. 17(b), imaging device 10 has already started distributing a wide-field-of-view image, and communication terminal 30A has already started distributing an image with a normal field of view. Therefore, image viewing screen 210 has a first image field 211 and a second image field 212. A wide-field-of-view image is displayed in first image field 211, and an image with a normal field of view is displayed in second image field 212. If there are three or more locations transmitting images, image viewing screen 210 is divided according to the number of transmitting locations.

[0164] A wide-field-of-view image mark 213 is displayed in the first image field 211. The wide-field-of-view image mark 213 is set by the screen generation unit 52 of the communication management system 50 after determining that the image to be displayed in the first image field 211 is a wide-field-of-view image. The wide-field-of-view image mark 213 may also be determined and displayed by the communication terminal 30B. By looking at the wide-field-of-view image mark 213, user B knows that the viewpoint can be changed. Also, the first image field 211 displays a device name 214 (transmitted from the imaging device 10 together with the wide-field-of-view image). The device name 214 is set by user A, as will be described later.

[0165] Participant name 215 is displayed in second image field 212. Participant name 215 is the participant name of a user who has already entered the participant name input field 202 (in this case, since user A has already entered the room, "AAA" entered by user A in participant name input field 202) displayed in participant name input field 202.

[0166] FIG. 18 is a sequence diagram illustrating a process in which user B (or communication terminal 30B) enters a virtual room.

[0167] S1: First, user B at site B performs an operation to display a list screen of virtual rooms. When the reception unit 32 receives the operation to display the list screen, the display control unit 33 of the communication terminal 30B causes the display 306 to display a selection screen.

[0168] S2: When user B selects a selection button for a virtual room, the reception unit 32 of the communication terminal 30B receives the selection of the virtual room. The display control unit 33 of the communication terminal 30B displays the entry screen 200 on the display 306.

[0169] S3: User B enters the necessary information and presses the enter button 203. When the reception unit 32 receives the press, the communication unit 31 of communication terminal 30B transmits a request to enter the virtual room to the communication management system 50. This entry request includes the virtual room ID indicating the virtual room selected in step S2, the user ID of logged-in user B, and the IP address of communication terminal 30B, which is the requesting terminal.

[0170] S4: As a result, the communication unit 51 of the communication management system 50 receives the entry request. The communication group management unit 56 registers the logged-in user ID and IP address in the virtual room information specified by the virtual room ID in the virtual room information storage unit 5002.

[0171] S5: Then, the communication unit 51 of the communication management system 50 transmits a message indicating that the user has entered the room to the communication terminal 30B. As a result, the communication unit 31 of the communication terminal 30B receives the message indicating that the user has entered the room.

[0172] FIG. 19 shows an example of a virtual room association screen (part 1) 260 for associating the imaging device 10 with a virtual room. The screen configuration may be the same for VR goggles 89 and smart glasses 88. The virtual room association screen (part 1) 260 has a virtual room list 261. The virtual room list 261 displays individual virtual room fields 262 to 264 based on the virtual rooms created in the tenant. Each of the individual virtual room fields 262 to 264 has a link issue button 265, an enter button 266, a settings button 267, and a virtual room name 268. The link issue button 265 is a button for issuing a link to the virtual room (a URL for invitation) and a passcode. The enter button 266 is a button for user A to enter the virtual room. The settings button 267 is a button for associating the imaging device 10 with the virtual room. The virtual room name 268 is the same as the name stored in the virtual room information storage unit 5002. Therefore, user A presses setting button 267. By pressing setting button 267, communication terminal 30A displays virtual room association screen (part 2) 270.

[0173] Furthermore, if a device has already been associated with a virtual room, the name 269 of the device is displayed in the individual virtual room field (individual virtual room field 264 in the drawing).

[0174] 20 shows an example of the virtual room matching screen (part 2) 270. The virtual room matching screen (part 2) 270 is displayed as a pop-up on the virtual room matching screen (part 1) 260. The screen transition from the virtual room matching screen (part 1) 260 to the virtual room matching screen (part 2) 270 does not go through the communication management system 50, but a screen transition that goes through the communication management system 50 is also possible.

[0175] The virtual room association screen (part 2) 270 has a name 271 (not yet registered and therefore unregistered in the figure) of the image capture device 10 currently (already) associated with the virtual room, a connection button 272, and a storage button 273. The connection button 272 is a button for displaying a list of devices registered with the tenant. The storage button 273 is a button for displaying a list of storages 90 in which the image capture device 10 associated with the virtual room stores wide-field images. Pressing the connection button 272 causes the communication terminal 30A to display the virtual room association screen (part 3).

[0176] The communication terminal 30A sends the virtual room ID to the communication management system 50 and obtains the names of the devices (including the device IDs, etc.) registered with the tenant in which the virtual room is created, and the names of the devices (including the device IDs, etc.) associated with the virtual room.

[0177] 21 shows an example of a virtual room association screen (part 3) 280. The virtual room association screen (part 3) 280 has a name 281 of the imaging device 10 currently (already) associated with the virtual room, a list of devices that can be added 282, and a save button 283. User A selects a device that he or she wishes to associate with the virtual room from the list of devices that can be added 282, and presses the save button 283. This associates the device with the virtual room (the imaging device ID is registered in the virtual room information storage unit 5002).

[0178] <Wide-field image transmission start process for imaging device> In this way, devices such as the image capture device 10 are associated with the virtual room, but user A needs to operate the device to start transmitting images.

[0179] For the VR goggles 89 and smart glasses 88, user A turns on and off the transmission of images by operating the device itself. This is because currently, a dedicated application for the communication system 1 does not run for the VR goggles 89 and smart glasses 88. If a dedicated application for the communication system 1 runs for the VR goggles 89 and smart glasses 88 as well, user A can remotely turn on and off the transmission of images.

[0180] In the case of the imaging device 10, if the application is enabled, user A can enter the virtual room and turn on or off the transmission of wide-field images from the menu.

[0181] FIG. 22 shows an example of a wide-field-of-view image transmission start / stop dialog box 290 displayed by the communication terminal 30A. The wide-field-of-view image transmission start / stop dialog box 290 is displayed as a pop-up on the image viewing screen 210. Assume that user A operates the communication terminal 30A to enter a virtual room associated with the imaging device 10. The wide-field-of-view image transmission start / stop dialog box 290 displays a name 292 of the imaging device 10 associated with this virtual room. A toggle button 291 is displayed near the name 292, and user A can operate the toggle button 291 to turn on or off the transmission of wide-field images by the imaging device 10. Note that the method of turning on or off using the toggle button is merely an example, and any method may be used as long as the setting is made in response to a user operation. For example, the user may set the setting by selecting a radio button or a predetermined icon, operating a menu, or the like. Alternatively, the transmission of wide-field-of-view images may be automatically started after the imaging device 10 enters the room, eliminating the need for user operation. In addition, certain conditions such as the date and time, the number of users who have entered the room, or the participation of a specific user may be determined in advance, and the transmission of the wide-field image may begin when it is determined that these conditions have been met.

[0182] The communication terminal 30A transmits the state of the toggle button 291 to the communication management system 50. The communication management system 50 transmits a transmission start request or a transmission stop request to the imaging device 10 according to the state of the toggle button 291.

[0183] Fig. 22(a) shows a state in which toggle button 291 is off. Therefore, no wide-field image is displayed in Fig. 22(a). On the other hand, in Fig. 22(a), when communication terminal 30A enters the room, an image with a normal angle of view captured by imaging device 9 of communication terminal 30A has already been shared and is displayed on image viewing screen 210.

[0184] 22(b) shows a state in which the toggle button 291 is on. When the toggle button 291 is on, the communication management system 50 sends an on request to the imaging device 10, and the imaging device 10 starts transmitting a wide-field image. As a result, two images are shared in one virtual room, and the image viewing screen 210 is divided into two.

[0185] <<Procedure for registering an imaging device in a virtual room>> Next, the procedure for registering the imaging device 10 in the virtual room described in the screen transition will be described with reference to Fig. 23. Fig. 23 is an example of a sequence diagram showing the procedure for user A to register the imaging device 10 in the virtual room.

[0186] S11: First, user A connects the communication terminal 30A to the communication management system 50, inputs authentication information (user ID, password, etc.), and requests to log in to the tenant to which the user belongs. The reception unit 32 of the communication terminal 30A receives the operation.

[0187] S12: The communication unit 31 of the communication terminal 30A specifies authentication information and sends a login request to the communication management system 50. The communication unit 51 of the communication management system 50 receives the login request, and the authentication unit 55 performs authentication. Here, it is assumed that the authentication is successful.

[0188] S13: In response to a user operation, the screen generation unit 52 of the communication management system 50 generates the device registration screen 220, and the communication unit 51 transmits screen information of the device registration screen 220 to the communication terminal 30A.

[0189] S14: The communication unit 31 of the communication terminal 30A receives the screen information of the device registration screen 220, and the display control unit 33 displays the device registration screen 220. User A selects the type of device (assuming that the imaging device 10 is selected here), and then inputs the name and description of the imaging device 10. The reception unit 32 receives the input.

[0190] S15: The communication unit 31 of the communication terminal 30A specifies the name and description entered by user A and transmits a request for a two-dimensional code to the communication management system 50.

[0191] S16: The communication unit 51 of the communication management system 50 receives the request for the two-dimensional code. The communication group management unit 56 generates a URL (connection destination for registration) in association with the name and description, and generates a two-dimensional code including the URL, temporary ID, and password. The communication unit 51 of the communication management system 50 transmits the two-dimensional code to the communication terminal 30A. The communication unit 31 of the communication terminal 30A receives the two-dimensional code, and the display control unit 33 displays the two-dimensional code.

[0192] S17: Next, the user A operates the imaging device 10 that he / she wishes to associate with the virtual room, and captures an image of the two-dimensional code. The reception unit 12 of the imaging device 10 receives the operation.

[0193] S18: The imaging processing unit 13 of the imaging device 10 generates image data by performing imaging processing including the two-dimensional code, and the analysis unit 14 analyzes the image data to extract the URL, temporary ID, and password. As a result, the registration request unit 15 connects to the URL via the connection unit 16, specifies the temporary ID and password, and transmits a registration request for the imaging device 10 to the communication management system 50.

[0194] S19: The communication unit 51 of the communication management system 50 receives the temporary ID and password, and determines whether they match the temporary ID and password associated with the URL to which the authentication unit 55 has connected. Here, it is assumed that they match.

[0195] S20: In response to a request to register the imaging device 10, the communication group management unit 56 of the communication management system 50 generates an imaging device ID and registers it in the tenant to which user A has logged in. Note that a name and description are associated with the imaging device ID.

[0196] S21: The communication unit 51 of the communication management system 50 transmits the imaging device ID to the imaging device 10. The connection unit 16 of the imaging device 10 receives the imaging device ID and stores it in the storage unit 1000.

[0197] S22: The communication terminal 30A is notified of the completion of registration, and user A then starts associating the imaging device 10 with the virtual room. User A displays the virtual room association screen (part 1) 260 on the communication terminal 30A and selects the virtual room to which the imaging device 10 registered with the tenant is to be associated. The reception unit 32 of the communication terminal 30A receives the selection.

[0198] S23: Next, user A displays the virtual room association screen (part 2) 270 on the communication terminal 30A and presses the button to add a device. The reception unit 32 of the communication terminal 30A receives the press.

[0199] S24: The communication unit 31 of the communication terminal 30A requests the communication management system 50 to provide the devices registered with the tenant and the devices associated with the virtual room ID selected in step S23.

[0200] S25: The communication unit 51 of the communication management system 50 receives a request for devices registered with the tenant and devices associated with the virtual room ID, and the screen generation unit 52 generates a virtual room correspondence screen (third) 280 including the device IDs of the devices registered with the tenant and the devices associated with the virtual room ID. The communication unit 51 of the communication management system 50 transmits screen information of the virtual room correspondence screen (third) 280 to the communication terminal 30A.

[0201] S26: The communication unit 31 of the communication terminal 30A receives the screen information of the virtual room association screen (third) 280, and the display control unit 33 displays the virtual room association screen (third) 280. The user A selects the imaging device 10 to be associated with the virtual room. The reception unit 32 of the communication terminal 30A receives the selection, and the imaging device ID is identified.

[0202] S27: The communication unit 31 of the communication terminal 30A transmits an association request to the communication management system 50, specifying the virtual room ID selected in step S23 and the imaging device ID selected in S27.

[0203] S28: The communication unit 51 of the communication management system 50 receives the association request, and the communication group management unit 56 registers the imaging device 10 in the virtual room. That is, the communication group management unit 56 registers the imaging device ID in the virtual room information storage unit 5002.

[0204] S29: Now that the imaging device ID has been associated with the virtual room, the communication unit 51 of the communication management system 50 transmits the virtual room ID, name, and description to the imaging device 10. The communication management system 50 may use push notification, or the imaging device 10 may transmit the information by polling. The connection unit 16 of the imaging device 10 receives the virtual room ID, name, and description, and stores them in the storage unit 1000. As a result, when the imaging device 10 transmits a wide-field image, the imaging device ID, virtual room ID, name, description, etc. can be attached.

[0205] S30: The communication terminal 30A is notified of the completion of the association, and as a result, the user A turns on the toggle button 291 of the imaging device 10 associated with the virtual room on the image viewing screen 210. The reception unit 32 of the communication terminal 30A receives the on-signal.

[0206] S31: The communication unit 31 of the communication terminal 30A specifies the imaging device ID and sends a request to start transmitting the wide-field image to the communication management system 50. User A may also directly operate a button on the imaging device 10 to start transmitting the wide-field image. User A may also operate the communication unit 31 of the communication terminal 30A to send a request to stop transmission to the communication management system 50.

[0207] S32: The communication unit 51 of the communication management system 50 receives the transmission start request and requests the imaging device 10 identified by the imaging device ID to start transmission. The communication management system 50 may use push notification, or the imaging device 10 may use polling. The connection unit 16 of the imaging device 10 receives the transmission start request, and the imaging processing unit 13 begins capturing images. The image transmission control unit 18 repeatedly transmits wide-field images via the connection unit 16 at a determined FPS or an FPS according to the bandwidth. Therefore, the communication terminal 30 that has entered the virtual room can display the situation at site A on the image viewing screen 210 in real time.

[0208] <Distribution of wide-field images, etc.> The flow of sharing a wide-field image and an image with a normal angle of view will be described with reference to Fig. 24. Fig. 24 is an example of a sequence diagram illustrating the flow of sharing a wide-field image. In Fig. 24, communication terminals 30A and 30B have entered the virtual room. Furthermore, communication terminal 30A has an imaging device 9 with a normal angle of view, which is shared with communication terminal 30B. Images captured by smart glasses 88 associated with the virtual room, instead of the imaging device 9 of communication terminal 30A, may also be shared.

[0209] S41: The imaging unit 34 of the communication terminal 30A repeatedly captures images, and the communication unit 31 designates the ID of the virtual room that the user has entered and repeatedly transmits the images and audio to the communication management system 50.

[0210] S42, S43: When the communication unit 51 of the communication management system 50 receives the image and audio, the image distribution unit 54 acquires the IP addresses of the communication terminals 30A and 30B that have entered the virtual room from the virtual room information storage unit 5002, and transmits the image and audio via the communication unit 51. Note that in Fig. 24, the communication unit 31 of the communication terminal 30A receives and displays an image with a normal angle of view from the communication management system 50, but the image with a normal angle of view captured by the imaging unit 34 may be displayed without receiving it.

[0211] S44: Next, in response to a transmission start request made by the imaging device 10 turning on the toggle button 291, the imaging processing unit 13 repeatedly captures wide-field images, and the image transmission control unit 18 repeatedly transmits the wide-field images and audio to the communication management system 50 by specifying the virtual room ID, imaging device ID, name, and description via the connection unit 16.

[0212] S45, S46: When the communication unit 51 of the communication management system 50 receives the wide-field image and audio, the image distribution unit 54 obtains the IP addresses of the communication terminals 30A and 30B that have entered the virtual room from the virtual room information storage unit 5002, and transmits the wide-field image and audio via the communication unit 51.

[0213] S47: Next, a communication terminal 30C equipped with an imaging device 9 newly enters the virtual room.

[0214] S48: The communication unit 31 of the communication terminal 30C transmits to the communication management system 50 an image and sound with a normal angle of view.

[0215] S49-S51: The communication unit 51 of the communication management system 50 receives images and audio with a normal angle of view from the communication terminal 30C, obtains the IP addresses of the communication terminals 30A-30C that have entered the virtual room from the virtual room information storage unit 5002, and the image distribution unit 54 transmits images and audio with a normal angle of view.

[0216] S52: The communication unit 51 of the communication management system 50 also transmits the wide-field image and audio to the communication terminal 30C that has entered the same virtual room.

[0217] In this way, users A and B who enter the same virtual room can share wide-field images captured by the imaging device 10 associated with the virtual room. Note that the order of transmission of the images shown in Fig. 24 is an example, and the wide-field image may be shared first, or the normal angle of view image may be shared first.

[0218] A few more details about the smart glasses 88 and the VR goggles 89: The smart glasses 88 have a camera with a normal angle of view and a display function. Images with a normal angle of view captured by the camera held by the smart glasses 88 are distributed in the same manner as the imaging devices 8 and 9. The display function held by the smart glasses 88 is flat like a normal display, so a portion of a wide-field-of-view image is displayed at a viewpoint specified by the user. The VR goggles 89 have a display function (and may also have a camera with a normal angle of view). The display function held by the smart glasses 88 projects a wide-field-of-view image with a viewpoint determined by the orientation of the user's head, so a portion of the wide-field-of-view image is displayed at a viewpoint corresponding to the orientation of the user's head. While viewing a wide-field-of-view image with the smart glasses 88 or the VR goggles 89, the user can send an imaging request specifying the viewpoint information being viewed to the communication management system 50.

[0219] <An example of a chat screen> In the following, an example of processing video and audio that is performed by the communication management system 50 based on a message entered by a user on the chat screen 400 and the communication terminal 30 based on the control data will be described.

[0220] <<Human Masking Processing>> FIG. 25 is a diagram illustrating the process of masking a person using a chat screen 400 displayed by the communication terminal 30. In FIG. 25, it is assumed that the communication terminal 30 is displaying video of a construction site. The chat screen 400 has a video field 401, a participant image 403, and a chat field 402. Video from the construction site is displayed in real time in the video field 401. In FIG. 25, two videos are displayed in the video field 401, but these may be captured by two imaging devices 10 or displayed by participants in the online conference. The participant image 403 is a video of the location where the participants of the online conference who are currently participating in the virtual room are located. The chat field 402 is a field in which chat content (messages entered by the user and responses from the chatbot) flows from top to bottom.

[0221] In Figure 25(a), two people 404 and 405 are shown. In this situation, when a user of a communication terminal 30 is viewing video of a construction site from a remote location, the user wants to hide the people 404 and 405 in the video to protect the privacy of the people 404 and 405. As shown in Figure 25(a), the user addresses the message to the chatbot (for example, by typing "chatbot") and enters a message 411 into the input field 410 into the communication terminal 30 saying, "I want to hide the people in the video."

[0222] The communication unit 51 of the communication management system 50 receives the message 411. The chat question analysis unit 62 detects that the message 411 is addressed to a chatbot and analyzes the message 411. The chat question analysis unit 62 determines that the message 411 is a request related to video and sets a control mode based on the analysis result of the message 411. The chat question analysis unit 62 requests a person masking process as the control mode from the video analysis unit 63. The video analysis unit 63 detects a person appearing in the video and returns coordinates indicating the person's area (hereinafter referred to as person coordinates) to the chat question analysis unit 62. The chat question analysis unit 62 uses the person coordinates to generate control data (an example of information related to the video data) instructing "to hide the display of the person area (rectangular area)." Furthermore, the chat question response unit 61 creates a response sentence 412 to the message 411 saying, "The person appearing in the video has been hidden."

[0223] The communication unit 51 of the communication management system 50 transmits the control data, the video frame, and the response message to the communication terminal 30. The communication unit 31 of the communication terminal 30 receives the control data, the video frame, and the response message, and the control data analysis unit 38 analyzes the control data. As a result of the analysis, the control data analysis unit 38 requests the video display unit 39 to perform image processing to mask the people 404 and 405 in the video frame. The video display unit 39 performs this processing on the video frame, and displays the video frame on the display 506. As shown in FIG. 25(b), the people 404 and 405 that were displayed in FIG. 25(a) are now masked by rectangles 406 and 407. In addition, the chat display unit 41 displays the response message 412 in the chat field 402.

[0224] In this way, the communication terminal 30 can perform real-time person masking processing on the video data captured by the imaging device 10 based on the chat message entered by the user in the input field 410. By feeding back the question sent by the user to the chatbot onto the video being viewed, the viewing experience is not interrupted. Furthermore, the user can accurately convey their request to the communication management system 50 through chat (linguistic information).

[0225] Conventional chatbots only respond in text, which is less intuitive than video because it can convey a limited amount of information. In this embodiment, chat instructions are reflected in the video, so users don't have to go out of their way to check the chat box.

[0226] <<Focus on the specified person>> Next, a display example of a video displayed by communication terminal 30 with the viewpoint aligned with a person designated based on control data will be described with reference to Fig. 26. Fig. 26 is a diagram illustrating the process of aligning the viewpoint with a person using chat screen 400 displayed by communication terminal 30.

[0227] Figure 26(a) shows two people 404 and 405. In this situation, a user of a communication terminal 30 is viewing video of a construction site from a remote location and wants to check the work of person 405 who appears in the video. As shown in Figure 26(a), the user enters a message 421 into the communication terminal 30 in an input field 410 addressed to the chatbot, saying, "I want the view to be fixed on worker A."

[0228] The communication unit 51 of the communication management system 50 receives the message 421. The chat question analysis unit 62 detects that the message 421 is addressed to a chatbot and analyzes the message 421. The chat question analysis unit 62 determines that the message 421 is a request related to video, and sets a control mode based on the analysis result of the message.

[0229] The chat question analysis unit 62 requests the video analysis unit 63 to perform person tracking processing as a control mode. The video analysis unit 63 finds worker A by detecting people in the video and identifying them through facial recognition. If the video analysis unit 63 cannot recognize the faces of people 404 and 405, for example, because they are facing away from each other, the video analysis unit 63 may enlarge both of the two candidate people 404 and 405 and display them on the communication terminal 30. The user can select person 404 or 405 for which they want to perform person tracking processing.

[0230] The video analysis unit 63 transmits the coordinates indicating the area of ​​worker A to the chat question analysis unit 62. The chat question analysis unit 62 generates control data (an example of information related to the video data) that instructs "moving the viewpoint to the person area of ​​worker A" using the coordinates indicating the area of ​​worker A. In addition, the chat question response unit 61 creates a response sentence 422 to the message 421, saying "The viewpoint has been adjusted to worker A."

[0231] The communication unit 51 of the communication management system 50 transmits the control data, the video frame, and the response message to the communication terminal 30. The communication unit 31 of the communication terminal 30 receives the control data, the video frame, and the response message, and the control data analysis unit 38 analyzes the control data. As a result of the analysis, the control data analysis unit 38 requests the video display unit 39 to perform image processing to enlarge the person area of ​​worker A in the video frame, with worker A as the center of the viewpoint. The video display unit 39 performs this processing and displays the video frame on the display 506. As shown in FIG. 26(b), the person area 424 of worker A is enlarged and displayed. In addition, the chat display unit 41 displays the response message 422 in the chat field 402.

[0232] <<Display Documents>> Next, a display example of materials displayed by the communication terminal 30 based on the control data will be described with reference to Fig. 27. Fig. 27 is a diagram illustrating a process of displaying materials using the chat screen 400 displayed by the communication terminal 30.

[0233] In this way, we wanted to increase knowledge about the construction site while the user of the communication terminal 30 was viewing the video of the site from a remote location. As shown in Figure 27(a), the user entered a message 431 into the communication terminal 30 in the input field 410, addressed to the chatbot, stating, "I would like to know if any accident reports have been submitted at site A in the past."

[0234] The communication unit 51 of the communication management system 50 receives the message 431. The chat question analysis unit 62 detects that the message 431 is addressed to a chatbot and analyzes the message 431. The chat question analysis unit 62 determines that the message 431 is a request for materials, and sets a control mode for material display based on the analysis result of the message 431. Since material display only needs to be performed once, the setting for the material display control mode is deleted after the search.

[0235] Since the control mode is document display, the chat question analysis unit 62 searches for the relevant document in the document storage unit 5102. The chat question analysis unit 62 acquires the document URL from the document storage unit 5102 and includes the document URL in the control data (an example of information related to the storage destination of the document). Furthermore, the chat question response unit 61 creates a response sentence 432 to the message 431 saying, "An accident report at site A has been found. It will be displayed." Note that the search does not have to be performed by the chat question analysis unit 62, and a process for document search may be registered in the communication management system 50 separately from the chat question analysis unit 62.

[0236] The communication unit 51 of the communication management system 50 transmits the control data, the video frame, and the response message to the communication terminal 30. The communication unit 31 of the communication terminal 30 receives the control data, the video frame, and the response message, and the control data analysis unit 38 analyzes the control data. As a result of the analysis, the control data analysis unit 38 requests the video display unit 39 to process displaying the material at the material URL. The video display unit 39 connects to the material URL, acquires the material, and displays the material 434 on the display 506. As shown in FIG. 27(b), the material 434 is additionally displayed in the video column 401. In addition, the chat display unit 41 displays the response message 432 in the chat column 402.

[0237] In this way, the communication terminal 30 can display materials in real time on the video data captured by the imaging device 10 based on chat messages entered by the user in the input field 410. By feeding back the question text sent by the user to the chatbot onto the video being viewed, the viewing experience is not interrupted. Furthermore, the user can accurately convey their requests to the communication management system 50 through chat (linguistic information).

[0238] Conventional chatbots only return responses in text, which limits the amount of information they can provide and makes it difficult to understand intuitively. In this embodiment, documents are displayed separately from the video in response to chat instructions, allowing users to view both the documents and the video.

[0239] The object of the viewpoint tracking is not limited to a person, but may be a building, heavy machinery, or the directions of north, south, east, or west.

[0240] <<Alarm sounding process>> Next, an example of audio output when communication terminal 30 sounds an alarm based on control data will be described with reference to Fig. 28. Fig. 28 is a diagram illustrating a process of sounding an alarm using chat screen 400 displayed by communication terminal 30.

[0241] A user of a communication terminal 30 is viewing video footage of a construction site from a remote location, and wants to know if there are any unusual noises coming from heavy machinery. Because the site is noisy and there is a possibility that a human ear might miss the noise, we decided to have the system make the judgment. As shown in Figure 28(a), the user enters a message 441 into the communication terminal 30 in the input field 410, addressing the chatbot and stating, "If there are any unusual noises from heavy machinery, please sound an alarm."

[0242] The communication unit 51 of the communication management system 50 receives the message 441. The chat question analysis unit 62 detects that the message 441 is addressed to a chatbot and analyzes the message 441. The chat question analysis unit 62 determines that the message 441 is a request related to voice, and sets a control mode based on the analysis result of the message 441.

[0243] The chat question analysis unit 62 requests the voice analysis unit 64 to perform abnormal sound detection processing as a control mode. The voice analysis unit 64 monitors the latest voice frame, and if an abnormal sound occurs, notifies the chat question analysis unit 62. Upon receiving the notification, the chat question analysis unit 62 generates control data (an example of information related to voice output) instructing "to sound an alarm." The chat question analysis unit 62 does not generate control data for sounding an alarm while no abnormal sound occurs. Furthermore, the chat question response unit 61 creates a response sentence 442 to the message 441 saying, "I have set it to sound an alarm if an abnormal sound occurs."

[0244] The communication unit 51 of the communication management system 50 transmits control data (if an alarm is to be sounded), a video frame, and a response message to the communication terminal 30. The communication unit 31 of the communication terminal 30 receives the control data, the video frame, and the response message, and the control data analysis unit 38 analyzes the control data. As a result of the analysis, the control data analysis unit 38 requests the audio playback unit 40 to sound an alarm. The audio playback unit 40 outputs the alarm from the speaker and also plays the audio frame. The control data analysis unit 38 may cause the chat display unit 41 to display a message such as "An abnormal sound detection alarm has been sounded." In other words, the audio playback unit 40 may output audio that is not processed audio data. The chat display unit 41 also displays a response message 442 in the chat field 402.

[0245] In this way, the communication terminal 30 can perform abnormal sound detection processing in real time from the audio data acquired by the imaging device 10, based on the chat message entered by the user in the input field 410. The user sends a question to the chatbot, and the question is fed back into the audio data being viewed, so the viewing experience is not interrupted. Furthermore, the user can accurately convey their request to the communication management system 50 through chat (linguistic information).

[0246] Conventional chatbots only return responses in text, which is less intuitive than voice because the amount of information that can be conveyed is limited. In this embodiment, chat instructions are reflected in voice, so users do not have to go out of their way to check the chat box.

[0247] <<Noise Removal>> Next, an example in which communication terminal 30 outputs audio from which noise has been removed based on control data will be described with reference to Fig. 29. Fig. 29 is a diagram illustrating the process of removing noise from audio data using chat screen 400 displayed by communication terminal 30.

[0248] A user of a communication terminal 30 is viewing video of a construction site from a remote location and wants to remove noise from the audio data. As shown in Figure 29(a), the user enters a message 471 into the communication terminal 30 in the input field 410, addressing the message to the chatbot, saying, "Please remove noise from the audio data."

[0249] The communication unit 51 of the communication management system 50 receives the message 471. The chat question analysis unit 62 detects that the message 471 is addressed to a chatbot and analyzes the message 471. The chat question analysis unit 62 determines that the message 471 is a request related to voice, and sets a control mode based on the analysis result of the message 471.

[0250] The chat question analysis unit 62 requests the voice analysis unit 64 to perform noise canceling processing as a control mode. The voice analysis unit 64 performs noise canceling processing on the latest voice frame and notifies the chat question analysis unit 62 of the result. The chat question analysis unit 62 generates control data (an example of information about the voice data) indicating that "noise canceling processing has been performed." Furthermore, the chat question response unit 61 creates a response sentence 442 to the message 471 saying, "Noise has been removed from the voice data."

[0251] The communication unit 51 of the communication management system 50 transmits the control data, the video frame, and the response message to the communication terminal 30. The communication unit 31 of the communication terminal 30 receives the control data, the video frame, and the response message, and the control data analysis unit 38 analyzes the control data. As a result of the analysis, the control data analysis unit 38 determines that "noise canceling processing has been performed," so it does not request the audio playback unit 40 to remove noise. However, it is preferable to display an icon or the like indicating that "noise canceling processing has been performed" on the video display unit 39. The audio playback unit 40 plays the audio data from which the noise has been removed. Furthermore, as shown in FIG. 29(b), the chat display unit 41 displays the response message 472 in the chat field 402.

[0252] <Control mode and control data> The control modes and control data will be described in detail with reference to Figs. 30 and 31. Fig. 30 is a diagram showing an example of a control mode. The control mode indicates the content of the analysis process executed by the communication management system 50. Once a control mode is set, analysis process is executed for each video frame or audio frame, and control data is created for each video frame or audio frame. Once a control mode is set, it continues until a request to end the control mode is sent by message from the communication terminal 30. Fig. 30 shows "person masking, abnormal sound detection, document display" as an example of a control mode. The number of control modes that can be set simultaneously is not limited to one, and multiple control modes may be set simultaneously.

[0253] FIG. 31 is a diagram showing an example of control data in JSON format. The control data may be in XML format or CSV format. FIG. 31(a) is control data in which the communication terminal 30 performs person masking processing, FIG. 31(b) is control data in which the communication terminal 30 performs abnormal sound detection (the abnormal sound detection itself is performed by the communication management system 50), and FIG. 31(c) is control data in which the communication terminal 30 displays materials. The control data in FIG. 31(a) includes a timestamp 451 and two person coordinates 452 (coordinates of the detected person area). The timestamp 451 is the date and time when the person was detected. The number of person coordinates 452 depends on the number of people detected from the video frame.

[0254] The control data in Figure 31(b) includes a timestamp 453 and a status 454. The timestamp 453 is the date and time when the abnormal sound was detected. The status 454 indicates whether or not the abnormal sound was detected. The control data in Figure 31(c) includes a URL 456. The URL 456 is the destination where the material is saved.

[0255] FIG. 32 is a diagram illustrating person coordinates when the control data is person masking. In FIG. 32, four people are detected and surrounded by rectangular frames 461 to 464. The person coordinates of one of the people are specified by "left, top (upper left corner)" and "right, bottom (lower right corner)." Communication terminal 30 receives the control data and performs a masking process on the coordinates of the person area in the video frame. Furthermore, if the control data is abnormal sound detection and status 454 indicates that an abnormal sound has been detected, communication terminal 30 sounds an alarm. If the control data is material display, communication terminal 30 accesses the URL of the material and displays the material in video column 401.

[0256] <Action or Processing> Next, the operation or processing of the communication management system 50 and the communication terminal 30 will be described with reference to FIGS.

[0257] <<Overall flow>> 33 is a flowchart illustrating the process in which the communication management system 50 sets the control mode and control data in response to a message from the communication terminal 30. The process in FIG. 33 assumes that the communication terminal 30 has entered a virtual room and is viewing a video. Furthermore, no control mode has yet been set.

[0258] The communication unit of the communication management system 50 waits for a message from the communication terminal 30 (S101). The process branches into four depending on whether the communication unit 51 receives a message and the contents of the message.

[0259] (i) If the communication unit 51 does not receive a message, the chat question analysis unit 62 determines whether the control mode is set (S102). If the control mode is set, the process proceeds to step S104. If the control mode is not set, the process proceeds to step S105.

[0260] (ii) When the communication unit 51 receives a message addressed to the chatbot, the chat question analysis unit 62 analyzes the message (S106). Then, the chat question response unit 61 creates a response sentence for the chatbot (S107).

[0261] If the message is analyzed to be a request for video, audio, or materials, the chat question analysis unit 62 sets the control mode to "person masking," "abnormal sound detection," or "material display" (S103).

[0262] The chat question analysis unit 62 creates control data using the analysis results performed by the video analysis unit 63 or the audio analysis unit 64, or the processing results (URL) performed by the chat question analysis unit 62, depending on the control mode (S104).

[0263] The communication unit 51 of the communication management system 50 transmits the video frame (which may also include an audio frame), the control data, and the response message to the communication terminal 30 (S105). Note that if the control mode is not set, the control data is not transmitted to the communication terminal 30.

[0264] (iii) If the communication unit 51 receives a message requesting termination of the control mode, the chat question analysis unit 62 deletes the control mode setting (S108). In this way, the control mode can be terminated by a request to terminate the control mode being sent from the communication terminal 30. However, for the control mode of displaying materials, the control mode setting may be deleted after one display of materials. A request to terminate the control mode is made by the user inputting a message such as "Please terminate the person masking process" or "Please terminate the current control mode" into the input field 410.

[0265] (iv) When the communication unit 51 receives a message requesting the termination of the online conference, the connection management unit 58 terminates the online conference. In this way, an online conference can also be terminated by a request to terminate the online conference being sent from the communication terminal 30. The request to terminate the online conference is made by the user inputting a message such as "End the conference" or "End communication" into the input field 410. Ending the online conference stops the transmission of image data from the communication terminal 30 and the transmission of image data from the participant's location. Note that the user may terminate only the transmission of image data from the communication terminal 30 while continuing the online conference, or may terminate the online conference while continuing the transmission of image data from the communication terminal 30.

[0266] <<Generating control data>> Fig. 34 is a flowchart illustrating the flow of processing for creating control data by the communication management system 50. The processing in Fig. 34 is performed when the communication management system 50 receives a message addressed to a chatbot.

[0267] First, the chat question analysis unit 62 breaks down the message into words and performs syntax analysis using a general natural language processing technique (S201). Below, we will explain the process of extracting requests related to video, audio, and materials separately.

[0268] The chat question analysis unit 62 performed a process of extracting a request related to the video (S211). The chat question analysis unit 62 determines a video analysis process (person masking process, person tracking process, etc.) corresponding to the request related to the video (S212).

[0269] The chat question analysis unit 62 sets the control mode in accordance with the determined video analysis process (S213). The video analysis unit 63 executes the video analysis process for each video frame in accordance with the control mode (S214).

[0270] The chat question analysis unit 62 acquires the analysis results (such as person coordinates) from the video analysis unit 63 (S215). The chat question analysis unit 62 generates control data including the analysis results (S216).

[0271] Next, in step S221, the chat question analysis unit 62 performs a voice-related request extraction process (S221). The chat question analysis unit 62 determines a voice analysis process (abnormal sound detection process, noise canceling process, etc.) corresponding to the voice-related request (S222).

[0272] The chat question analysis unit 62 sets the control mode in accordance with the determined voice analysis process (S223). The voice analysis unit 64 executes the voice analysis process for each voice frame in accordance with the control mode (S224).

[0273] The chat question analysis unit 62 acquires the analysis result (such as sounding an alarm) from the voice analysis unit 64 (S225). The chat question analysis unit 62 generates control data including the analysis result (S226).

[0274] Next, in step S231, the chat question analysis unit 62 performs a request extraction process for materials (S231).The chat question analysis unit 62 searches the material storage unit 5102 for materials that match the request (S232).

[0275] The chat question analysis unit 62 sets the document display to the control mode (S233). The chat question analysis unit 62 acquires the URL of the document based on the search result of step S232 (S234). The chat question analysis unit 62 generates control data including the analysis result (S235). Note that once the chat question analysis unit 62 acquires the URL of the document, it deletes the control mode setting.

[0276] Since the control data is generated in steps S216, S226, and S235, the control data, video frames, audio frames, and response sentence are transmitted to the communication terminal 30.

[0277] <<Adding or deleting video analysis processing or audio analysis processing>> 35 is a flowchart illustrating the process by which a system administrator adds or deletes video analysis processing or audio analysis processing to the communication management system 50. The system administrator connects the communication terminal 30 to the communication management system 50 and displays the management page.

[0278] The communication unit 51 of the communication management system 50 transmits screen information of the registered process list screen generated by the screen generation unit 52 to the communication terminal 30. The communication unit 31 of the communication terminal 30 receives the screen information, and the display control unit displays the registered process list screen on the display 506 (S301).

[0279] The system administrator selects an operation on the registered process list screen (S302). When the system administrator performs an operation to add video analysis process or audio analysis process, the communication unit 31 of the communication terminal 30 uploads a process definition file to the communication management system 50 (S303). When adding video analysis process or audio analysis process, the user prepares a process definition file in advance. The process definition file describes an algorithm that indicates the process content, related information (such as keywords) for matching with requests, and content to be output to the control data.

[0280] The communication unit 51 of the communication management system 50 receives the new process definition file and registers the video analysis process or audio analysis process based on the process definition file in the video analysis unit 63 or audio analysis unit 64 (S304).

[0281] When the system administrator performs an operation to delete a video analysis process or an audio analysis process on the registered process list screen, the reception unit 32 receives the selection of the video analysis process or the audio analysis process to be deleted (S305). The communication unit 31 of the communication terminal 30 transmits identification information of the video analysis process or the audio analysis process to be deleted to the communication management system 50.

[0282] The communication unit 51 of the communication management system 50 receives the identification information of the video analysis process or audio analysis process to be deleted, and deletes the video analysis process or audio analysis process instructed by the video analysis unit 63 or audio analysis unit 64 (S306).

[0283] When the system administrator performs an operation to terminate the addition or deletion of a registered process, the process of FIG. 35 ends.

[0284] <<Communication terminal processing>> Fig. 36 is a flowchart illustrating the processing executed by communication terminal 30. Fig. 36 shows what processing communication terminal 30 performs based on user operations input to communication terminal 30 and information received by communication unit 31. The processing in Fig. 36 starts when communication terminal 30 enters a virtual room and starts viewing video.

[0285] The chat input receiving unit 37 waits for a message input from a user for chat (S401).

[0286] (i) If no message is input, the communication unit 31 determines whether or not control data has been received from the communication management system 50 (S403).

[0287] When control data is received, the control data analysis unit 38 analyzes the control data, and the video display unit 39 processes the video frames in accordance with the control data, or the audio playback unit 40 processes the audio frames in accordance with the control data (S404). Alternatively, the audio playback unit 40 outputs audio, such as an alarm, that is separate from the audio frames.

[0288] Next, it is determined whether the communication unit 31 has received a response from the chatbot (S405). If not, the process returns to step S401.

[0289] When the communication unit 31 receives a response message from the chatbot, the chat display unit 41 displays the response message in the chat field 402 (S406).

[0290] (ii) If a message is input and it is a question addressed to the chatbot, the communication unit 31 sends the question addressed to the chatbot to the communication management system 50 (S402). After this, the process proceeds to step S403, and the subsequent processes are the same as those described above.

[0291] (iii) If a message is input and it is a request to terminate the control data, the communication unit 31 sends a request to terminate the control mode (for example, a message requesting that the processing of XX be terminated) to the communication management system 50 (S407). As a result, the analysis processing that had been set up until then will no longer be performed (and the control data will no longer be sent to the communication terminal 30). The process returns to step S401.

[0292] (iv) If a message is input and is a request to end the conference, the communication unit 31 transmits a request to end the online conference to the communication management system 50 (S408). In this case, the processing of FIG. 36 ends.

[0293] <Examples of application of communication systems in telemedicine> FIG. 37 is a diagram illustrating an example of remote communication in which the communication system 1 is applied to telemedicine. The explanation of FIG. 37 will mainly focus on the differences from FIG. 1. Site A in FIG. 37 is an operating room, but the process flow of (1) to (6) may be the same as in FIG. 1. In FIG. 37, a patient is placed on an operating table 355 and undergoes surgery by a medical professional such as a doctor. The medical professional (corresponding to a user) operates on the patient using various surgical tools 354, such as forceps and a scalpel. The medical professional can wear smart glasses 88, which can transmit images of the medical professional's surgical field to a communication network N. Various cameras, such as an operating room camera 351, an operating field camera 352, and an endoscope 353, are installed in the operating room. All of the cameras and smart glasses 88 in the operating room are associated with virtual rooms.

[0294] A main unit 356 is placed in the operating room to monitor the patient's vital signs, the operating status of medical equipment, and the like. The main unit 356 corresponds to the communication terminal 30 of this embodiment. The communication terminal 30 (main unit 356) in the operating room may also have the function of receiving images from the endoscope 353 and the surgical field camera 352 in addition to the functions shown in FIG. 1. The communication terminal 30 can display the received images on the display 306 and can also transmit them to the communication management system 50 as images from the base of the communication terminal 30. The operation panel 357 is an input interface that accepts various operations, and medical personnel may be able to operate the equipment in the operating room via the operation panel 357. Furthermore, the endoscope 353, the surgical field camera 351, and the surgical field camera 352 may communicate directly with the communication management system 50 without going through the communication terminal 30.

[0295] The communication terminal 30 may also have the functionality of an electronic medical record system, or may have the functionality to communicate with the electronic medical record system. The communication terminal 30 may display electronic medical record information on the display 306.

[0296] 38 is a diagram showing an example of a virtual room association screen (part 1) 360 for associating an imaging device with a virtual room in the case of remote medical care. In the explanation of FIG. 38, differences from FIG. 19 will be mainly explained.

[0297] In the case of telemedicine, a list of virtual rooms 361 associated with, for example, surgeries or medical examinations performed remotely is displayed on the virtual room association screen (part 1) 360. Medical cameras 362 including spherical cameras are associated with the virtual rooms. The medical cameras 362 include endoscopes, surgical field cameras used to capture surgical field images in the operating room, cameras that capture microscopic images, and the like.

[0298] <Major Effects> The communication terminal 30 of this embodiment can process the video data or audio data captured by the imaging device 10 in real time using chat messages entered by the user in the input field 410. The user sends a question to the chatbot, and the question is fed back to the video or audio being viewed, so the viewing experience is not interrupted. Furthermore, the user can accurately convey their requests to the communication management system 50 through chat (linguistic information).

[0299] Conventional chatbots only respond in text, which is less intuitive than video and audio because the amount of information that can be conveyed is limited. In this embodiment, chat instructions are reflected in video and audio, so users do not have to go out of their way to check the chat box.

[0300] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0301] For example, in this embodiment, the communication terminal 30 receives control data from the communication management system 50 and processes the video data or audio data. However, this processing may be performed by the communication management system 50, and the processed video data or audio data may be transmitted to the communication terminal 30. Alternatively, the communication management system 50 may acquire materials and transmit them to the communication terminal 30.

[0302] Furthermore, in this embodiment, the communication management system 50 has a chat function and an online conference function, but there may be a server device that has both a chat function and an online conference function.

[0303] Furthermore, the processing performed on video data is not limited to person masking processing and person tracking processing, but may include various other processing such as overall masking processing, overall high resolution, video data reduction, moving object detection, abnormal behavior detection such as theft and vandalism, disaster detection such as injury and accident, weather detection, etc. The processing performed on audio data is not limited to abnormal sound detection and noise canceling processing, but may include various other processing such as detection of specific linguistic information (such as danger), voice recognition processing, etc.

[0304] Furthermore, in the present embodiment, an example has been described in which the image capture device 10 processes the video data and audio data. However, the communication management system 50 may generate control data for processing the video data and audio data from the participant's location. In this case, the user can simply input a message to the chatbot, such as "I want to hide the person appearing in the video from Mr. / Ms. XX's location." Furthermore, the communication management system 50 or the communication terminal 30 may perform the processing.

[0305] 13 and other configuration examples are divided according to main functions to facilitate understanding of the processing by the communication management system 50, the imaging device 10, and the communication terminal 30. The method of dividing the processing units and their names do not limit the present invention. The processing by the communication management system 50, the imaging device 10, and the communication terminal 30 can be divided into even more processing units depending on the processing content. Furthermore, the processing can also be divided so that one processing unit includes even more processes.

[0306] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above.

[0307] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, communication management system 50 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0308] Furthermore, the communication management system 50 may be integrated into one server device, or may be divided into multiple devices.

[0309] <Additional Notes> [Appendix 1] A communication terminal having a chat function and capable of communicating via a network with a communication management system that receives video data from an imaging device, a chat input receiving unit that receives input of a chat message; a communication unit that transmits the message to the communication management system and receives information about the video data based on the message and the video data from the communication management system; a video display unit that displays the video data processed using information about the video data; A communication terminal having the above configuration. [Appendix 2] the communication management system analyzes the video data based on the message to generate information about the video data; The communication terminal according to claim 1, wherein the video display unit processes part or all of the video data based on information about the video data and displays the processed video data. [Appendix 3] the communication management system receives audio data from the imaging device; the communication management system processes the voice data based on the message to generate information about the voice data; the communication unit receives information about the voice data based on the message and the voice data from the communication management system; a voice playback unit that plays back the voice data processed by the communication management system; 3. The communication terminal according to claim 1 or 2, [Appendix 4] the communication management system receives audio data from the imaging device; the communication management system analyzes the voice data based on the message to generate information related to voice output of the communication terminal; the communication unit receives information regarding the voice output of the communication terminal based on the message and the voice data from the communication management system; an audio playback unit that outputs audio based on the audio data and information about the audio output; 3. The communication terminal according to claim 1 or 2, [Appendix 5] The communication management system analyzes the message to generate information regarding a storage location of the material; the communication unit receives information regarding a storage location of the material based on the message from the communication management system; the video display unit displays the acquired material using information regarding the storage destination of the material; 5. A communication terminal according to any one of appendices 1 to 4. [Appendix 6] When the information on the video data indicates that a process for reducing visibility of a person is to be performed and the information indicates the coordinates of the person, the video display unit: 3. The communication terminal according to claim 2, wherein the visibility reduction processing is performed on an area of ​​the person coordinates in the video data, and the video data on which the person visibility reduction processing has been performed is displayed. [Appendix 7] When the information on the video data indicates that person tracking processing is to be performed and indicates the coordinates of the person, the video display unit: 3. The communication terminal according to claim 2, wherein the person tracking process is performed to enlarge an area of ​​the person coordinates in the video data, and the video data in which the person is enlarged is displayed. [Appendix 8] If the information about the audio data indicates that noise canceling processing has been performed, 4. The communication terminal according to claim 3, wherein the audio playback unit plays back the audio data that has been subjected to noise canceling processing. [Appendix 9] When the information regarding the audio output indicates that an abnormal sound has been detected, 5. The communication terminal according to claim 4, which outputs the audio data and an audio message informing the user that the abnormal sound has been detected. [Appendix 10] the communication management system receives audio data from the imaging device; the communication management system executes a registered video analysis process or audio analysis process to generate information about the video data, information about the audio data based on the message, or information about audio output of the communication terminal based on the message; a reception unit that receives a selection of a processing definition file related to the video analysis processing or the audio analysis processing, the communication unit transmits a registration request for the process definition file to the communication management system. A communication terminal according to any one of Supplementary notes 1 to 9. [Explanation of symbols]

[0310] 10. Imaging device 30 Communication terminal 50 Communication Management System [Prior art documents] [Patent documents]

[0311] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-504616

Claims

1. A communication terminal having a chat function and capable of communicating via a network with a communication management system that receives video data from an imaging device, a chat input receiving unit that receives input of a chat message; a communication unit that transmits the message to the communication management system and receives information about the video data based on the message and the video data from the communication management system; a video display unit that displays the video data processed using information about the video data; A communication terminal having the above configuration.

2. the communication management system analyzes the video data based on the message to generate information about the video data; The communication terminal according to claim 1 , wherein the video display unit processes a part or all of the video data based on information related to the video data, and displays the processed video data.

3. the communication management system receives audio data from the imaging device; the communication management system processes the voice data based on the message to generate information about the voice data; the communication unit receives information about the voice data based on the message and the voice data from the communication management system; a voice playback unit that plays back the voice data processed by the communication management system; 3. The communication terminal according to claim 1, further comprising:

4. the communication management system receives audio data from the imaging device; the communication management system analyzes the voice data based on the message to generate information related to voice output of the communication terminal; the communication unit receives information regarding the voice output of the communication terminal based on the message and the voice data from the communication management system; an audio playback unit that outputs audio based on the audio data and information about the audio output; 3. The communication terminal according to claim 1, further comprising:

5. The communication management system analyzes the message to generate information regarding a storage location of the material; the communication unit receives information regarding a storage location of the material based on the message from the communication management system; the video display unit displays the acquired material using information regarding the storage destination of the material; The communication terminal according to claim 1 .

6. When the information on the video data indicates that a process for reducing visibility of a person is to be performed and the information indicates the coordinates of the person, the video display unit: The communication terminal according to claim 2 , wherein the visibility reduction process is performed on an area of ​​the person coordinates in the video data, and the video data on which the visibility reduction process for the person has been performed is displayed.

7. When the information on the video data indicates that person tracking processing is to be performed and indicates the coordinates of the person, the video display unit: The communication terminal according to claim 2 , wherein the person tracking process is performed to enlarge an area of ​​the person coordinates in the video data, and the video data in which the person is enlarged is displayed.

8. If the information about the audio data indicates that noise canceling processing has been performed, The communication terminal according to claim 3 , wherein the audio playback unit plays back the audio data that has been subjected to noise canceling processing.

9. When the information regarding the audio output indicates that an abnormal sound has been detected, The communication terminal according to claim 4, wherein the communication terminal outputs the audio data and an audio signal informing the user that the abnormal sound has been detected.

10. the communication management system receives audio data from the imaging device; the communication management system executes a registered video analysis process or audio analysis process to generate information about the video data, information about the audio data based on the message, or information about audio output of the communication terminal based on the message; a reception unit that receives a selection of a processing definition file related to the video analysis processing or the audio analysis processing, the communication unit transmits a registration request for the process definition file to the communication management system. The communication terminal according to claim 1 .

11. A display method performed by a communication terminal that has a chat function and can communicate with a communication management system that receives video data from an imaging device via a network, comprising: A chat input receiving unit receives input of a chat message; a process in which a communication unit transmits the message to the communication management system, and receives information about the video data based on the message and the video data from the communication management system; a process in which a video display unit displays the video data processed using information related to the video data; Display method.

12. A communication terminal having a chat function and capable of communicating with a communication management system that receives video data from an imaging device via a network, a chat input receiving unit that receives input of a chat message; a communication unit that transmits the message to the communication management system and receives information about the video data based on the message and the video data from the communication management system; a video display unit that displays the video data processed using information about the video data; A program to function as a

13. A communication management system having a chat function and transmitting video data received from an imaging device to a communication terminal, a video analysis unit that analyzes chat messages received from the communication terminal; a chat question analysis unit that generates information about the video data using the analysis result by the video analysis unit; a communication unit that transmits information about the video data and the video data to the communication terminal; A communication management system having:

Images