Terminal, reproduction method, and program

Abstract

To provide a technique for displaying record information of remote communication on the basis of screen information displayed by an application under execution and image information around a device.SOLUTION: A record information creation system including an information recording application executed in a terminal, wherein the information recording application includes a record video display section that displays a record video created based on screen information displayed by a teleconference application selected in the information recording application and a surrounding image acquired by a device, and the record video display section displays the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a recorded information display system, a program, and a recorded information display method. [Background technology]

[0002] 2. Description of the Related Art A remote conference system is known in which images and audio are transmitted in real time from one location to one or more other locations, and users at remote locations can hold a conference using images and audio.

[0003] There is also known a technology for displaying a meeting record screen based on information obtained by photographing and collecting sound from a 360-degree surrounding area (see, for example, Patent Document 1). Patent Document 1 discloses a meeting record screen that is composed of an area that displays participants in a conference room and a speech history area that displays each speech in chronological order, based on information obtained by photographing and collecting sound from a 360-degree surrounding area. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies have a problem in that they do not display recorded information of remote communication based on the screen information displayed by the running application and the surrounding image information. That is, conventional technologies can display information captured by images of the inside of a conference room, but do not display it together with the screen of an application such as a remote conference application. As a result, it is not possible to view the state of remote communication while capturing images of the inside of the conference room.

[0005] In view of the above-mentioned problems, the present invention aims to provide a technology for displaying recorded information of remote communication based on screen information displayed by a running application and image information of the device's surroundings. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a recorded information display system including an information recording application executed on a terminal device, wherein the information recording application has a recorded video display unit that displays a recorded video created based on screen information displayed by a remote conferencing application selected by the information recording application and surrounding image information acquired by the device, and the recorded video display unit displays the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information. [Effects of the Invention]

[0007] It is possible to provide a record information display system that displays record information of remote communication based on screen information displayed by a running application and image information of the device's surroundings, and allows users to view the state of remote communication while capturing images of the inside of a conference room. [Brief explanation of the drawings]

[0008] [Figure 1] 10A and 10B are diagrams illustrating an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of a recorded information display system. [Figure 3] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing system and a terminal device. [Figure 4] FIG. 2 is a diagram illustrating an example of the hardware configuration of a meeting device. [Figure 5] FIG. 10 is a diagram illustrating the imaging range of the meeting device. [Figure 6] 10A and 10B are diagrams illustrating the cutting out of a panoramic image and a speaker image. [Figure 7] FIG. 1 is an example of a functional block diagram illustrating the functions of a terminal device, a meeting device, and an information processing system in a recorded information display system, divided into blocks. [Figure 8] 4 is a diagram showing an example of moving image recording information stored in an information storage unit. FIG. [Figure 9]10 is an example of application-side conference information stored in an application-side conference information storage unit. [Figure 10] FIG. 2 is a diagram showing recording information stored in a recording information storage unit. [Figure 11] 10 is an example of side conference information stored in a conference information storage unit. [Figure 12] FIG. 2 is a diagram illustrating the structure of text data stored in the storage service system. [Figure 13] FIG. 10 is a diagram showing an example of an initial screen displayed by an information recording application running on the terminal device after login. [Figure 14] 10A and 10B are diagrams illustrating an operation method when the fixed display button is on. [Figure 15] FIG. 10 is a diagram illustrating an example of an unrecognized device screen. [Figure 16] FIG. 10 is a diagram showing an example of a recording setting screen displayed by an information recording application. [Figure 17] FIG. 10 is a diagram showing an example of the display of the recorded content confirmation window when the camera toggle button is on and the PC screen toggle button is off. [Figure 18] FIG. 10 is a diagram showing an example of the display of the recorded content confirmation window when the camera toggle button is on and the PC screen toggle button is on. [Figure 19] FIG. 10 is a diagram showing an example of a recording screen displayed by the information recording application during recording. [Figure 20] FIG. 10 is a diagram illustrating an example of a conference list screen displayed by the information recording application. [Figure 21] FIG. 10 is a diagram showing an example of a recorded video display screen after a recorded video is selected and displayed by the information recording application. [Figure 22] FIG. 10 is a diagram showing an example of an editing screen for a recorded video. [Figure 23] 10 is a sequence diagram illustrating an example of a procedure in which the information recording application records a panoramic image, a speaker image, and an application screen. [Figure 24] 10 is a sequence diagram illustrating an example of a process in which an information recording application downloads and displays a recorded video. [Figure 25]FIG. 2 is a diagram showing an example of a recorded moving image display screen 240 on which a panoramic image and a speaker image are displayed. [Figure 26] FIG. 10 is a diagram showing an example of a recorded video display screen on which a speaker image and an application screen are displayed when the Video & Content button is pressed. [Figure 27] FIG. 10 is a diagram showing an example of a recorded video display screen on which an application screen is displayed when a content button is pressed. [Figure 28] 10A and 10B are diagrams showing recorded videos displayed in an image display field that are switched in response to an operation on a recorded video selection menu. [Figure 29] 10 is an example of a sequence diagram illustrating a process in which an information recording application switches a recorded video displayed in an image display field in response to a user operation. [Figure 30] 10 is an example of a sequence diagram illustrating a process in which an information recording application displays text data in association with the display time of a recorded video displayed in an image display field. [Figure 31] 10A and 10B are diagrams for explaining a typical example of searching for text data and displaying the corresponding recorded video. [Figure 32] 10 is an example of a sequence diagram illustrating a process in which an information recording application displays a recorded video in an image display field in accordance with searched text data. DETAILED DESCRIPTION OF THE INVENTION

[0009] A recorded information display system and a recorded information display method performed by the recorded information display system will be described below as an example of an embodiment of the present invention.

[0010] <An example of how to create minutes for a remote meeting> First, an outline of a method for creating minutes using a panoramic image and an application screen will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. As shown in Fig. 1, a user at a home location 102 is using a remote conference service system 90 to hold a remote conference with another location 101.

[0011] The recorded information display system 100 of this embodiment creates recorded information (minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) acquired by a meeting device 60 equipped with an imaging means capable of capturing 360° of surroundings, a microphone, and a speaker, by processing information captured from the surroundings, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. Regarding audio, the recorded information display system 100 combines audio received by a remote conference application (hereinafter referred to as a remote conference app 42) with audio acquired by the meeting device 60, and includes the combined audio in the recorded information. Note that in this embodiment, it is also possible to simply view and display the recorded information without creating it. An outline of this is provided below.

[0012] (1) An information recording application 41 and a remote conference application 42, which will be described later, are running on the terminal device 10. In addition, an application for displaying materials may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including the audio received by the remote conference application 42 from other locations) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired with the audio from the remote conference application 42.

[0013] (2) The meeting device 60 is equipped with a microphone and performs processing to extract the speaker from the panoramic image based on the direction from which the voice was captured, to create a speaker image (an example of speaker image information). The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display a panoramic image 203 and a speaker image 204. The information recording application 41 saves any application screen selected by the user (for example, the screen 103 of the remote conference application 42), the panoramic image 203, and the speaker image 204. The application screen is an example of screen information (described later) displayed by each application, such as the remote conference application. Since each of these is saved repeatedly, they are saved as separate videos. Hereinafter, when the application screen, the panoramic image 203, and the speaker image 204 are not to be distinguished from one another, they are referred to as "recorded videos."

[0015] (4) The information recording application 41 accepts editing work (cutting out unnecessary parts by the user) and completes the recorded video. The recorded video constitutes part of the recorded information.

[0016] (5) The information recording application 41 transmits the recorded video and audio data to the storage service system 70 and stores them.

[0017] (6) The information recording application 41 also sends a request to convert the voice data into text to the information processing system 50. The information processing system 50 then transmits the voice data to the voice recognition service system 80, which converts the voice data into text data. The text data also includes data on how many minutes after the start of recording that the voice was spoken.

[0018] In the case of real-time text conversion, the meeting device 60 transmits the audio directly to the information processing system 50 .

[0019] (7) The information processing system 50 additionally stores the text data in the storage service system 70 that stores the recorded video. The text data constitutes part of the recorded information.

[0020] The information processing system 50 can perform billing processing for the user according to the service used. For example, the billing is calculated based on the amount of text data, the file size of the recorded video, the processing time, and the like.

[0021] (8) The information recording application 41 can play back recorded video together with audio data. The information recording application 41 can also display text data corresponding to the recorded video. The arrangement of the panoramic image 203, the speaker image 204, and the screen 103 of the remote conference application 42 is merely an example. For example, the panoramic image 203 and the speaker image 204 are arranged on the left side, and the screen 103 of the remote conference application 42 is arranged on the right side.

[0022] In this way, when the recorded video is displayed, a panoramic image of the surroundings including the user, an image of the speaker, and the screen of an app that was displayed during the remote conference, such as the remote conference app 42, are displayed. Therefore, when a participant or a non-participant in the remote conference views the recorded video as minutes, the panoramic image, the image of the speaker, and the screen of the app are displayed on one screen, so that the situation during the remote conference is reproduced with a sense of realism.

[0023] <Terminology> An application (app) is software developed and used for a specific function or purpose. Applications include native apps and web apps. Web apps (cloud-side apps provided by cloud services) can also operate in conjunction with native apps or web browsers.

[0024] A running app is an app that is in a state between when it is launched and when it is closed. The app does not have to be the active app (the foreground app), it just needs to be running in the background.

[0025] A device is a device that can capture images of the device's surroundings and collect surrounding sounds. The device may be used by connecting to a terminal device, may be built into a terminal device, or may be used by connecting to a cloud service without being directly connected to a terminal device. In this embodiment, the device is described using the term meeting device.

[0026] The image of the surroundings of the meeting device acquired by the meeting device is image information acquired by the meeting device by capturing an image of the surrounding space (for example, an area or space of 180° to 360° horizontally) surrounding the meeting device, and refers to an image acquired by performing predetermined processing on the curved image information captured by the meeting device. The predetermined processing is various processing for creating surrounding image information from the captured information, such as flattening the captured curved image. In addition to processing for creating a surrounding image, the predetermined processing may also include processing for cutting out a speaker image and combining surrounding images and speaker images. In this embodiment, the surrounding image is described by the term panoramic image. A panoramic image is an image with a field of view of approximately 180° to 360° horizontally. A panoramic image does not have to be captured by a single meeting device; multiple imaging devices with normal fields of view may be combined. The meeting device is intended to be installed on a table or other suitable location for use in meetings at a base or to grasp the surrounding situation, but it may also be used for monitoring (security, disaster prevention, etc.), watching over (childcare, elderly care, etc.), or analyzing the situation on-site (solutions, marketing, etc.).

[0027] Recorded information is information recorded by the information recording application 41, and is information that is stored and saved so that it can be viewed as information linked to the identification information of a particular conference (meeting), and includes, for example, the following information: Video information created based on the screen information displayed by the selected app (such as a remote conferencing app) and image information about the device's surroundings acquired by the device. During a meeting, audio information is acquired and synthesized between the remote conference app (terminal device) and the meeting device at the base. -Text information converted from the captured audio. Other data and images related to the meeting, such as document files used during the meeting, notes added, translations of text data, and images and stroke data created using the cloud electronic whiteboard service during the meeting. When the information recording application 41 records the screen of a remote conference application or the state of a conference at a location, the recorded information may be minutes of the conference that was held. Minutes are an example of recorded information, and the name of the recorded information varies depending on the content of the remote conference or the location, and may be called, for example, a record of communication or a record of the location situation. Recorded information also includes files in multiple formats, such as video files (e.g., combined image videos), audio files, text data (text data obtained by voice recognition), document files, image files, and table-format files. Because the files are associated with each other based on the conference identification information, they can be viewed together or selectively in chronological order.

[0028] A tenant is a group of users (such as a company, a local government, or some of these organizations) who have signed a contract to receive services from a service provider. In this embodiment, the creation of recorded information and conversion to text data are performed because the tenant has signed a contract with the service provider.

[0029] Remote communication refers to communication through audio and video using software and terminal devices with people at physically distant locations. An example of remote communication is a remote conference, which may also be called a meeting, a conference, a consultation, a contract application, a gathering, a get-together, a seminar, a training session, a study group, a seminar, or a workshop.

[0030] A base is a place where activities are based. An example of a base is a conference room. A conference room is a room set up primarily for meetings. A base can also be a home, a reception desk, a store, a warehouse, an outdoor site, or any other location, as long as it is a place or space where terminal equipment, devices, etc. can be installed.

[0031] Although voice refers to speech sounds uttered by humans, surrounding sounds, etc., and voice data refers to voice converted into data, in the present embodiment, a strict distinction will not be made between them.

[0032] <System configuration example> Next, the system configuration of the recorded information display system 100 will be described with reference to Fig. 2. Fig. 2 shows an example of the configuration of the recorded information display system 100. Fig. 2 shows one location (own location 102) among multiple locations where a remote conference is held, and a terminal device 10 at the own location 102 communicates with an information processing system 50, a storage service system 70, and a remote conference service system 90 via a network. A meeting device 60 is also placed at the own location 102, and the terminal device 10 is connected to this meeting device 60 via a USB cable or the like so as to be able to communicate with it.

[0033] At least an information recording application 41 and a remote conference application 42 run on the terminal device 10. The remote conference application 42 can communicate with terminal devices at other locations 101 via a remote conference service system 90 on the network, allowing users at each location to hold conferences from remote locations. The information recording application 41 creates record information for the remote conferences held by the remote conference application 42 using the functions of the information processing system 50 and the meeting device 60.

[0034] In this embodiment, an example of creating recorded information during a remote conference will be described, but the conference does not have to involve communication between a remote location and another remote location. In other words, the conference may involve only participants from within a single location. In this case, only the audio collected by the meeting device 60 is saved without synthesis, and the processing of the information recording application 41 remains unchanged.

[0035] The terminal device 10 has a built-in camera with a normal angle of view (or it may be external), which captures a front image including the user 107 operating the terminal device 10. A normal angle of view is an image that is not a panoramic image, and in this embodiment, it is a planar image that is not curved, such as a spherical image. The terminal device 10 also has a built-in microphone (or it may be external), which collects surrounding sounds of the user operating the terminal device 10. Therefore, the user can hold a conventional remote conference using the remote conference app 42 without being aware of the information recording app 41. The information recording app 41 and the meeting device 60 do not affect the remote conference app 42, except for increasing the processing load on the terminal device 10. The remote conference app 42 can also transmit panoramic images and speaker images captured by the meeting device 60 to the remote conference system 90.

[0036] The information recording application 41 is an application that records information by communicating with the meeting device 60 and creating recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capturing device, a microphone, and a speaker. While the camera of the terminal device 10 can only capture a limited range in front of it, the meeting device 60 can capture images of the entire surroundings (not necessarily the entire periphery) of the meeting device 60. The meeting device 60 can always fit multiple participants 106 shown in FIG. 2 within its field of view.

[0037] In addition, the meeting device 60 extracts a speaker image from a panoramic image, synthesizes the audio acquired by the meeting device 60 with the audio output by the terminal device 10 (including the audio received by the remote conference application 42), and so on. The meeting device 60 is not limited to being placed on a desk, a table, or the like, and may be placed anywhere at the home base 102. Since the meeting device 60 can capture spherical images, it may be placed on the ceiling, for example. The meeting device 60 may also be installed at another base or at any base.

[0038] The information recording application 41 displays a list of applications running on the terminal device 10, saves the above-mentioned recorded information, displays recorded videos, accepts editing requests, etc. The information recording application 41 also displays a list of remote conferences that have been held or are scheduled to be held, etc. The list of remote conferences is used for information related to the recorded information, allowing the user to associate remote conferences with the recorded information.

[0039] The remote conference application 42 is an application that enables a terminal device to remotely communicate with other terminal devices by establishing a communication connection with other terminal devices at other locations 101, transmitting and receiving images and audio, displaying images and outputting audio, etc. The remote conference application can also be called a remote communication application, a remote information common application, etc.

[0040] The information recording application 41 and the remote conference application 42 may be web applications or native applications. A web application is an application that performs processing through collaboration between a program on a web server and a program on a web browser or a native application, and does not need to be installed on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used. In this embodiment, both applications will be described as native applications.

[0041] The terminal device 10 may be a general-purpose information processing device with a communication function, such as a PC (Personal Computer), a smartphone, or a tablet terminal. The terminal device 10 may also be an electronic whiteboard, a game console, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, industrial machinery, medical equipment, or network home appliances. The terminal device 10 may be any device on which at least the information recording application 41 and the remote conference application 42 can run.

[0042] The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and a platform service. The server applications manage remote conference lists, manage recorded information recorded in remote conferences, and manage various settings and storage paths. The platform service handles user authentication, contracts, billing, etc.

[0043] All or part of the functions of the information processing system 50 may exist in a cloud environment or an on-premise environment. The information processing system 50 may be configured with multiple server devices or may be configured with a single information processing device. For example, the server application and the infrastructure service may be provided by separate information processing devices, and each function within the server application may have its own information processing device. The information processing system 50 may be integrated with the conference management system 9, storage service system 70, and voice recognition service system 80 described below.

[0044] The conference management system 9 is a system that manages information related to conferences held by tenants that use the information processing system 50. The information processing system 50 acquires conference information from the conference management system 9 and manages the conference information in association with recording information.

[0045] The storage service system 70 is a storage means on a network and provides a storage service that accepts the saving of files, etc. Known examples of the storage service system 70 include OneDrive (registered trademark), Google Workspace (registered trademark), and DropBox (registered trademark). The storage service system 70 may also be an on-premise NAS (Network Attached Storage), etc.

[0046] The voice recognition service system 80 provides a service that performs voice recognition on voice data and converts it into text data. The voice recognition service system 80 may be a general-purpose commercial service, or may be part of the functions of the information processing system 50. Furthermore, the voice recognition service system 80 may be set up and used as a different service system for each user, tenant, or conference.

[0047] <Hardware configuration example> The hardware configuration of the information processing system 50 and the terminal device 10 according to this embodiment will be described with reference to FIG.

[0048] <<Information processing system and terminal device>> Fig. 3 is a diagram showing an example of the hardware configuration of an information processing system 50 and a terminal device 10 according to this embodiment. As shown in Fig. 3, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0049] Of these, the CPU 501 controls the overall operation of the information processing system 50 and the terminal device 10. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 3.

[0050] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0051] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be described using Figure 4. Figure 4 is an example of a hardware configuration diagram of a meeting device 60 capable of capturing 360-degree video. In the following, the meeting device 60 is described as a device that uses an imaging element to capture 360-degree video around the device at a predetermined height, but the imaging element may be one, two, or any number of elements. Furthermore, it does not necessarily have to be a dedicated device; a PC, digital camera, smartphone, etc. may have substantially the same functionality by attaching a 360-degree video imaging unit to it after the fact.

[0052] As shown in Figure 4, the meeting device 60 is composed of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a concave terminal 621 for Micro USB.

[0053] The imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 with a 360° angle of view for forming a hemispherical image, and imaging elements 603 (image sensors) provided for each wide-angle lens. The imaging elements 603 include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lens 602 into electrical image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers that set various commands and parameters required for the operation of the imaging elements. The imaging unit 601 may be a 360° camera, an example of an imaging means capable of capturing a 360° view around the meeting device 60. A 360° angle of view may be achieved by combining information acquired by multiple imaging elements (e.g., 180° + 180°).

[0054] The imaging elements 603 (image sensors) of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I / F bus. Meanwhile, the imaging elements 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an audio sensor 618, and the like are also connected to the bus 610.

[0055] The image processing unit 604 takes in the image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates data for a panoramic image and a speaker image from the fisheye video. Furthermore, the image processing unit 604 synthesizes the panoramic image and the speaker image, etc., and outputs a single video.

[0056] The imaging control unit 605 generally uses the I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0057] Furthermore, the imaging control unit 605 instructs the imaging elements 603a and 603b to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from the PC. Some meeting devices 60 have a preview display function or a function that supports video display on a display (for example, a PC or smartphone display). In this case, the image data is output from the imaging element 603 continuously at a predetermined frame rate (frames per minute).

[0058] As will be described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0059] The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 via the I / F bus and performs predetermined processing on the sound data.

[0060] The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of processed equirectangular projection images.

[0061] The operation unit 615 is a general term for operation buttons such as an imaging start button 615a. By operating the operation unit 615, the user can start imaging or recording, as well as turn the power on / off, establish communication connections, and input settings such as various imaging modes and imaging conditions.

[0062] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device may be, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection I / F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. The video data and image data stored in the DRAM 614 are transmitted to an external terminal via the external device connection I / F 616 or recorded on external media. Furthermore, by using multiple external device connection I / Fs 616, for example, image information captured by the meeting device 60 may be transmitted to a PC via USB for recording, while video (e.g., screen information to be displayed in a remote conference app) may be acquired from the PC to the meeting device 60, and the video may then be transmitted from the meeting device 60 via HDMI to another external device (e.g., a display, a projector, an electronic whiteboard) for display.

[0063] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the meeting device 60, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also communicate with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0064] The audio sensor 618 is a sensor that acquires 360° audio information to identify from which direction a loud audio is being input within 360° around (horizontal plane) the meeting device 60. The audio processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction within 360°.

[0065] Note that other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use this information for image correction or adding location information.

[0066] The image processing unit 604 also performs the following processes.

[0067] The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB interpolation processing) on ​​the RAW data input from the image sensor that inputs spherical images to create a fisheye image (image made up of curved images).The CPU 611 then performs flattening processing such as DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image (image made up of flat images) that captures 360 degrees around the meeting device 60.

[0068] The CPU 611 creates speaker images in the following way. The CPU 611 creates speaker images by cutting out the speaker from a panoramic image (flat video) that captures 360° of the surrounding area. The CPU 611 cuts out the speaker image from the panoramic image, using the voice input direction identified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. In this case, the method for cutting out a person's image from the voice input direction is to cut out a 30° area centered on the identified voice direction from the 360°, and perform face detection within that area to cut out the image. The CPU 611 further identifies the speaker images of a specific number of people (e.g., three people) who have spoken most recently from the cut-out speaker images.

[0069] The panoramic image and the one or more speaker images may be sent separately to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the one or more speaker images are sent separately from the meeting device 60 to the information recording application 41.

[0070] Figure 5 is a diagram illustrating the imaging range of the meeting device 60. As shown in Figure 5(a), the meeting device 60 captures an image in a 360° range in the horizontal direction. As shown in Figure 5(b), the imaging range of the meeting device 60 is 0° in the horizontal direction at the height of the meeting device 60, and a predetermined angle above and below.

[0071] FIG. 6 is a diagram illustrating the cutting out of a panoramic image and a speaker image. As shown in FIG. 6, the image captured by the meeting device 60 forms a portion 110 of a sphere and therefore has a three-dimensional shape. As shown in FIG. 5(b), the meeting device 60 performs perspective projection transformation by dividing the angle of view into predetermined angles in the vertical direction and the horizontal direction. By performing perspective projection transformation across the entire 360° horizontal direction without gaps, a predetermined number of planar images are obtained, and by connecting the predetermined number of planar images horizontally, a panoramic image 111 is obtained. In addition, the meeting device 60 performs face detection within a predetermined range from the panoramic image, centered on the direction of the voice, and creates a speaker image 112 by cutting out an image 15° left and right from the center of the face (30° overall).

[0072] <About the function> Next, the functional configuration of the recorded information display system 100 will be described with reference to Fig. 7. Fig. 7 is an example of a functional block diagram that explains the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the recorded information display system 100 by dividing them into blocks.

[0073] <<Terminal Device>> The information recording application 41 running on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a video storage unit 17, an audio data processing unit 18, a recorded video display unit 19, an upload unit 20, an editing processing unit 21, and a search unit 22. Each of these units included in the terminal device 10 is a function or a means for performing a function that is realized when any of the components shown in FIG. 3 operates in response to an instruction from the CPU 501 in accordance with the information recording application 41 deployed from the HD 504 to the RAM 503. The terminal device 10 also includes a storage unit 1000 configured by the HD 504 and the like shown in FIG. 3. An information storage unit 1001 is configured in the storage unit 1000.

[0074] The communication unit 11 communicates various types of information with the information processing system 50 via a network. For example, the communication unit 11 receives a list of remote conferences from the information processing system 50 and transmits a request for recognizing voice data to the information processing system 50.

[0075] The display control unit 13 displays various screens serving as a user interface in the information recording application 41 in accordance with screen transitions set in the information recording application 41. The operation accepting unit 12 accepts various operations for the information recording application 41.

[0076] The application screen acquisition unit 14 acquires screen information displayed by an application selected by the user or screen information of the desktop screen from an operating system (OS). If the application selected by the user is a remote conference application 42, the unit acquires screens generated by the remote conference application 42 (images captured by the terminal device cameras at each location, images displayed with shared materials, participant icons, participant names, etc.). The screen information displayed by the application (application screen) is information displayed as a window by a running application, including the remote conference application, and acquired as an image by an information recording application. The application window is displayed on a monitor or the like, with the window area represented as a portion of the entire desktop image. The screen information displayed by the application can be acquired by other applications (such as an information recording application) as an image file or a video file consisting of multiple consecutive images via an API of the operating system (OS) or the API of the application that displays the application. The screen information of the desktop screen is information composed of an image of the desktop screen generated by the OS and can be acquired as an image file or a video file via the API of the OS. These image files may be in bitmap, PNG, or other formats. The video file format may be MP4 or other formats.

[0077] The audio acquisition unit 15 acquires audio output from the microphone or earphones of the terminal device 10 (including audio data received from the remote conference application 42 during the remote conference). The audio acquisition unit 15 can acquire audio even when the output audio is muted. Regarding audio data, the user does not need to perform any operation, such as selecting the remote conference application 42. The audio acquisition unit 15 can acquire audio that the terminal device 10 can output via the OS or application API (Application Interface). This also acquires audio data received by the remote conference application 42 from other locations 101. If the remote conference application 42 is not running or if a remote conference is not in progress, the information recording application 41 may not be able to acquire audio data. Note that the audio acquired by the audio acquisition unit 15 may not include audio collected by the terminal device 10, and may only include the audio data to be output. This is because the meeting device 60 separately collects audio from the locations.

[0078] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 via wireless LAN, Bluetooth (registered trademark), or the like. The device communication unit 16 receives the panoramic image and the speaker image from the meeting device 60, and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0079] The video storage unit 17 stores the panoramic image and speaker image received by the device communication unit 16, and the app screen acquired by the app screen acquisition unit 14, as separate recorded videos. The video storage unit 17 also stores the synthesized audio data separately. The video storage unit 17 may combine the audio data with the panoramic image to create a panoramic image with audio.

[0080] The voice data processing unit 18 requests the information processing system 50 to convert the synthesized voice data received from the meeting device 60 into text data.

[0081] The recorded video display unit 19 displays the recorded video. The recorded video is stored in the terminal device 10 while being recorded, and is then uploaded to the information processing system 50.

[0082] When the remote conference ends, the upload unit 20 transmits the recorded video and audio data to the information processing system 50.

[0083] The editing processing unit 21 executes editing of recorded moving images (deleting a part, splicing, etc.) in response to user operations.

[0084] The search unit 22 searches the text data for the keyword entered by the user. For a portion of the text data that matches the search, the time at which the comment was made after the start of the conference is also specified.

[0085] Fig. 8 shows video recording information stored in the information storage unit 1001. The video recording information includes items such as a conference ID, a recording ID, an update date and time, a title, an upload, and a save destination. When a user logs in to the information processing system 50, the information recording application 41 can download the conference information that the information processing system 50 acquires from the conference management system 9 and the application-side conference information that the information processing system 50 has. The conference ID and other information included in the conference information are reflected in the video recording information. The video recording information in Fig. 8 is held by a terminal device 10 operated by a certain user.

[0086] The conference ID is identification information for identifying the remote conference that has been held. The conference ID is assigned when a remote conference schedule is registered in the conference management system 9, or is assigned by the information processing system 50 in response to a request from the information recording application 41. The conference management system 9 is a system for registering conference and remote conference schedules, a URL (conference link) for starting a remote conference, reservation information for devices to be used in the conference, and the like, and is a scheduler or the like connected from the terminal device 10 via a network. The conference management system 9 can also transmit the registered schedule, etc. to the information processing system 50.

[0087] The recording ID is identification information for identifying the video recorded during a remote conference. The recording ID is assigned by the meeting device 60, but may also be assigned by the information recording application 41 or the information processing system 50. A different recording ID is assigned to the same conference ID when recording stopped during the remote conference but was resumed for some reason.

[0088] The update date and time is the date and time when the recorded video was updated (recording ended). If the recorded video was edited, this is the edit date and time.

[0089] The title is the name of the conference. It may be set when the conference is registered in the conference management system 9, or may be set arbitrarily by the user.

[0090] · Upload indicates whether the recorded video has been uploaded to the information processing system 50 or not.

[0091] The destination indicates the location (URL or file path) where the recorded video and text data are saved in the storage service system 70. This allows users to view the uploaded recorded video at their discretion. Note that the panoramic image, speaker image, app screen, and text data are saved with separate file names, for example, following the URL.

[0092] <<Meeting Devices>> Returning to Figure 7, the meeting device 60 has a terminal communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, and a voice synthesis unit 65. Each of these units in the meeting device 60 is a function or a means for performing the function, which is realized when any of the components shown in Figure 4 operates in response to an instruction from the CPU 611 in accordance with a program loaded from the ROM 612 to the DRAM 614.

[0093] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may communicate with the terminal device 10 via wireless LAN, Bluetooth (registered trademark), or the like.

[0094] The panoramic image creating unit 62 creates a panoramic image. The speaker image creating unit 63 creates a speaker image. The methods for creating these images have been explained with reference to FIGS.

[0095] The sound collection unit 64 converts the audio signal acquired by the microphone of the meeting device 60 into audio data (digital). As a result, the content of speech made by the user or participant at the base on the terminal device 10 side is collected.

[0096] The voice synthesis unit 65 synthesizes the voice transmitted from the terminal device 10 with the voice collected by the voice collection unit 64. As a result, the voices spoken at the other locations 101 and the voices spoken at the own location 102 are combined.

[0097] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a conference information acquisition unit 54, and a text conversion unit 55. Each of these units included in the information processing system 50 is a function or a means for performing a function that is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The information processing system 50 also includes a storage unit 5000 configured using the HD 504 and the like shown in Fig. 3. The storage unit 5000 includes an application-side conference information storage unit 5001 and a recording information storage unit 5002.

[0098] The communication unit 51 transmits and receives various types of information to and from the terminal device 10. For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 and receives a request to recognize voice data from the terminal device 10.

[0099] The authentication unit 52 authenticates the user operating the terminal device 10. For example, the authentication unit 52 authenticates the user based on whether or not authentication information (user ID and password) included in an authentication request received by the communication unit 51 matches authentication information stored in advance. Note that the authentication information may be an IC card number, biometric authentication information such as face or fingerprint, or the like. The authentication unit 52 may also perform authentication using an external authentication system or an authentication method such as OAUTH.

[0100] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 executes a native application, the screen information is stored in the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 executes a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), or the like.

[0101] The conference information acquisition unit 54 acquires conference information from the conference management system 9 using each user's account or a system account assigned to the information processing system 50. The conference information acquisition unit can acquire a list of remote conferences that users belonging to a tenant have viewing authority for. Note that viewing authority may be granted to conference information managed by the conference management unit 54 directly from the information recording application of the terminal device 10. The remote conference information that users belonging to a tenant have viewing authority for includes information on conferences created by the user and information on conferences for which the user has been granted viewing authority by other users. A conference ID is assigned to each remote conference, and the remote conference and recorded information are associated by the conference ID.

[0102] The text conversion unit 55 converts the voice data requested by the terminal device 10 to text data into text data using an external voice recognition service. The text conversion unit 55 may also perform the conversion itself.

[0103] 9 shows an example of application-side conference information stored in the application-side conference information storage unit 5001. The application-side conference information is conference information on the information recording application 41 side that is created separately from the conference information when the information recording application 41 creates recording information.

[0104] The conference ID is identification information of the conference. The conference ID associates the conference information managed by the conference management system 9 with the application-side conference information.

[0105] The application ID is identification information of the information recording application 41.

[0106] The user ID to which the access token is issued is identification information of the storage service system 70 that issues the access token.

[0107] The URL of the access token issuer is the URL of the storage service system 70 that issues the access token.

[0108] The issue date is the date and time when the access token was issued. The issue date and time is determined at the time of issuance.

[0109] The validity date is the date and time when the access token becomes valid. The validity date and time is determined by the issuer.

[0110] The expiration date is the expiration date of the access token. The expiration date is determined by the issuer.

[0111] The permission information is the permission of the user to perform certain operations using the access token. The permission information is determined by the issuer.

[0112] The display name is the display name of the user in the information recording application 41.

[0113] Last name is the user's last name.

[0114] First name is the user's first name.

[0115] FIG. 10 shows the recording information stored in the recording information storage unit 5002. The recording information includes a list of recorded videos recorded by all users belonging to the tenant. The recording information includes items such as a meeting ID, a recording ID, an update date and time, a title, and a storage destination. These items may be the same as those in FIG. 8. The storage destination (path information such as a URL of a cloud storage system) may be entered by the user on a user setting screen of the information recording application 41 of the terminal device 10, and the desired storage destination information may be stored in the recording information storage unit 5002.

[0116] <<Conference Management System>> Returning to Fig. 7, the explanation will be made. The conference management system 9 has a conference information management unit 31. Each of these units in the conference management system 9 is a function or a means for performing the function, which is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The conference management system 9 also has a storage unit constructed by the HD 504 etc. shown in Fig. 3. A conference information storage unit 3001 is constructed in the storage unit 3000.

[0117] The conference information management unit 31 manages information about conferences held by tenants. The conference information is stored in the conference information storage unit 3001.

[0118] 11 shows an example of conference information stored in the conference information storage unit 3001. The conference information is managed by conference ID and includes the items shown in the figure.

[0119] The conference ID is the identification information for the conference.

[0120] The tenant ID is the identification information of the tenant where the conference is held.

[0121] · The title is the subject or agenda of the meeting.

[0122] The organizer is the organizer of the meeting.

[0123] · Participants is a list of participants invited to the meeting.

[0124] Viewing privileges are a list of users who have access to meeting resources, including recorded video.

[0125] Ad-hoc participants are a list of guest participants.

[0126] Location is information about the conference room, such as the name of the conference room.

[0127] Start time is the scheduled time the meeting will begin.

[0128] The end time is the scheduled time when the meeting will end.

[0129] The conference creator is the user ID etc. who registered the conference information.

[0130] The password is the password that participants use to log in to the conference.

[0131] <<Storage Service System>> The storage service system 70 may be any service system that stores information. Here, the structure of text data stored in the storage service system 70 will be described.

[0132] FIG. 12 is a diagram illustrating the structure of text data stored in the storage service system 70. As shown in FIG. 12, the text data is associated with the following items: ID, time, user, and text. ID is identification information assigned when text data is segmented into texts according to a predetermined rule. The predetermined rule is set in the speech recognition service system 80, and may, for example, segment the text when silence continues for a certain period of time, forcibly segment the text at certain intervals even if there is no silence, or segment the text into sentences detected by morphological analysis. time is the time of speech based on the duration from the start of recording. Since the so-called time is also saved at the start of recording, the time (absolute time) when the text string was spoken can also be determined. user indicates whether the speech was from the user's location or another location, as determined by sound pressure, etc. The text string is part of the segmented text data.

[0133] In this way, since time and text are associated with each other, when the text data matches the search, the information recording application 41 can display the recorded video from the display time corresponding to this time.

[0134] <Screen transition> Next, several screens that the terminal device 10 displays during a remote conference will be described with reference to Fig. 13 to Fig. 22. Fig. 13 shows an initial screen 200 after login that is displayed by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs authentication information and successfully logs in, the initial screen 200 of Fig. 13 is displayed.

[0135] The initial screen 200 has a fixed display button 201, a front change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter, referred to as speaker images 204 when no distinction is made), and a recording start button 205. If the meeting device 60 is already running and capturing images of the surroundings at the time of login, the panoramic image 203 and speaker image 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether to start recording while viewing these. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker image 204 are not displayed.

[0136] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or may display only the speaker images 204 of the N most recently spoken participants. Fig. 13 shows an example in which speaker images 204 of up to three people are displayed. Until a participant speaks, there may be no speaker image 204 (the speaker images are added one by one as the participant speaks), or three speaker images 204 of participants in a predetermined direction may be displayed (the speaker images are replaced as the participant speaks).

[0137] When no one is speaking, such as immediately after the meeting device 60 is started up, a predetermined direction (0°, 120°, 240°, etc.) within the horizontal 360° is created as the speaker image 204. If a fixed display, which will be described later, is set, the setting of the fixed display takes priority.

[0138] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 in close-up as a speaker image 204 .

[0139] FIG. 14 is a diagram illustrating an operation method when the fixed display button 201 is on. For example, the user moves a rectangular window 206 on the panoramic image 203 using a pointing device such as a mouse or a touch panel. The user aligns the window 206 with a whiteboard, podium, or other object included in the panoramic image 203. The user's operation is transmitted to the meeting device 60, which then creates an image of the range indicated by the window within the 360° horizontal direction, with the same size as the speaker image 204, and transmits the image to the terminal device 10. This allows the speaker image 204 to continue displaying a whiteboard or other object other than the speaker.

[0140] Returning to FIG. 13 , the front change button 202 is a button that allows the user to perform an operation to change the front of the panoramic image 203 (because a panoramic image covers 360° horizontally, the right and left edges are aligned). The user can slide the panoramic image 203 left and right with a pointing device to determine which participant will be displayed in the front. The user's operation is transmitted to the meeting device 60, which then changes the angle of the front within the 360° horizontal direction, creates a panoramic image, and transmits it to the terminal device 10.

[0141] When the user presses the record start button 205, the information recording application 41 displays the recording setting screen 210 shown in FIG.

[0142] If the meeting device 60 is not connected or powered on when the information recording application 41 is started, the device unrecognized screen 250 of FIG. 15 is displayed.

[0143] 15 shows an example of an unrecognized device screen 250. The unrecognized device screen 250 displays a message 251 saying "Device cannot be recognized. Please turn on the device and connect." The user can check the power and connection status of the meeting device 60 by looking at this message.

[0144] 16 is an example of a recording setting screen 210 displayed by the information recording application 41. On the recording setting screen 210, the user can set whether to record (whether to include in the recording) the panoramic image and speaker image created by the meeting device 60, and the desktop screen or the screen of a running application of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or the screen of a running application, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) is recorded.

[0145] The camera toggle button 211 is a button for switching on and off the recording of the panoramic image and the speaker image created by the meeting device 60. The camera toggle button 211 may be set to record the panoramic image and the speaker image separately.

[0146] The PC screen toggle button 212 is a button that switches on and off recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10. When the PC screen toggle button 212 is in the on state, the desktop screen is recorded.

[0147] If the user wants to record the screen of an application, the user further selects the application in the application selection field 213. The application selection field 213 displays the names of applications currently running on the terminal device 10 in a pull-down menu. The user can select the application to record. The information recording application 41 acquires this application name from the OS. The information recording application 41 can display only those applications that have a UI (screen) among the currently running applications. The selected applications may include the remote conference application 42. Therefore, the information recording application 41 can record videos of materials displayed in the remote conference application 42 and participants at each location. In addition, the applications displayed in the pull-down menu include various applications currently running on the terminal device, such as a presentation application, a word processing application, a spreadsheet application, an application for creating and editing documents or other materials, a cloud electronic whiteboard application, and a web browser application. Therefore, the user can flexibly select the application screens to include in the recorded video.

[0148] When recording on an app basis, the user can select multiple apps, and the information recording app 41 can record the screens of all the selected apps.

[0149] When both the camera toggle button 211 and the PC screen toggle button 212 are off, the recording content confirmation window 214 displays "Only audio will be recorded." This audio includes the audio output by the terminal device 10 (audio received by the remote conference app 42 from other locations 101) and the audio collected by the meeting device 60. In other words, as long as a remote conference is being held, the audio from the remote conference app 42 and the audio from the meeting device 60 are saved regardless of whether images are recorded. However, the user may be able to selectively stop saving the audio from the remote conference app 42 and the audio from the meeting device 60 using user settings.

[0150] The recorded video is recorded as follows according to the combination of on and off of the camera toggle button 211 and the PC screen toggle button 212. The recorded video is also displayed in real time in the recorded content confirmation window 214.

[0151] When the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image are displayed in the recording content confirmation window 214 .

[0152] When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen has also been selected), the recording content confirmation window 214 displays the desktop screen or the screen of the selected application.

[0153] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, are displayed side by side in the recording content confirmation window 214.

[0154] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or cases where the panoramic image, speaker image, and app screen are not recorded at all, but in this embodiment, for convenience, the images created by the information recording application 41 are referred to as recorded videos.

[0155] 17 shows an example of the display of the recorded content confirmation window 214 when the camera toggle button 211 is on and the PC screen toggle button 212 is off. In FIG. 17, the panoramic image 203 and the speaker image 204 are displayed large.

[0156] Fig. 18 shows an example of the display of the recorded content confirmation window 214 when the camera toggle button 211 is on and the PC screen toggle button 212 is on. In Fig. 18, a panoramic image 203 and a speaker image 204 are displayed on the left, and an application screen 217 is displayed on the right.

[0157] Therefore, the recording content confirmation window 214 allows the user to check what content will be recorded in the video (particularly images from the meeting device 60) according to the settings on the recording setting screen 210 before starting recording.

[0158] 18 shows an example of a recorded video display when only one app is selected, but when two or more apps are selected, the screens of the second and subsequent apps are linked sequentially to the right. Alternatively, the screens of the second and subsequent apps may be arranged in two dimensions, vertically and horizontally.

[0159] Returning to FIG. 16, the explanation will be made. The recording setting screen 210 has a check box 215 along with the message "Automatically transcribe recording after uploading." The recording setting screen 210 also has a start recording now button 216. When the user checks the check box 215, text data into which comments made during the remote conference are converted is attached to the recorded video. In this case, after the recording is completed, the information recording application 41 uploads the audio to the information processing system 50 along with a request to convert it into text data. Furthermore, when the user presses the start recording now button 216, the recording in progress screen 220 of FIG. 19 is displayed.

[0160] FIG. 19 is an example of a recording screen 220 that the information recording application 41 displays during recording. Note that the explanation of FIG. 19 will mainly focus on the differences from FIG. 16. The recording screen 220 displays a recorded video that is being recorded under the conditions set by the user on the recording setting screen 210. The recording screen 220 of FIG. 19 is when the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays a panoramic image 203 and a speaker image 204 (both of which are videos) created by the meeting device 60. The recording screen 220 displays a recording icon 225, a pause button 226, and an end recording button 227.

[0161] If the user sets the PC screen toggle button 212 to ON, the recording screen 220 displays the panoramic image and speaker image next to the desktop screen and application screen, as shown in FIG.

[0162] The pause button 226 is a button for stopping recording, and after stopping, recording can also be resumed. The end recording button 227 is a button for ending recording. The pause button 226 does not change the recording ID, but the end recording button 227 changes the recording ID. After pausing, when recording is resumed, the user can reset the recording conditions set on the recording setting screen 210. In this case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, by pressing the end recording button 227), or may concatenate multiple files so that they form a single continuous video (for example, by pressing the pause button 226). Furthermore, when the information recording application 41 displays a recorded video, multiple recording files may be displayed continuously as a single video.

[0163] The recording in progress screen 220 also has an Get Information From Calendar button 221, a conference name field 222, a time field 223, and a location field 224. The Get Information From Calendar button 221 is a button that the user uses to obtain conference information from the conference management system 9. When the Get Information From Calendar button 221 is pressed, the information recording application 41 obtains and displays a list of conferences that the user has permission to view from the information processing system 50. The user selects the upcoming remote conference from the conference list. The information recording application 41 can also obtain application-side conference information. As a result, the conference information is reflected in the conference name field 222, the time field 223, and the location field 224. The conference name field 222 reflects the title of the conference information, the time field 223 reflects the start time and end time, and the location field 224 reflects the location. The remote conference in the conference management system 9 is associated with the recorded information by a conference ID.

[0164] Once the remote conference is over and the user stops recording, a recorded video with audio is created.

[0165] FIG. 20 shows an example of the conference list screen 230 displayed by the information recording application 41. The conference list screen 230 is a list of conferences, but can also display a list of recorded information recorded during remote conferences. It also includes conferences that were held only in a certain conference room, regardless of whether they were remote conferences. The conference list screen 230 displays an integrated display of conference information that the logged-in user has permission to view in the conference information storage unit 3001, application-side conference information, and information stored in the video recording information storage unit 5002 that is associated with the remote conference. Information about video recording information stored in the information storage unit 1001 may be further integrated.

[0166] The conference list screen 230 is displayed when the user selects the conference list tab 231 on the initial screen 200 in FIG. 13. The conference list screen 230 displays a list 236 of recorded information that the user has permission to view. The conference creator (minutes creator) can set viewing permissions for participants. The conference list may be a list of stored recorded information or a list of conference schedules and conference data.

[0167] The conference list screen 230 has the following items: a check box 232 , an update date and time 233 , a title 234 , and a status 235 .

[0168] The checkboxes 232 are used to select recorded files. The checkboxes 232 are used when the user wants to delete recorded files all at once.

[0169] The update date and time 233 indicates the start and end times of recording of the recorded video. If the video has been edited, the edit date and time may be used.

[0170] The title 234 is the title of the meeting (e.g., agenda). It may be transcribed from the meeting information or set by the user.

[0171] The status 235 indicates whether or not the recorded video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, an upload button is displayed. If there is recorded video that has not been uploaded, it is recommended that the information recording application 41 automatically uploads it when the user logs in to the information processing system 50.

[0172] When the user selects an arbitrary title or the like from the list 236 of recorded videos with a pointing device, the information recording application 41 displays the recorded video display screen 240 shown in Fig. 21. On the recorded video display screen 240, it is possible to display the recorded video, etc.

[0173] 21 shows an example of a recorded video display screen 240 after a recorded video is selected, displayed by the information recording application 41. The recorded video display screen 240 has an image display field 241, a transcription button 242, one or more text display fields 243, an auto scroll button 244, and a search button 245.

[0174] The image display field 241 has a display button 241a, a rewind button 241b, a fast-forward button 241c, a time indicator 241d, a display speed button 241e, a volume button 241f, etc. The image display field 241 displays a recorded video. In FIG. 21 , the recorded video in the image display field 241 has a panoramic image and a speaker image arranged on the left side, and the screen of the remote conference app 42 displayed on the right side. During the remote conference, the screen of the remote conference app 42 changes to an image of the location or an image of the document. Therefore, the user can view the screen of a desired scene by operating various buttons.

[0175] If the audio data of the recorded video being displayed in the image display field 241 has been converted into text data, the content of the comment is displayed as text in the text display field 243. The text display field 243 displays the text data into which the synthesized audio data has been converted by voice recognition.

[0176] The transcription button 242 is a button that allows the user to switch whether or not the text data displayed in the text display field 243 is displayed in accordance with the display time of the recorded video.

[0177] The automatic scroll button 244 is a button that allows the user to switch whether or not to automatically scroll the text data regardless of the display time.

[0178] The search button 245 is a button that allows the user to search for text data by specifying a keyword. Details will be described later.

[0179] Furthermore, the recorded video display screen 240 may allow the recorded video to be downloaded.

[0180] FIG. 22 shows an example of an editing screen 260 for recorded video. The editing screen 260 is displayed automatically from the recording screen 220 or when the user performs a predetermined operation on the recorded video display screen 240. The editing screen 260 has a first display field 261 and a second display field 262. The first display field 261 displays the recorded video at a certain moment during display, and the second display field 262 displays the frames that make up the recorded video in chronological order. The user can select one or more frames and delete unnecessary frames. The user can also extract some of the frames and insert them after any frame. The editing processing unit 21 edits the recorded video in response to user operations, and overwrites the edited recorded video or saves it separately.

[0181] <Operation or processing procedure> Next, the operations and processes performed by the recorded information display system 100 based on the above configuration will be described.

[0182] <<Saving recorded video>> The process of saving recorded moving images will be described with reference to Fig. 23. Fig. 23 is an example of a sequence diagram showing the procedure by which the information recording application 41 records a panoramic image, a speaker image, and an application screen.

[0183] S1: The user starts the information recording application 41 on the terminal device 10 and connects the terminal device 10 (information recording application 41) to the information processing system 50. If the access token has expired, the display control unit 13 displays a login screen. The user inputs authentication information (e.g., user ID and password) for logging in to the tenant into the information recording application 41. The operation reception unit 12 of the information recording application 41 receives the input.

[0184] S2: The communication unit 11 of the information recording application 41 transmits a login request to the information processing system 50 by specifying authentication information.

[0185] S3: The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 52 authenticates the user based on the authentication information. Here, it is assumed that the authentication is successful. The communication unit 51 of the information processing system 50 transmits the access token 1 to the information recording application 41. Although not explicitly shown in the figure, from then on, the communication unit 51 attaches the access token 1 to communications with the information processing system 50. The access token 1 is associated with the authority of the logged-in user.

[0186] S4: Similarly, the user also logs in to the storage service system 70. This is because the recorded information is saved in the storage service system 70. The user inputs authentication information (e.g., user ID and password) for logging in to the storage service system 70. The operation reception unit 12 of the information recording application 41 receives the input.

[0187] S5: The communication unit 11 of the information recording application 41 transmits a login request to the information processing system 50, specifying the authentication information.

[0188] S6: The communication unit 51 of the information processing system 50 receives the login request, and since it is a login request to the storage service system 70, transfers the login request to the storage service system 70.

[0189] S7: The storage service system 70 authenticates the user based on the authentication information. Here, it is assumed that the authentication is successful. The storage service system 70 sends the access token 2 to the information processing system 50.

[0190] S8: The communication unit 51 of the information processing system 50 receives the access token 2 and transmits it to the information recording application 41. Although not shown explicitly in the figure, thereafter, the communication unit 51 attaches the access token 2 to communications with the storage service system 70. The access token 2 is associated with the authority of the logged-in user.

[0191] S21: Next, the user operates the teleconferencing application 42 to start a teleconferencing. Here, it is assumed that the teleconferencing application 42 of the home base 102 and the teleconferencing application 42 of the other base 101 have started a teleconferencing. The teleconferencing application 42 of the home base 102 transmits images captured by a camera and audio collected by a microphone of the terminal device 10 to the teleconferencing application 42 of the other base 101. The teleconferencing application 42 of the other base 101 displays the received images on a display and outputs the received audio from a speaker. Similarly, the teleconferencing application 42 of the other base 101 transmits images captured by a camera and audio collected by a microphone of the terminal device 10 to the teleconferencing application 42 of the home base 102. The teleconferencing application 42 of the home base 102 displays the received images on a display and outputs the received audio from a speaker. Each teleconferencing application 42 repeats this process to realize the teleconferencing.

[0192] S22: The user makes settings related to recording on the recording setting screen 210 of the information recording application 41 shown in Fig. 16. The operation accepting unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that the camera toggle button 211 and the PC screen toggle button 212 are both on.

[0193] If the user has already reserved a remote conference in advance, the user can display a list of remote conferences by pressing the Get Information button 221 on the calendar in Fig. 19 and select a remote conference to associate with the recorded video. Since the user has already logged in to the information processing system 50, the information processing system 50 identifies remote conferences that the logged-in user has permission to view. The information processing system 50 transmits a list of the identified remote conferences to the terminal device 10, and the user selects a remote conference that is currently being held or will be held in the future. This determines information about the remote conference, such as the conference ID.

[0194] In addition, even if the user has not reserved a remote conference in advance, the user can create a conference when creating a recorded video. The following describes a case where the information recording application 41 creates a conference when creating a recorded video and obtains a conference ID from the information processing system 50.

[0195] S23: The user instructs the information recording application 41 to start recording (start recording now button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0196] S24: Since the remote conference has not been selected (the conference ID has not been determined), the communication unit 11 of the information recording application 41 transmits a remote conference creation request to the information processing system 50.

[0197] S25: The communication unit 51 of the information processing system 50 receives the remote conference creation request, the conference information acquisition unit 54 acquires a unique conference ID assigned by the conference management system 9, and the communication unit 51 transmits the conference ID to the information recording application 41.

[0198] S26: The conference information acquisition unit 54 also transmits the destination where the recorded video is to be saved (the URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

[0199] S27: When the communication unit 11 of the information recording application 41 receives the conference ID and the destination to save the recorded file, the video storage unit 17 determines that preparations for recording are complete and starts recording.

[0200] S28: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In FIG. 23, the application selected by the user is the remote conference application 42.

[0201] S29: The video storage unit 17 of the information recording application 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. When notifying, the video storage unit 17 may also notify that the camera toggle button 211 is on (requesting a panoramic image and a speaker image). Regardless of whether a request is made, the meeting device 60 sends the panoramic image and the speaker image to the information recording application 41.

[0202] S30: When the terminal communication unit 61 of the meeting device 60 receives a recording start signal, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or may be obtained from the information processing system 50.

[0203] S31: The voice acquisition unit 15 of the information recording application 41 acquires the voice data output by the terminal device 10 (voice data received by the remote conference application 42).

[0204] S32: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and a synthesis request to the meeting device 60.

[0205] S33: The terminal communication unit 61 of the meeting device 60 receives the voice data and the synthesis request, and the voice synthesis unit 65 synthesizes the received voice data with the surrounding voice data collected by the sound collection unit 64. For example, the voice synthesis unit 65 adds the two pieces of voice data together. Because clear voices around the meeting device 60 are recorded, the accuracy of converting voices around the meeting device 60 (conference room side) into text is improved.

[0206] This audio synthesis can also be performed by the terminal device 10. However, by distributing the recording function to the terminal device 10 and the audio processing to the meeting device 60, the load on the terminal device 10 and the meeting device 60 can be reduced. The recording function may be distributed to the meeting device 60 and the audio processing may be distributed to the terminal device 10.

[0207] S34: Furthermore, the panoramic image creating unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creating unit 63 creates a speaker image.

[0208] S35: The device communication unit 16 of the information recording application 41 repeatedly acquires the panoramic image and the speaker image from the meeting device 60. The device communication unit 16 also repeatedly requests and acquires the synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making a request to the meeting device 60. Alternatively, the meeting device 60 may automatically send the panoramic image and the speaker image upon receiving a notification that the camera toggle button 211 is on. The meeting device 60 may automatically send the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0209] S36: The display control unit 13 of the information recording application 41 displays the application screen, panoramic image, and speaker image side by side on the recording screen 220. The video storage unit 17 of the information recording application 41 also stores the application screen, panoramic image, and speaker image acquired from the remote conference application 42 as separate recorded videos. That is, the video storage unit 17 creates each recorded video by specifying the repeatedly received application screen, panoramic image, and speaker image as frames that make up the video. The video storage unit 17 also stores the audio data received from the meeting device 60.

[0210] The information recording application 41 repeats the above steps S31 to S36.

[0211] S37: When the remote conference ends and recording is no longer necessary, the user instructs the information recording application 41 to end recording (for example, by pressing the recording end button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0212] S38: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. The meeting device 60 continues creating panoramic images and speaker images and synthesizing audio. However, the meeting device 60 may change the processing options, such as changing the resolution or fps, depending on whether recording is in progress.

[0213] S39: The video storage unit 17 of the information recording application 41 combines the audio data with the recorded video to create a combined image video with audio. If no recorded video is saved, the audio data may be independent. Also, there are three types of recorded video: the app screen, the panoramic image, and the speaker image, and which recorded video each will be combined with is determined in advance along with the priority. Also, the recorded video and audio data do not necessarily have to be combined.

[0214] S40: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the recording setting screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. In detail, the audio data processing unit 18 specifies the URL of the save destination via the communication unit 11, and transmits a request to convert the audio data combined with the recorded video to the information processing system 50 together with the conference ID and recording ID.

[0215] S41: The communication unit 51 of the information processing system 50 receives a request to convert the voice data, and the text conversion unit 55 converts the voice data into text data using the voice recognition service system 80. The communication unit 51 saves the text data in the same storage destination (the URL of the storage service system 70) as the storage destination of the recorded video. In the video recording information storage unit 5002, the text data is associated with the recorded video by the conference ID and the video recording ID. The text data may be managed by the conference management unit 54 of the information processing system 50 and stored in the storage unit 5000. Alternatively, the terminal device 10 may request voice recognition from the voice recognition service system 80 and store the text data obtained from the voice recognition service system 80 in a storage destination. The voice recognition service system 80 returns the converted text data to the information processing system 50, or may directly send it to the URL of the storage destination. The voice recognition service system 80 may select or switch between multiple services depending on setting information set by the user in the information processing system 50.

[0216] S42: Furthermore, the upload unit 20 of the information recording application 41 saves the recorded video in the destination for saving the recorded video via the communication unit 11. In the video recording information storage unit 5002, the recorded video is associated with the conference ID and the video recording ID. The recorded video is recorded as having been uploaded.

[0217] The save location is notified to the user, so the user can share the recorded video with participants by informing them of the save location by email, etc. Even if the recorded video, audio data, and text data are created using different devices, they can be collected and stored in a single storage location, making them easy for users to view later.

[0218] In the case of real-time voice recognition, the meeting device 60 or the terminal device 10 transmits voice data in real time to the information processing system 50. The terminal device 10 displays the text data transmitted from the meeting device 60 or returned from the information processing system 50 on the recording screen 220 and also stores the text data.

[0219] The processing of steps S31 to S36 does not have to be performed in the order shown in FIG. 23, and the synthesis of audio data and the saving of recorded moving images may be performed in the reverse order.

[0220] <<Displaying recorded video>> Next, a process in which a user displays a recorded video stored in the storage service system 70 will be described with reference to Fig. 24. Fig. 24 is an example of a sequence diagram illustrating a process in which the information recording application 41 downloads and displays a recorded video.

[0221] S51: A user who wants to display recorded video displays the conference list screen in the information recording application 41. To do this, the user selects the conference list tab 231. The operation acceptance unit 12 of the information recording application 41 accepts the selection.

[0222] S52: The communication unit 11 of the information recording application 41 designates the access token 1 and transmits the conference list to the information processing system 50.

[0223] S53: The communication unit 51 of the information processing system 50 receives the conference list request, and the conference information acquisition unit 54 transmits the conference list request to the conference management system 9.

[0224] S54, S55: The conference information management unit 31 of the conference management system 9 identifies conference information that the user of the access token 1 has permission to view. In addition, the conference information management unit 31 acquires, from the information processing system 50, application-side conference information that is associated with the identified conference information by the conference ID.

[0225] S56: The conference information management unit 31 integrates the conference information of the conference management system 9 with the application-side conference information, and the conference management system 9 transmits the integrated conference list to the information processing system 50. The information processing system 50 may also perform the integration.

[0226] S57: The communication unit 51 of the information processing system 50 receives the conference list, and the screen generation unit 53 generates a conference list screen.

[0227] S58: The communication unit 51 of the information processing system 50 transmits the screen information of the conference list screen to the information recording application 41. The communication unit 11 of the information recording application 41 receives the screen information of the conference list screen, and the display control unit 13 displays it.

[0228] S59: The user selects the conference for which the user wants to display the recorded video. The operation reception unit 12 of the information recording application 41 receives the selection (the conference ID is identified).

[0229] S60: The communication unit 11 of the information recording application 41 sends a request for recorded information to the information processing system 50, specifying the conference ID.

[0230] S61: The communication unit 51 of the information processing system 50 receives a request for recorded information, and first, the conference information acquisition unit confirms the existence of the conference with the conference management system 9. Here, it is assumed that the conference exists. Note that this confirmation process may be omitted.

[0231] S62: Since a conference exists, the screen generating unit 53 obtains from the recording information storage unit 5002 the recording information (conference ID, recording ID, update date and time, title, save destination, etc.) associated with the conference ID.

[0232] S63: The screen generation unit 53 uses the recording information to create the recorded video display screen 240. At this point, the recorded video has not yet been acquired, so the image display field 241 may be blank or may contain a title. Also, if there is a thumbnail image of the recorded video in the recording information storage unit 5002, it is preferable that this be displayed in the image display field 241.

[0233] S64: The communication unit 51 of the information processing system 50 transmits the screen information of the recorded moving image display screen 240 to the information recording application 41. The communication unit 11 of the information recording application 41 receives the screen information of the recorded / playback screen 240, and the display control unit 13 displays it.

[0234] S65: The user starts displaying the recorded video (turns on the display button 241a). The operation accepting unit 12 of the information recording application 41 accepts the operation.

[0235] S66: The communication unit 11 of the information recording application 41 specifies the access token 2 and connects to the destination stored in the information storage unit 1001.

[0236] S67: The storage service system 70 returns to the information recording application 41 the URL where the storage destination (recorded information (recorded video and text data)) is stored.

[0237] S68: The communication unit 11 of the information recording application 41 receives this URL and connects to it, allowing the recorded information to be downloaded. The recorded video display unit 19 displays the recorded video on the recorded video display screen 240, and displays the corresponding text data.

[0238] S69: The user can also delete conference information from the information recording application 41. The user selects the conference to be deleted on the conference list screen. The operation accepting unit 12 of the information recording application 41 accepts the operation (the conference ID is identified).

[0239] S70: The communication unit 11 of the information recording application 41 specifies the access token 2 to connect to the storage destination of the storage service system 70 stored in the information storage unit 1001, and deletes the recorded information.

[0240] S71: Next, the communication unit 11 of the information recording application 41 connects to the conference management system 9 using the access token 1, and transmits a request to delete the conference information by specifying the conference ID.

[0241] S72: The conference management system 9 receives the deletion request, and the conference information management unit 31 deletes the conference information identified by the conference ID and the application-side conference information.

[0242] <Details when viewing recorded video> As explained above, the recorded information includes: a. Panoramic image b. Speaker image c. One or more app screens d. Text data The audio data may be combined with any of a to c. a to c may not be present depending on the user's recording settings. d may also not be present if the audio data has not been converted into text data. However, if a to c are present, the user can switch the recorded video to be displayed on the recorded video display screen 240.

[0243] Fig. 25 shows an example of a recorded video display screen 240 on which a panoramic image 203 and a speaker image 204 are displayed. Note that the explanation of Fig. 25 will mainly focus on the differences from Fig. 21. The recorded video display screen 240 displays a recorded video selection menu 249. The recorded video selection menu 249 has a video button 246, a content button 247, and a video & content button 248. The video button 246 is a button that displays a panoramic image and a speaker image, the content button 247 is a button that displays an app screen, and the video & content button 248 is a button that displays a speaker image and an app screen.

[0244] In FIG. 25, the video button 246 is pressed, and the panoramic image 203 and the speaker image 204 are displayed in the image display field 241.

[0245] 26 shows a recorded video display screen 240 in which the video & content button 248 is pressed and the speaker image 204 and the app screen 217 are displayed. The speaker image 204 and the app screen 217 are displayed in an image display field 241. Note that the area in which people are lined up vertically is where the speaker images are lined up vertically. Although a panoramic image is not displayed, a panoramic image may also be displayed.

[0246] 27 shows a recorded moving image display screen 240 on which the content button 247 is pressed and the app screen 217 is displayed. The app screen 217 is displayed in an image display field 241.

[0247] Fig. 28 shows recorded videos displayed in the image display field 241, which are switched in response to an operation on the recorded video selection menu 249. As shown in Fig. 28, the user can operate the recorded video selection menu 249 to switch the recorded video to be displayed.

[0248] <<Recorded video switching process>> FIG. 29 is an example of a sequence diagram illustrating a process in which the information recording application 41 switches the recorded moving image displayed in the image display field 241 in response to a user operation.

[0249] S81: The user inputs an operation to switch the recorded moving image to be displayed to the recorded moving image selection menu 249. The operation acceptance unit 12 of the information recording application 41 accepts the operation.

[0250] S82: The recorded moving image display unit 19 of the information recording application 41 acquires the current display time of the recorded moving image displayed in the image display field 241.

[0251] S83: The recorded moving image display unit 19 displays the switched-to recorded moving image in the image display field 241 from the display time.

[0252] In this way, even if the recorded video is switched, the recorded video after the switch can be displayed from the display time before the switch.

[0253] 29, the information recording application 41 processes the downloaded recording information on the terminal device side to switch the recorded video, but the switching of the recorded video may be performed by the information processing system 50. For example, such processing may be performed when the information processing system 50 displays the recorded information as a Web application.

[0254] <<Displaying the correspondence between text data and recorded video>> Next, the display of the text data and the recorded video by the information recording application 41 in correspondence with each other will be described with reference to Fig. 30. Fig. 30 is an example of a sequence diagram illustrating the process in which the information recording application 41 displays the text data in correspondence with the display time of the recorded video displayed in the image display field 241.

[0255] S91: The user turns on the transcription button 242 on the recorded video display screen 240. The operation acceptance unit 12 of the information recording application 41 accepts the operation.

[0256] S92: The recorded moving image display unit 19 of the information recording application 41 repeatedly acquires the current display time of the recorded moving image displayed in the image display field 241.

[0257] S93: The recorded moving image display unit 19 compares the time contained in the text data with the display time, and if there is a character string of text within a predetermined time from the display time, displays this character string of text in the text display field 243. Therefore, according to the display time, for example, the character string of text is displayed in chronological order from top to bottom.

[0258] In this way, the text data is displayed in correspondence with the recorded video, so the user can easily check the content of what was said when the recorded video was displayed.

[0259] 30, the information recording application 41 associates the recorded video with the text data, but the synchronization process may be performed by the information processing system 50. In this case, the information recording application 41 transmits the display time to the information processing system 50, and the information processing system 50 returns the text data corresponding to the display time to the information recording application 41.

[0260] <<Searching for text data>> Next, the search for text data by the information recording application 41 and the corresponding display of recorded moving images will be described with reference to FIGS.

[0261] Figure 31 is a diagram that schematically illustrates the search for text data and the corresponding display of recorded video. In Figure 31(a), the user enters "keyword" in search box 265 and searches for text data using this keyword. Search unit 22 searches the text data for "keyword" and displays "It keyword it," "that keyword it," and "this keyword it" as search results 266.

[0262] In Figure 31 (b), the user selects "this keyword it" with a pointing device. When the operation reception unit 12 receives the selection of "this keyword it," the recorded video display unit 19 obtains the time associated with "this keyword it" from the text data. The recorded video display unit 19 displays the currently displayed recorded video in the image display field 241 from the time indicated by the time (it may also display a still image).

[0263] FIG. 32 is an example of a sequence diagram illustrating a process in which the information recording application 41 displays a recorded video in the image display field 241 in accordance with the searched text data.

[0264] S101: The user inputs a keyword to be searched and issues a search instruction. The operation reception unit 12 of the information recording application 41 receives the search instruction.

[0265] S102: The search unit 22 of the information recording application 41 searches the text data for the keyword.

[0266] S103: The display control unit 13 displays a list of character strings in text as search results.

[0267] S104: The user selects an arbitrary character string for text. The operation reception unit 12 of the information recording application 41 receives the selection.

[0268] S105: The recorded moving image display unit 19 of the information recording application 41 displays the recorded moving image from the time associated with the character string of text.

[0269] In this way, when text data is searched, the recorded video is displayed in correspondence with the text data, so the user can reduce the effort of searching for the recorded video at the time of the statement.

[0270] In Fig. 32, the information recording application 41 searches for text data and displays the recorded video, but this processing may also be performed by the information processing system 50. In this case, the information recording application 41 transmits a keyword to the information processing system 50, and the information processing system 50 searches for text data using the keyword. The information processing system 50 transmits the search results to the information recording application 41. The information recording application 41 transmits the text character string selected by the user to the information processing system 50. The information processing system 50 requests the information recording application 41 to display from the time associated with the text character string. When the information processing system 50 displays the recorded video, it transmits the recorded video from the time associated with the text character string.

[0271] <Major Effects> As described above, the recorded information display system of this embodiment simultaneously displays a panoramic image of the surroundings including the user, an image of the speaker, and the screen of an app displayed during the remote conference, such as the remote conference app 42. Therefore, when participants in the remote conference or non-participants view the recorded video as minutes, the situation during the remote conference is reproduced with a sense of realism. Furthermore, the system displays a video created by recording both screen information displayed by an application (e.g., a remote conference app) selected to be included in the recorded information and image information of the surroundings of the device within the location (e.g., within a conference room) from start to finish. Therefore, even if the screen content displayed by the remote conference app 42 changes during the conference (recording), the system can display recorded information that thoroughly records the situation at the location (e.g., within a conference room) along with the content of the remote conference (remote communication). Furthermore, the information recording app 41 can display recorded video and text data in association with each other. Furthermore, the information recording app 41 can selectively display recorded video in response to user operations. The information recording app 41 can display recorded video in association with a character string that matches a search.

[0272] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0273] For example, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. The meeting device 60 may also be an omnidirectional camera, a microphone, and a speaker connected by a cable.

[0274] Furthermore, a meeting device 60 may also be placed at another location 101. The other location 101 separately uses the meeting device 60 to create combined image video and text data. Furthermore, multiple meeting devices 60 may be placed at one location. In this case, multiple pieces of record information are created for each meeting device 60.

[0275] The layout of the panoramic image 203, speaker image 204, and app screen in the recorded video used in this embodiment is merely an example. The panoramic image 203 may be on the bottom and the speaker image 204 on top, or the user may change the layout, or may individually switch between displaying and hiding the panoramic image 203 and speaker image 204 when displaying the recorded video.

[0276] In addition, the configuration examples in Figure 7 and the like are divided according to main functions to make it easier to understand the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the terminal device 10, the meeting device 60, and the information processing system 50 can be divided into even more processing units depending on the processing content. Furthermore, the processing can be divided so that one processing unit includes even more processes. For example, a system for creating recorded videos and a system for displaying recorded videos may be separate.

[0277] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, information processing system 50 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0278] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in FIG. 23, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Furthermore, the information processing system 50 may be integrated into a single server device, or may be divided into multiple devices.

[0279] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a conventional circuit module designed to perform each of the above-described functions. <Appendix to the Claims> [Claim 1] A recorded information display system including an information recording application executed on a terminal device, the information recording application has a recorded video display unit that displays a recorded video created based on screen information displayed by a remote conference application selected by the information recording application and surrounding image information acquired by a device; The recorded information display system is characterized in that the recorded moving image display unit displays the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information. [Claim 2] 2. The recorded information display system according to claim 1, wherein the screen information is information obtained as an image by the information recording application when a running application including the remote conference application is displayed as a window. [Claim 3] 3. The recorded information display system according to claim 1, wherein the image information about the surroundings of the device is image information acquired by the device by capturing an image of a 360° area surrounding the device. [Claim 4] 4. The recorded information display system according to claim 1, wherein the image information of the surroundings of the device is transmitted from the device installed at a base to the information recording application. [Claim 5] The recorded information display system of claim 1, characterized in that the recorded video display unit further displays text data based on the audio data collected by the device and the audio data output by the terminal device on the same screen as the recorded video. [Claim 6] the text data is voice data obtained by synthesizing voice data collected by the device and voice data output by the terminal device, 6. The recorded information display system according to claim 5, wherein the recorded moving image display unit displays the text data converted from the synthesized voice data by voice recognition. [Claim 7] 7. The recorded information display system according to claim 5, wherein the recorded moving image display unit scrolls and displays the text data in accordance with the display time of the recorded moving image. [Claim 8] a search unit that searches the text data for keywords; a display control unit that displays a portion of the text data that matches the search, The recorded information display system of claim 5 or 6, characterized in that when a portion of the text data is selected, the recorded video display unit displays the recorded video at the speech time associated with the portion of the text data. [Claim 9] the recorded video includes the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information; The recorded information display system of any one of claims 1 to 8, characterized in that the recorded video display unit switches between displaying the surrounding image information and the speaker image information, displaying the screen information, or displaying the speaker image information and the screen information, depending on the user's operation. [Claim 10] The recorded information display system of claim 9, characterized in that the recorded video display unit displays the surrounding image information and the speaker image information, the screen information, or the speaker image information and the screen information after the user switches between displaying the surrounding image information and the speaker image information, the screen information, or the speaker image information and the screen information, from the display time when the user switches between displaying the surrounding image information and the speaker image information, the screen information, or the speaker image information and the screen information. [Explanation of symbols]

[0280] 10 Terminal Equipment 50 Information Processing Systems 60 Meeting Devices 100 Record Information Display System [Prior art documents] [Patent documents]

[0281] [Patent Document 1] Patent Publication No. 2021-105688

Claims

1. A recorded information display system including an information recording application executed on a terminal device, the information recording application has a recorded video display unit that displays a recorded video created based on screen information displayed by a remote conference application selected by the information recording application and surrounding image information acquired by a device; The recorded information display system is characterized in that the recorded moving image display unit displays the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information.

2. 2. The recorded information display system according to claim 1, wherein the screen information is information obtained as an image by the information recording application, which displays a running application including the remote conference application as a window.

3. 3. The recorded information display system according to claim 1, wherein the image information about the surroundings of the device is image information acquired by the device capturing an image of a 360° area surrounding the device.

4. The recorded information display system according to claim 1 , wherein the image information of the surroundings of the device is transmitted from the device installed at a base to the information recording application.

5. The recorded information display system of claim 1, characterized in that the recorded video display unit further displays text data based on the audio data collected by the device and the audio data output by the terminal device on the same screen as the recorded video.

6. the text data is voice data obtained by synthesizing voice data collected by the device and voice data output by the terminal device, 6. The recorded information display system according to claim 5, wherein the recorded moving image display unit displays the text data converted from the synthesized voice data by voice recognition.

7. 7. The recorded information display system according to claim 5, wherein the recorded moving image display unit scrolls and displays the text data in accordance with the display time of the recorded moving image.

8. a search unit that searches the text data for keywords; a display control unit that displays a portion of the text data that matches the search, The recorded information display system described in claim 5 or 6, characterized in that when a portion of the text data is selected, the recorded video display unit displays the recorded video at the speech time corresponding to the portion of the text data.

9. the recorded video includes the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information; The recorded information display system of claim 1, characterized in that the recorded video display unit switches between displaying the surrounding image information and the speaker image information, displaying the screen information, or displaying the speaker image information and the screen information, depending on the user's operation.

10. The recorded information display system of claim 9, characterized in that the recorded video display unit displays the surrounding image information and the speaker image information, the screen information, or the speaker image information and the screen information after switching based on the display time when the user switches between displaying the surrounding image information and the speaker image information, the screen information, or the speaker image information and the screen information.

11. A program executed on a terminal device, causing the terminal device to function as a recorded video display unit that displays a recorded video created based on screen information displayed by a remote conference application selected by the program and surrounding image information acquired by the device; The recorded video display unit is a program for displaying the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information.

12. A recorded information display method performed by a recorded information display system including an information recording application executed on a terminal device, comprising: the information recording application displays a recorded video created based on screen information displayed by the remote conference application selected by the information recording application and surrounding image information acquired by the device; A recorded information display method, comprising displaying the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information.

Images