Recording information display system, program, recording information display method, recording information playback system

Abstract

To display recorded information of remote communication on the basis of screen information displayed by an application being executed and image information of the surroundings of a device.SOLUTION: A recorded information creation system includes an information recording application executed by a terminal device. The information recording application has a recorded moving image display unit which displays screen information displayed by a remote conference application selected by the information recording application and a recorded moving image created on the basis of a surrounding image acquired by a device. The recorded moving image display unit is characterized by displaying the surrounding image information, speaker image information cut off from the surrounding image information, and the screen information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0006] , ,

[0001] The present invention relates to a recording information display system, a program, and a recording information display method.

Background Art

[0002] A remote conferencing system is known that transmits images and sounds in real time from one base to one or more other bases and enables users at remote locations to hold meetings using images and sounds.

[0003] Also, a technique for displaying a minutes screen based on information obtained by photographing and collecting sound from all 360 degrees around is known (see, for example, Patent Document 1). Patent Document 1 discloses a minutes screen composed of a region for displaying participants in a conference room and a speech history region arranged in chronological order for each speech, based on information obtained by photographing and collecting sound from all 360 degrees around.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technology has a problem that recording information of remote communication is not displayed based on the screen information displayed by the running application and the surrounding image information. That is, conventionally, even if information obtained by imaging the inside of a conference room can be displayed, it has not been displayed together with the screen of an application such as a remote conferencing application. For this reason, it has been impossible to view the state during remote communication while imaging the inside of a conference room.

[0005] In view of the above problems, an object of the present invention is to provide a technique for displaying recording information of remote communication based on the screen information displayed by a running application and the image information around the device.

Means for Solving the Problems

[0006] In view of the above issues, the present invention provides a recorded information display system including an information recording application executed on a terminal device, wherein the information recording application has a recorded video display unit that displays a recorded video created based on screen information displayed by a remote conferencing application selected by the information recording application and surrounding image information acquired by the device, and the recorded video display unit displays the surrounding image information, speaker image information cropped from the surrounding image information and the screen information. [Effects of the Invention]

[0007] Based on screen information displayed by the running application and image information of the device's surroundings, this system can display recorded information of remote communication, allowing users to view what happened when remote communication was conducted while imaging the inside of a conference room. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram outlines the creation of recording information that saves the screen of an application running during a remote meeting, along with a panoramic image of the surroundings. [Figure 2] This diagram shows an example configuration of a recorded information display system. [Figure 3] This figure shows an example of the hardware configuration of an information processing system and terminal device. [Figure 4] This figure shows an example of a hardware configuration for a meeting device. [Figure 5] This diagram illustrates the imaging range of the meeting device. [Figure 6] This diagram illustrates the process of extracting images from a panoramic image and a speaker image. [Figure 7] This is an example of a functional block diagram that explains the functions of terminal devices, meeting devices, and information processing systems in a recorded information display system by dividing them into blocks. [Figure 8] This figure shows an example of video recording information stored in the information storage unit. [Figure 9]This is an example of application-side meeting information stored in the application-side meeting information storage unit. [Figure 10] This diagram shows the recorded information stored in the recorded information storage unit. [Figure 11] This is an example of side meeting information stored in the meeting information storage unit. [Figure 12] This diagram illustrates the structure of text data stored in a storage service system. [Figure 13] This figure shows an example of the initial screen displayed after login by an information recording application running on a terminal device. [Figure 14] This diagram explains how to operate the system when the fixed display button is turned on. [Figure 15] This figure shows an example of a screen indicating that a device is not recognized. [Figure 16] This figure shows an example of a recording settings screen displayed by an information recording app. [Figure 17] This diagram shows an example of how the recording content confirmation window looks when the camera toggle button is on and the PC screen toggle button is off. [Figure 18] This diagram shows an example of how the recording content confirmation window looks when the camera toggle button and the PC screen toggle button are both turned on. [Figure 19] This figure shows an example of a recording screen displayed by an information recording app while recording. [Figure 20] This figure shows an example of a meeting list screen displayed by an information recording app. [Figure 21] This figure shows an example of the screen displaying recorded videos after selecting a recorded video from an information recording app. [Figure 22] This figure shows an example of the editing screen for recorded video. [Figure 23] This is an example of a sequence diagram showing the procedure by which an information recording app records panoramic images, speaker images, and the app screen. [Figure 24] This is an example of a sequence diagram illustrating the process by which an information recording application downloads and displays recorded videos. [Figure 25]It is a diagram showing an example of a recorded video display screen 240 on which a panoramic image and a speaker image are displayed. [Figure 26] It is a diagram showing an example of a recorded video display screen on which a video & content button is pressed and a speaker image and an app screen are displayed. [Figure 27] It is a diagram showing an example of a recorded video display screen on which a content button is pressed and an app screen is displayed. [Figure 28] It is a diagram showing a recorded video displayed in an image display column that switches according to an operation on a recorded video selection menu. [Figure 29] It is an example of a sequence diagram explaining a process in which an information recording app switches a recorded video displayed in an image display column according to a user operation. [Figure 30] It is an example of a sequence diagram explaining a process in which an information recording app displays corresponding text data according to the display time of a recorded video displayed in an image display column. [Figure 31] It is a diagram schematically explaining the search of text data and the corresponding display of a recorded video. [Figure 32] It is an example of a sequence diagram explaining a process in which an information recording app displays a recorded video corresponding to text data searched in an image display column.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, as an example of an embodiment for carrying out the present invention, a recording information display system and a recording information display method performed by the recording information display system will be described.

[0010] <An Example of a Method for Creating Minutes of a Remote Meeting> First, referring to FIG. 1, an outline of a method for creating minutes using a panoramic image and an app screen will be described. FIG. 1 is a diagram explaining an outline of the creation of recording information for saving the screen of an app executed during a remote meeting together with a surrounding panoramic image. As shown in FIG. 1, a user at the illustrated home base 102 is using a remote conferencing service system 90 to hold a remote meeting with another base 101.

[0011] The recording information display system 100 of this embodiment creates recording information (meeting minutes) using a horizontal panoramic image (hereinafter referred to as the panoramic image) acquired by processing information captured by a meeting device 60 equipped with an imaging means capable of capturing images of the surroundings in a 360° range, a microphone, and a speaker, and a screen created by an application (hereinafter referred to as the app) executed by the terminal device 10. For audio, the recording information display system 100 combines the audio received by the remote conferencing application (hereinafter referred to as the remote conferencing app 42) and the audio acquired by the meeting device 60 and includes them in the recording information. In this embodiment, it is also possible to simply view and display the recording information without creating it. The outline is described below.

[0012] (1) The terminal device 10 is running the information recording application 41 and the remote conferencing application 42, which will be described later. Other applications, such as one for displaying documents, may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including the audio received by the remote conferencing application 42 from other locations) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired with the audio from the remote conferencing application 42.

[0013] (2) The meeting device 60 is equipped with a microphone and, based on the direction from which the sound was acquired, processes the image to extract the speaker from the panoramic image and creates a speaker image (an example of speaker image information). The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display a panoramic image 203 and a speaker image 204. The information recording application 41 saves the application screen selected by the user (for example, the screen 103 of the remote conferencing application 42), the panoramic image 203, and the speaker image 204, respectively. The application screen is an example of screen information (described later) displayed by each application such as the remote conferencing application. Since these are saved repeatedly, each is saved as a separate video. Hereafter, when the application screen, panoramic image 203, and speaker image 204 are not distinguished, they will be referred to as "recorded video".

[0015] (4) The information recording application 41 accepts editing work (cutting out unnecessary parts by the user) and completes the recorded video. The recorded video constitutes part of the recorded information.

[0016] (5) The information recording application 41 sends the recorded video and audio data to the storage service system 70 for storage.

[0017] (6) The information recording application 41 also sends a request to the information processing system 50 to transcribe the audio data. The information processing system 50 sends the audio to the speech recognition service system 80, which converts the audio into text data. The text data also includes data on how many minutes after the start of recording the speaker spoke.

[0018] In the case of real-time text conversion, the meeting device 60 transmits the audio directly to the information processing system 50.

[0019] (7) The information processing system 50 additionally stores text data in the storage service system 70 which stores the recorded video. The text data constitutes part of the recorded information.

[0020] Furthermore, the information processing system 50 can perform billing processing for users based on the services they have used. For example, billing is calculated based on the amount of text data, the file size of recorded videos, processing time, etc.

[0021] (8) The information recording app 41 can play back the recorded video along with the audio data. The information recording app 41 can also display text data in conjunction with the recorded video. The arrangement of the panoramic image 203, speaker image 204, and the screen 103 of the remote conferencing app 42 is just one example, but for example, the panoramic image 203 and speaker image 204 are placed on the left, and the screen 103 of the remote conferencing app 42 is placed on the right.

[0022] Thus, when the recorded video is displayed, a panoramic image of the surroundings including the user, the speaker's image, and the screen of the application used during the remote meeting, such as the remote conferencing application 42, are displayed. Therefore, when a participant or non-participant of the remote meeting views the recorded video as meeting minutes, the panoramic image, speaker's image, and application screen are displayed on a single screen, recreating the atmosphere of the remote meeting with a sense of presence.

[0023] <About Terminology> An application (app) is software developed and used for a specific function or purpose. Applications can be native apps or web apps. A web app (a cloud-based app provided by a cloud service) may also work in conjunction with a native app or web browser.

[0024] A running app refers to an app in its state from the time it is launched until it is closed. The app does not need to be active (the app in the foreground); it only needs to be running in the background.

[0025] A device is a device that can capture images of its surroundings and collect sounds from its surroundings. A device may include those used in conjunction with a terminal device, those built into a terminal device, or those used by connecting to a cloud service without directly connecting to a terminal device. In this embodiment, it will be described using the term "meeting device."

[0026] The image of the surroundings of the meeting device acquired by the meeting device is image information acquired by the meeting device capturing the surrounding space (for example, a region or space of 180° to 360° in the horizontal direction), and refers to an image acquired by performing predetermined processing on the curved surface image information captured by the meeting device. The predetermined processing includes various processes for creating surrounding image information from the captured information, such as a flattening process for the captured curved surface image. In addition to the process of creating the surrounding image, the predetermined processing may also include a process of cutting out the speaker image, and a merging process that combines the surrounding image and the speaker image, etc. In this embodiment, the surrounding image is described using the term panoramic image. A panoramic image is an image with a field of view of approximately 180° to 360° in the horizontal direction. The meeting device does not have to capture a panoramic image with a single unit; multiple imaging devices with normal fields of view may be combined. Meeting devices are intended for use at locations such as tables for meetings and to monitor the surrounding environment. However, they may also be used for other purposes, such as monitoring (security, disaster prevention, etc.), supervision (childcare, elder care, etc.), and on-site situation analysis (solutions, marketing, etc.).

[0027] Recorded information refers to information recorded by the information recording application 41, which is stored and made viewable as information linked to the identification information of a particular meeting, and includes, for example, the following information: Video information created based on screen information displayed by the selected application (such as a remote conferencing application), image information of the device's surroundings acquired by the device, etc. • Audio information acquired and synthesized during a meeting using a remote conferencing application (terminal device) and a meeting device at the local office. • Text information obtained by converting the acquired audio into text. Other data and images related to the meeting. For example, document files used during the meeting, additional notes, translated text data, images and stroke data created using a cloud-based electronic whiteboard service during the meeting. When the information recording application 41 records the screen of a remote meeting application or the proceedings of a meeting at a local office, the recorded information may become the minutes of the meeting that took place. Meeting minutes are just one example of recorded information; the name of the recorded information changes depending on the content of the remote meeting or the activities conducted at the local office. For example, it could be called a communication record, a local office status record, etc. The recorded information also includes multiple file formats, such as video files (combined video images, etc.), audio files, text data (text data from speech recognition), document files, image files, and tabular files. Since the files are related to the meeting identification information, they can be viewed chronologically, either all at once or selectively.

[0028] A tenant is a group of users (companies, local governments, and some of these organizations) that have contracted to receive services from a service provider. The creation of record information and its conversion to text data in this embodiment are performed because the tenant has a contract with the service provider.

[0029] Remote communication refers to communication with someone located at a physically distant location, using software and terminal devices to communicate via voice and video. One example of remote communication is a remote conference, which may also be called a meeting, discussion, consultation, contract application, gathering, assembly, seminar, workshop, study group, seminar, training session, etc.

[0030] A base of operations refers to a place that serves as the foundation for activities. An example of a base of operations is a conference room. A conference room is a room set up primarily for the purpose of holding meetings. A base of operations can also be a home, a reception area, a store, a warehouse, or an outdoor work site, as long as there is a place or space where terminal equipment or devices can be installed.

[0031] Speech refers to human speech sounds and ambient sounds, while speech data is the data representation of speech. However, in this embodiment, we will not strictly distinguish between the two.

[0032] <Example System Configuration> Next, the system configuration of the recording information display system 100 will be explained with reference to Figure 2. Figure 2 shows an example of the configuration of the recording information display system 100. In Figure 2, one of the multiple locations where remote meetings are held (local location 102) is shown, and the terminal device 10 at local location 102 communicates with the information processing system 50, the storage service system 70, and the remote meeting service system 90 via the network. A meeting device 60 is also located at local location 102, and the terminal device 10 is connected to this meeting device 60 via a USB cable or the like for communication.

[0033] Terminal device 10 operates at least an information recording application 41 and a remote conferencing application 42. The remote conferencing application 42 can communicate with terminal devices at other locations 101 via a remote conferencing service system 90 on the network, allowing users at each location to hold meetings remotely. The information recording application 41 creates recording information for remote meetings conducted by the remote conferencing application 42 using the functions of the information processing system 50 and the meeting device 60.

[0034] In this embodiment, we describe an example of creating recording information during a remote meeting, but the meeting does not have to involve communication between a single location and a remote location. In other words, the meeting may only include participants from within the same location. In this case, only the audio collected by the meeting device 60 is saved without synthesis, and there is no change in the processing of the information recording application 41.

[0035] The terminal device 10 has a built-in (or external) camera with a normal field of view that captures an image of the front, including the user 107 operating the terminal device 10. A normal field of view is an image that is not a panoramic image, but in this embodiment, it is mainly a planar image that is not curved, such as a 360-degree image. The terminal device 10 also has a built-in (or external) microphone that collects ambient sounds from the user operating the terminal device 10. Therefore, the user can conduct a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42, except for an increase in the processing load of the terminal device 10. The remote conference application 42 can also transmit panoramic images and speaker images captured by the meeting device 60 to the remote conference system 90.

[0036] The information recording application 41 is an application that records information by communicating with the meeting device 60 and creating recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capture device, a microphone, and a speaker. The camera on the terminal device 10 can only capture a limited area in front of it, but the meeting device 60 can capture the entire surrounding area (not necessarily all around) of the meeting device 60. The meeting device 60 can always keep multiple participants 106, as shown in Figure 2, within its field of view.

[0037] In addition, the meeting device 60 extracts speaker images from the panoramic image, and combines the audio acquired by the meeting device 60 with the audio output by the terminal device 10 (including the audio received by the remote conferencing application 42). The meeting device 60 is not limited to being placed on a table or stand; it can be placed anywhere within the local site 102. Since the meeting device 60 can capture a 360-degree image, it can be placed on the ceiling, for example. The meeting device 60 can also be installed at other sites or at any of the sites.

[0038] The information recording application 41 displays a list of applications running on the terminal device 10, saves the recorded information described above, displays recorded videos, accepts editing requests, and so on. The information recording application 41 also displays a list of remote meetings that have been held or are scheduled to be held. The list of remote meetings is used for information related to the recorded information, allowing the user to link remote meetings with the recorded information.

[0039] The remote conferencing application 42 is an application that enables a terminal device to communicate remotely with other terminal devices at other locations 101 through communication connections, transmission and reception of images and audio, display of images, and output of audio. The remote conferencing application can also be called a remote communication application or a remote information sharing application.

[0040] The information recording application 41 and the remote conferencing application 42 may be either web applications or native applications. A web application is an application in which a program on a web server and a program on a web browser or a native application work together to process data, and does not require installation on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used by it. In this embodiment, both will be described as native applications.

[0041] The terminal device 10 may be a general-purpose information processing device equipped with communication functions, such as a PC (Personal Computer), smartphone, or tablet device. Other examples of terminal devices 10 include electronic whiteboards, game consoles, PDAs (Personal Digital Assistants), wearable PCs, car navigation systems, industrial machinery, medical equipment, and network-connected home appliances. The terminal device 10 only needs to be a device on which at least the information recording application 41 and the remote conferencing application 42 can run.

[0042] The information processing system 50 consists of one or more information processing devices located on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and infrastructure services. These server applications manage lists of remote meetings, manage recorded information from remote meetings, manage various settings and storage paths, etc. The infrastructure services handle user authentication, contracts, billing, etc.

[0043] Furthermore, all or part of the functions of the information processing system 50 may reside in a cloud environment or in an on-premises environment. The information processing system 50 may consist of multiple server devices or a single information processing device. For example, server applications and infrastructure services may be provided by separate information processing devices, and furthermore, there may be separate information processing devices for each function within the server application. The information processing system 50 may be integrated with the meeting management system 9, storage service system 70, and speech recognition service system 80 described below.

[0044] The meeting management system 9 is a system that manages information about meetings held in tenants using the information processing system 50. The information processing system 50 retrieves meeting information from the meeting management system 9 and manages this meeting information in association with recorded information.

[0045] The storage service system 70 is a storage means on a network and provides a storage service that accepts the storage of files, etc. Examples of storage service systems 70 include OneDrive®, Google Workspace®, and Dropbox®. The storage service system 70 may also be an on-premises NAS (Network Attached Storage).

[0046] The speech recognition service system 80 provides a service that performs speech recognition on voice data and converts it into text data. The speech recognition service system 80 may be a general-purpose commercial service or it may be part of the functions of the information processing system 50. In addition, the speech recognition service system 80 may be configured and used differently for each user, tenant, or meeting.

[0047] <Example Hardware Configuration> Referring to Figure 3, the hardware configuration of the information processing system 50 and terminal device 10 according to this embodiment will be described.

[0048] <<Information Processing Systems and Terminal Devices>> Figure 3 shows the hardware configuration of an example of an information processing system 50 and terminal device 10 according to this embodiment. As shown in Figure 3, the information processing system 50 and terminal device 10 are built using a computer and include a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0049] Of these, the CPU 501 controls the operation of the entire information processing system 50 and terminal device 10. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. External devices in this case include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 3.

[0050] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0051] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be explained using Figure 4. Figure 4 is an example of a hardware configuration diagram for a meeting device 60 capable of capturing 360° video. In the following, the meeting device 60 is assumed to be a device that uses an image sensor to capture 360° video around the device at a predetermined height, but it may have one or more image sensors. Furthermore, it does not necessarily have to be a dedicated device; a 360° video capture unit can be attached to a PC, digital camera, smartphone, etc., to achieve substantially the same function.

[0052] As shown in Figure 4, the meeting device 60 consists of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection interface 616, a communication unit 617, an antenna 617a, a sound sensor 618, and a recessed terminal 621 for Micro USB.

[0053] Of these, the imaging unit 601 includes a wide-angle lens (a so-called fisheye lens) 602 with a 360° field of view for forming a hemispherical image, and an image sensor 603 provided corresponding to each wide-angle lens. The image sensor 603 has an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lens 602 into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set. The imaging unit 601 may also be a 360° camera, and is an example of an imaging means capable of capturing images of 360° around the meeting device 60. Alternatively, multiple pieces of information acquired by multiple image sensors (e.g., 180° + 180°) may be combined to create a 360° field of view.

[0054] Each image sensor 603 of the imaging unit 601 is connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensor 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via bus 610. Furthermore, ROM 612, SRAM 613, DRAM 614, operation unit 615, external device connection I / F 616, communication unit 617, and sound sensor 618 are also connected to bus 610.

[0055] The image processing unit 604 receives image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates panoramic images and speaker images from the fisheye video. Furthermore, the image processing unit 604 combines the panoramic image and speaker images, etc., to output a single video.

[0056] The imaging control unit 605 generally uses the I2C bus to set commands and other information in the registers of the image sensor 603, with the imaging control unit 605 acting as the master device and the image sensor 603 as the slave device. It receives necessary commands and other information from the CPU 611. The imaging control unit 605 also uses the I2C bus to acquire status data and other information from the registers of the image sensor 603 and send it to the CPU 611.

[0057] Furthermore, the imaging control unit 605 instructs the image sensors 603a and 603b to output image data when the image capture start button on the operation unit 615 is pressed or when an image capture start instruction is received from the PC. Depending on the meeting device 60, there may also be functions to display a preview or video on a display (for example, the display of a PC or smartphone). In this case, the image data output from the image sensor 603 is performed continuously at a predetermined frame rate (frames / minute).

[0058] Furthermore, as will be described later, the imaging control unit 605 also functions as a synchronization control means that works in cooperation with the CPU 611 to synchronize the output timing of image data from the image sensor 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0059] Microphone 608 converts sound into sound (signal) data. Sound processing unit 609 receives the sound data output from microphone 608 through the I / F bus and performs predetermined processing on the sound data.

[0060] The CPU 611 controls the overall operation of the meeting device 60 and performs necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in progress. In particular, the DRAM 614 stores image data in progress and processed equirectangular projection image data from the image processing unit 604.

[0061] The control unit 615 is a collective term for the control buttons, such as the image capture start button 615a. By operating the control unit 615, the user can start imaging and recording, turn the power on / off, establish a communication connection, and input various settings such as imaging modes and imaging conditions.

[0062] The external device connection interface 616 is an interface for connecting various external devices. These external devices include, for example, a PC (Personal Computer), a display, a projector, and an electronic whiteboard. The external device connection interface 616 may also include, for example, a USB terminal or an HDMI® terminal. Video and image data stored in the DRAM 614 are transmitted to an external terminal or recorded on external media via this external device connection interface 616. Furthermore, multiple external device connection interfaces 616 may be used; for example, image information captured by the meeting device 60 may be transmitted to a PC via USB for recording, while simultaneously acquiring video (e.g., screen information displayed in a remote conferencing application) from the PC to the meeting device 60, and then transmitted from the meeting device 60 via HDMI to other external devices (displays, projectors, electronic whiteboards, etc.) for display.

[0063] The communication unit 617 may communicate with a cloud server via the internet using wireless communication technology such as Wi-Fi through the antenna 617a provided on the meeting device 60, and transmit the stored video data and image data to the cloud server. Alternatively, the communication unit 617 may communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy; registered trademark) or NFC.

[0064] The audio sensor 618 is a sensor that acquires 360° audio information in order to determine from which direction loud audio is being input within a 360° radius (horizontal plane) around the meeting device 60. The audio processing unit 609 identifies the direction of the strongest sound based on the input 360° audio parameters and outputs the audio input direction within the 360°.

[0065] Furthermore, other sensors (such as compass / accelerometers or GPS) can be used to calculate direction, position, angle, acceleration, etc., and these can be used for image correction or to add location information.

[0066] The image processing unit 604 also performs the following processing:

[0067] The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing, such as Bayer conversion (RGB interpolation processing), on the RAW data input from the image sensor that inputs a spherical image to create a fisheye image (an image consisting of curved images). Furthermore, it performs planarization processing, such as DeWarp processing (distortion correction processing), on the created fisheye image (a curved image) to create a panoramic image (an image consisting of flat images) that captures 360° around the meeting device 60.

[0068] The CPU 611 creates speaker images in the following way: The CPU 611 creates speaker images by extracting the speaker from a panoramic image (flat image) that captures the surrounding 360°. The CPU 611 uses the direction of the audio input, identified from the 360° output by the audio sensor 618 and the audio processing unit 609, as the direction of the speaker, and extracts the speaker image from the panoramic image. In this case, the method for extracting a person's image from the direction of the audio input is to cut out 30° around the audio direction identified from the 360°, and then perform face detection within that area to extract the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., 3 people) who have spoken most recently from the extracted speaker images.

[0069] The panoramic image and the images of one or more speakers may be sent individually to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the images of one or more speakers are sent individually from the meeting device 60 to the information recording application 41.

[0070] Figure 5 illustrates the imaging range of the meeting device 60. As shown in Figure 5(a), the meeting device 60 images a 360° range in the horizontal direction. As shown in Figure 5(b), the meeting device 60 has an imaging range that is 0° horizontal to the height of the meeting device 60, and a predetermined angle above and below it.

[0071] Figure 6 illustrates the extraction of panoramic and speaker images. As shown in Figure 6, the image captured by the meeting device 60 forms a part 110 of a sphere and therefore has a three-dimensional shape. As shown in Figure 5(b), the meeting device 60 divides the field of view into predetermined angles vertically and horizontally and performs perspective projection transformation. By performing perspective projection transformation seamlessly across the entire 360° horizontal direction, a predetermined number of planar images are obtained, and by stitching these predetermined number of planar images together horizontally, a panoramic image 111 is obtained. The meeting device 60 also performs face detection from the panoramic image within a predetermined range centered on the direction of sound, and creates speaker images 112 by extracting 15° to the left and right (30° in total) from the center of the face.

[0072] <About the features> Next, with reference to Figure 7, the functional configuration of the recorded information display system 100 will be described. Figure 7 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recorded information display system 100 by dividing them into blocks.

[0073] <<Terminal device>> The information recording application 41 operating on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a video storage unit 17, an audio data processing unit 18, a recorded video display unit 19, an upload unit 20, an editing processing unit 21, and a search unit 22. Each of these units in the terminal device 10 is a function or means of function realized by any of the components shown in Figure 3 operating according to instructions from the CPU 501 in accordance with the information recording application 41 deployed from the HD 504 to the RAM 503. The terminal device 10 also has a storage unit 1000 constructed from the HD 504, etc., as shown in Figure 3. An information storage unit 1001 is constructed in the storage unit 1000.

[0074] The communication unit 11 communicates various types of information with the information processing system 50 via the network. For example, the communication unit 11 receives a list of remote meetings from the information processing system 50 and sends voice data recognition requests to the information processing system 50.

[0075] The display control unit 13 displays various screens that serve as the user interface in the information recording application 41 according to the screen transitions set in the information recording application 41. The operation reception unit 12 receives various operations on the information recording application 41.

[0076] The application screen acquisition unit 14 acquires screen information displayed by the application selected by the user, or screen information of the desktop screen, etc., from the OS (Operating System), etc. If the application selected by the user is the remote conferencing application 42, the screen generated by the remote conferencing application 42 is obtained (images of terminal device users captured by terminal device cameras at each location, images of shared documents displayed, images including participant icons and participant names, etc.). The screen information displayed by the application (application screen) is information displayed as a window by a running application, including the remote conferencing application, etc., and acquired as an image by the information recording application. The application window is drawn as an area of ​​the window within the entire desktop image and displayed on a monitor, etc. The screen information displayed by the application can be acquired by other applications (information recording applications, etc.) as an image file or a video file composed of a series of images via the OS (Operating System) API or the API of the application that displays it. Similarly, the screen information of the desktop screen is information composed of images of the desktop screen generated by the OS, and can be acquired as an image file or video file via the OS API. The format of these image files may be bitmap, PNG, or other formats. The video file format can be MP4 or other formats.

[0077] The audio acquisition unit 15 acquires audio output from the microphone or earphones of the terminal device 10 (including audio data received from the remote conferencing application 42 during a remote conference). The audio acquisition unit 15 can acquire audio even if the output audio is muted. The user does not need to perform any operations such as selecting the remote conferencing application 42 regarding audio data; the audio acquisition unit 15 can acquire the audio that the terminal device 10 can output via the OS or application's API (Application Interface). This also allows the audio data received by the remote conferencing application 42 from other locations 101 to be acquired. If the remote conferencing application 42 is not running or a remote conference is not in progress, the information recording application 41 may not be able to acquire audio data. Note that the audio acquired by the audio acquisition unit 15 may not include the audio collected by the terminal device 10, but only the output audio data. This is because the meeting device 60 separately collects audio from the locations.

[0078] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 using wireless LAN, Bluetooth (registered trademark), etc. The device communication unit 16 receives panoramic images and speaker images from the meeting device 60 and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0079] The video storage unit 17 saves the panoramic image received by the device communication unit 16, the speaker image, and the application screen acquired by the application screen acquisition unit 14 as separate recorded videos. The video storage unit 17 also saves the synthesized audio data separately. The video storage unit 17 may combine the audio data with the panoramic image to create a panoramic image with audio.

[0080] The audio data processing unit 18 requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

[0081] The recorded video display unit 19 displays the recorded video. The recorded video is saved in the terminal device 10 during recording and then uploaded to the information processing system 50.

[0082] When the remote conference ends, the upload unit 20 transmits the recorded video and audio data to the information processing system 50.

[0083] The editing processing unit 21 performs editing of the recorded video (such as deleting parts or splicing them together) in response to user operations.

[0084] The search unit 22 searches text data using keywords entered by the user. For some of the text data that matches the search, the time elapsed since the start of the meeting when the statement was made is also identified.

[0085] Figure 8 shows the video recording information stored in the information storage unit 1001. The video recording information includes items such as meeting ID, recording ID, update date and time, title, upload, and storage location. When a user logs into the information processing system 50, the information recording application 41 can download meeting information obtained by the information processing system 50 from the meeting management system 9, as well as application-side meeting information held by the information processing system 50. The meeting ID and other information included in the meeting information are reflected in the video recording information. The video recording information in Figure 8 is held by a terminal device 10 operated by a particular user.

[0086] The meeting ID is identification information that identifies a remote meeting that has been held. The meeting ID is assigned when the remote meeting is scheduled in the meeting management system 9, or it is assigned by the information processing system 50 upon request from the information recording application 41. The meeting management system 9 is a system for registering meeting and remote meeting schedules, URLs (meeting links) for starting remote meetings, and reservation information for equipment used in meetings. It is a scheduler, etc., that is connected to from the terminal device 10 via the network. The meeting management system 9 can also transmit the registered schedules, etc., to the information processing system 50.

[0087] The recording ID is identification information that identifies recorded video footage from a remote meeting. The recording ID is assigned by the meeting device 60, but may also be assigned by the information recording application 41 or the information processing system 50. The same meeting ID may be assigned to a different recording ID if the recording of the remote meeting was stopped midway through but resumed for some reason.

[0088] The update date and time is the date and time when the recorded video was updated (recording ended). If the recorded video was edited, it will be the date and time when it was edited.

[0089] The title is the name of the meeting. This may be set when registering the meeting in the meeting management system 9, or it may be set arbitrarily by the user.

[0090] • Upload indicates whether or not the recorded video has been uploaded to the information processing system 50.

[0091] The storage location indicates the location (URL or file path) where the recorded video and text data are stored in the storage service system 70. Therefore, users can view the uploaded recorded video at their discretion. Note that panoramic images, speaker images, app screens, and text data are saved with separate filenames following the URL, for example.

[0092] <<Meeting Devices>> Let's return to Figure 7 for explanation. The meeting device 60 includes a terminal communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, and a speech synthesis unit 65. Each of these units in the meeting device 60 is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 611 in accordance with a program expanded from the ROM 612 to the DRAM 614.

[0093] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may also communicate with the terminal device 10 using wireless LAN, Bluetooth (registered trademark), or the like.

[0094] The panoramic image creation unit 62 creates a panoramic image. The speaker image creation unit 63 creates a speaker image. The methods for creating these images are explained in Figures 5 and 6.

[0095] The sound collection unit 64 converts the audio signal acquired by the microphone of the meeting device 60 into audio data (digital). This allows the content spoken by users and participants at the terminal device 10 to be collected.

[0096] The speech synthesis unit 65 synthesizes the speech transmitted from the terminal device 10 with the speech collected by the sound collection unit 64. This combines speech spoken at other locations 101 with speech spoken at the local location 102.

[0097] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a conference information acquisition unit 54, and a text conversion unit 55. Each of these units in the information processing system 50 is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 501 following a program deployed from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed from the HD 504 and other components shown in Figure 3. The storage unit 5000 contains an application-side conference information storage unit 5001 and a recording information storage unit 5002.

[0098] The communication unit 51 sends and receives various types of information with the terminal device 10. For example, the communication unit 51 may send a list of remote meetings to the terminal device 10 or receive a voice data recognition request from the terminal device 10.

[0099] The authentication unit 52 authenticates the user operating the terminal device 10. The authentication unit 52 authenticates the user, for example, by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, biometric authentication information such as a face or fingerprint, etc. The authentication unit 52 may also authenticate using an external authentication system or an authentication method such as OAuth.

[0100] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 runs a native application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc.

[0101] The meeting information acquisition unit 54 acquires meeting information from the meeting management system 9 using each user's account or a system account assigned to the information processing system 50. The meeting information acquisition unit can acquire a list of remote meetings that users belonging to the tenant have viewing rights to. Viewing rights may also be granted directly from the information recording application of the terminal device 10 to the meeting information managed by the meeting management unit 54. The information on remote meetings that users belonging to the tenant have viewing rights to includes information on meetings created by the user and information on meetings to which the user has been granted viewing rights by other users. Since a meeting ID is assigned to each remote meeting, the remote meeting and the recorded information are associated using the meeting ID.

[0102] The text conversion unit 55 converts the voice data requested by the terminal device 10 to be converted to text data into text data using an external voice recognition service. The text conversion unit 55 may also perform the conversion itself.

[0103] Figure 9 shows an example of application-side meeting information stored in the application-side meeting information storage unit 5001. Application-side meeting information is meeting information on the information recording application 41 side that is created separately from the meeting information when the information recording application 41 creates the recording information.

[0104] The meeting ID is the identification information for the meeting. The meeting ID links the meeting information managed by the meeting management system 9 with the meeting information on the application side.

[0105] The app ID is the identification information for the information recording app 41.

[0106] The user ID to which the access token is issued is the identification information of the storage service system 70 that issues the access token.

[0107] The URL of the access token issuer is the URL of the storage service system 70 that issues the access token.

[0108] The issuance date and time is the date and time the access token was issued. The issuance date and time is determined at the time of issuance.

[0109] • The effective date and time is the date and time when the access token becomes valid. The effective date and time are determined by the issuer.

[0110] • The expiration date is the expiration date of the access token. The expiration date is determined by the issuer.

[0111] • Permission information refers to the permissions a user has to perform using an access token. Permission information is determined by the issuer.

[0112] • The display name is the user's display name in the information recording application 41.

[0113] • The last name is the user's last name.

[0114] • The name is the user's name.

[0115] Figure 10 shows the recording information stored in the recording information storage unit 5002. The recording information contains a list of recorded videos recorded by all users belonging to the tenant. The recording information includes items such as meeting ID, recording ID, update date and time, title, and storage location. These items may be the same as in Figure 8. The storage location (path information such as the URL of the cloud storage system) may be entered by the user in the user settings screen of the information recording application 41 of the terminal device 10 and stored in the recording information storage unit 5002.

[0116] <<Meeting Management System>> Let's return to Figure 7 for explanation. The conference management system 9 has a conference information management unit 31. Each of these units in the conference management system 9 is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 501 in accordance with a program deployed from the HD 504 to the RAM 503. The conference management system 9 also has a storage unit constructed by the HD 504, etc., shown in Figure 3. A conference information storage unit 3001 is constructed in the storage unit 3000.

[0117] The meeting information management unit 31 manages meeting information held by tenants. The meeting information is stored in the meeting information storage unit 3001.

[0118] Figure 11 shows an example of meeting information stored in the meeting information storage unit 3001. The meeting information is managed by a meeting ID and has the items shown in the figure.

[0119] The meeting ID is the identification information for the meeting.

[0120] • The tenant ID is the identifying information for the tenant where the meeting is being held.

[0121] The title is the subject or agenda of the meeting.

[0122] The organizer is the person who organized the meeting.

[0123] • Participants is a list of those invited to the meeting.

[0124] • Users with viewing privileges are a list of users who have access to meeting resources, including recorded videos.

[0125] • Ad-hoc participants are a list of guest participants.

[0126] • The location is information about the meeting room, such as its name.

[0127] • The start time is the scheduled time when the meeting will begin.

[0128] • The end time is the scheduled time when the meeting will conclude.

[0129] The meeting creator is the user ID, etc., who registered the meeting information.

[0130] The password is the password that participants use to log in to the meeting.

[0131] <<Storage Service System>> The storage service system 70 can be any service system that stores information. Here, we will describe the structure of the text data stored in the storage service system 70.

[0132] Figure 12 illustrates the structure of text data stored in the storage service system 70. As shown in Figure 12, the text data is associated with the following items: ID, time, user, and text. The ID is identification information assigned when the text data is divided into text segments according to a predetermined rule. The predetermined rule is set in the speech recognition service system 80 and may include, for example, dividing the text after a certain period of silence, forcibly dividing it after a certain period of time even without silence, or dividing it into sentences detected by morphological analysis. Time is the speaking time based on the duration from the start of recording. Since the so-called time is also saved at the start of recording, the time (absolute time) when the text string was spoken can also be determined. User indicates whether the speech is from the local location or another location, as determined by sound pressure, etc. The text string is a part of the divided text data.

[0133] Since time and text are associated in this way, when text data matches the search criteria, the information recording app 41 can display the recorded video from the time corresponding to this time.

[0134] <Screen transition> Next, with reference to Figures 13 to 22, several screens displayed by the terminal device 10 during a remote conference will be described. Figure 13 is the initial screen 200 displayed after login by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user enters authentication information and successfully logs in, the initial screen 200 of Figure 13 is displayed.

[0135] The initial screen 200 has a fixed display button 201, a front view change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker image 204 when not distinguished), and a recording start button 205. If the meeting device 60 is already running and capturing images of the surroundings when the user logs in, the panoramic image 203 and speaker images 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether or not to start recording while viewing these images. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker images 204 are not displayed.

[0136] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or it may display only the speaker images 204 of the N people who have spoken most recently. Figure 13 shows an example where up to three speaker images 204 are displayed. There may be no speaker images 204 until a participant speaks (one is added each time a participant speaks), or three speaker images 204 of participants facing a predetermined direction may be displayed (they are swapped depending on the participant's speech).

[0137] If no one is speaking, such as immediately after the meeting device 60 is activated, a speaker image 204 will be created in a predetermined direction within the horizontal 360° (e.g., 0°, 120°, 240°). If a fixed display setting (described later) is enabled, that setting will take precedence.

[0138] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 as the speaker image 204 and zoom in on it.

[0139] Figure 14 illustrates the operation method when the fixed display button 201 is turned on. For example, the user moves a rectangular window 206 on the panoramic image 203 using a pointing device such as a mouse or touch panel. The user aligns the window 206 with a whiteboard, podium, or other elements included in the panoramic image 203. The user's operation is transmitted to the meeting device 60, which creates an image of the area indicated by the window within a horizontal 360°, at the same size as the speaker image 204, and transmits it to the terminal device 10. In this way, the speaker image 204 can continue to display elements other than the speaker, such as the whiteboard.

[0140] Returning to Figure 13, the front view change button 202 is a button for the user to change the front view of the panoramic image 203 (since the panoramic image captures 360° horizontally, the right and left edges coincide in terms of direction). The user can slide the panoramic image 203 left or right with the pointing device to determine which participant is in the front view. The user's operation is sent to the meeting device 60, which changes the angle that is considered the front view out of the 360° horizontal direction to create a panoramic image and sends it to the terminal device 10.

[0141] When the user presses the recording start button 205, the information recording application 41 displays the recording settings screen 210 shown in Figure 16.

[0142] If the meeting device 60 is not connected or powered off when the information recording app 41 is launched, the device unrecognized screen 250 shown in Figure 15 will be displayed.

[0143] Figure 15 shows an example of the device unrecognized screen 250. The device unrecognized screen 250 displays the message 251, "The device could not be recognized. Please power on the device and connect it." The user can see this to check the power and connection status of the meeting device 60.

[0144] Figure 16 shows an example of the recording settings screen 210 displayed by the information recording application 41. On the recording settings screen 210, the user can set whether to record (include in recording) the panoramic image and speaker image created by the meeting device 60, as well as the desktop screen or running application screen of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or running application screen, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) will be recorded.

[0145] The camera toggle button 211 is a button that switches the recording of panoramic images and speaker images created by the meeting device 60 on and off. The camera toggle button 211 may also be configured to record panoramic images and speaker images individually.

[0146] The PC screen toggle button 212 is a button that switches the recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10 on and off. When the PC screen toggle button 212 is on, the desktop screen is recorded.

[0147] If a user wants to record an app screen, they further select an app from the app selection field 213. The app selection field 213 displays the names of apps currently running on the terminal device 10 in a dropdown format. The user can select the app to record. This app name is obtained by the information recording app 41 from the OS. The information recording app 41 can display only apps that have a UI (user interface) among the running apps. The remote conferencing app 42 may be included in the selected apps. Therefore, the information recording app 41 can also record the materials displayed by the remote conferencing app 42 and the participants at each location in the video. In addition, the apps displayed in the dropdown are various apps running on the terminal device, such as presentation apps, word processing apps, spreadsheet apps, document creation and editing apps, cloud electronic whiteboard apps, and web browser apps. Therefore, the user can flexibly select which app screens to include in the recorded video.

[0148] Furthermore, when recording on an app-by-app basis, users can select multiple apps. The information recording app 41 can record the screens of all selected apps.

[0149] If both the camera toggle button 211 and the PC screen toggle button 212 are turned off, the recording content confirmation window 214 will display "Only audio will be recorded." This audio consists of the audio output by the terminal device 10 (audio received by the remote conferencing application 42 from other locations 101) and the audio collected by the meeting device 60. In other words, if a remote conference is being held, the audio from the remote conferencing application 42 and the audio from the meeting device 60 will be saved regardless of image recording. However, the user may selectively stop saving the audio from the remote conferencing application 42 and the audio from the meeting device 60 in their settings.

[0150] Depending on the on / off combination of the camera toggle button 211 and the PC screen toggle button 212, the recorded video will be recorded as follows. In addition, the recorded video will be displayed in real time in the recording content confirmation window 214.

[0151] If the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image will be displayed in the recording content confirmation window 214.

[0152] If the camera toggle button 211 is off and the PC screen toggle button 212 is on (screen is also selected), the desktop screen or the screen of the selected application will be displayed in the recording content confirmation window 214.

[0153] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the recording content confirmation window 214 displays the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, side by side.

[0154] Therefore, there may be cases where the panoramic image, speaker image, and app screen are not recorded, or where none of the panoramic image, speaker image, and app screen are recorded at all. However, in this embodiment, for convenience, the images created by the information recording app 41 are referred to as recorded videos.

[0155] Figure 17 shows an example of the recording content confirmation window 214 when the camera toggle button 211 is on and the PC screen toggle button 212 is off. In Figure 17, the panoramic image 203 and the speaker image 204 are displayed prominently.

[0156] Figure 18 shows an example of the recording content confirmation window 214 when the camera toggle button 211 and the PC screen toggle button 212 are turned on. In Figure 18, the panoramic image 203 and speaker image 204 are displayed on the left, and the application screen 217 is displayed on the right.

[0157] Therefore, the recording content confirmation window 214 allows the user to check, before recording starts, what kind of content will be recorded in the video (especially the images from the meeting device 60) according to the settings on the recording settings screen 210.

[0158] Figure 18 shows an example of how the recorded video is displayed when only one app is selected. However, if two or more apps are selected, the screens of the second and subsequent apps will be sequentially linked to the right. Alternatively, the screens of the second and subsequent apps may be arranged in a two-dimensional vertical and horizontal layout.

[0159] Let's return to Figure 16 for explanation. The recording settings screen 210 has a checkbox 215 along with the message, "Automatically transcribe the recording after uploading it." The recording settings screen 210 also has a "Start Recording Now" button 216. When the user checks the checkbox 215, the recorded video will have text data of the spoken words during the remote meeting attached to it. In this case, after the recording is finished, the information recording application 41 uploads the audio to the information processing system 50 along with a request for conversion to text data. Also, when the user presses the "Start Recording Now" button 216, the recording screen 220 in Figure 19 is displayed.

[0160] Figure 19 shows an example of the recording screen 220 displayed by the information recording application 41 during recording. The explanation of Figure 19 mainly focuses on the differences from Figure 16. The recording screen 220 displays the recorded video under the conditions set by the user in the recording settings screen 210. In Figure 19, the recording screen 220 is shown with the camera toggle button 211 on and the PC screen toggle button 212 off, and displays the panoramic image 203 and speaker image 204 (both videos) created by the meeting device 60. The recording screen 220 also displays the recording icon 225, the pause button 226, and the recording stop button 227.

[0161] Furthermore, if the user sets the PC screen toggle button 212 to ON, the recording screen 220 will display the panoramic image and speaker image alongside the desktop screen and application screen, as shown in Figure 18.

[0162] The pause button 226 is used to stop recording, and after stopping, it also accepts requests to resume recording. The recording end button 227 is used to end recording. The recording ID does not change when the pause button 226 is pressed, but it does change when the recording end button 227 is pressed. After pausing, the user can also set the recording conditions that were set on the recording settings screen 210 again when resuming recording. In that case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, when the recording end button 227 is pressed), or it may concatenate multiple files to form a single continuous video (for example, when the pause button 226 is pressed). Also, when the information recording application 41 displays the recorded video, it may display multiple recording files consecutively as a single video.

[0163] The recording screen 220 also has a "Get Information from Calendar" button 221, a meeting name field 222, a time field 223, and a location field 224. The "Get Information from Calendar" button 221 is a button for the user to retrieve meeting information from the meeting management system 9. When the "Get Information from Calendar" button 221 is pressed, the information recording application 41 retrieves and displays a list of meetings that the user has permission to view from the information processing system 50. The user selects the remote meeting to be held from the meeting list. The information recording application 41 can also retrieve meeting information from the application itself. As a result, the meeting information is reflected in the meeting name field 222, the time field 223, and the location field 224. The meeting name field 222 displays the title of the meeting information, the time field 223 displays the start and end times, and the location field 224 displays the location. In addition, the remote meeting and the recording information in the meeting management system 9 are associated with the meeting ID.

[0164] Once the remote meeting ends and the user stops recording, a video recording with audio is created.

[0165] Figure 20 shows an example of the meeting list screen 230 displayed by the information recording application 41. The meeting list screen 230 is a list of meetings, and can display a list of recorded information from remote meetings. It also includes meetings held only within a specific meeting room, regardless of whether they were remote meetings. The meeting list screen 230 displays a combined display of meeting information that the logged-in user has permission to view in the meeting information storage unit 3001, the application-side meeting information, and the information stored in the recording information storage unit 5002 associated with this remote meeting. Information from video recordings stored in the information storage unit 1001 may be further integrated.

[0166] The meeting list screen 230 is displayed when the user selects the meeting list tab 231 on the initial screen 200 in Figure 13. The meeting list screen 230 displays a list 236 of recorded information that the user has permission to view. The meeting creator (minutes creator) can set viewing permissions for participants. The meeting list may be a list of saved recorded information, or a list of meeting schedules or meeting data.

[0167] The meeting list screen 230 has the following items: checkbox 232, update date and time 233, title 234, and status 235.

[0168] Checkbox 232 accepts the selection of recording files. Checkbox 232 is used when the user wants to delete multiple recording files at once.

[0169] • The update time 233 indicates the start and end times of the recorded video. If the video has been edited, the edit time can be used instead.

[0170] • The title is 234, which is the title of the meeting (agenda, etc.). It can be copied from the meeting information or set by the user.

[0171] Status 235 indicates whether the recorded video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed; if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, the upload button is displayed. If there are unuploaded recorded videos, the information recording application 41 should automatically upload them when the user logs into the information processing system 50.

[0172] When the user selects a title or other item from the list of recorded videos 236 using the pointing device, the information recording application 41 displays the recorded video display screen 240 shown in Figure 21. The recorded video display screen 240 allows the user to view the recorded video and perform other actions.

[0173] Figure 21 shows an example of the recorded video display screen 240 displayed by the information recording application 41 after selecting a recorded video. The recorded video display screen 240 has an image display area 241, a transcription button 242, one or more text display areas 243, an auto-scroll button 244, and a search button 245.

[0174] The image display area 241 includes a display button 241a, a rewind button 241b, a fast-forward button 241c, a time indicator 241d, a display speed button 241e, a volume button 241f, etc. The image display area 241 displays the recorded video. In Figure 21, the recorded video in the image display area 241 has a panoramic image and a speaker image on the left, and the screen of the remote conferencing application 42 is displayed on the right. The screen of the remote conferencing application 42 changes to an image of the location or an image of the document during the remote conferencing. Therefore, the user can view the screen of the desired scene by operating various buttons.

[0175] If the audio data of the recorded video displayed in image display area 241 has been converted to text data, the spoken content will be displayed in text display area 243. Text display area 243 displays the text data converted from the synthesized audio data by speech recognition.

[0176] The transcription button 242 is a button that allows the user to toggle whether or not the text data displayed in the text display field 243 is displayed in accordance with the display time of the recorded video.

[0177] The automatic scroll button 244 is a button that allows the user to toggle whether or not the text data is automatically scrolled, regardless of the display time.

[0178] • Search button 245 is a button that allows users to search text data by specifying keywords. Further details will be provided later.

[0179] Additionally, the recorded video display screen 240 may allow for the download of recorded videos.

[0180] Figure 22 shows an example of the video editing screen 260. The editing screen 260 is accessed automatically from the recording screen 220, or when the user performs a predetermined operation on the video recording display screen 240. The editing screen 260 has a first display area 261 and a second display area 262. The first display area 261 displays the video recording at a given moment, and the second display area 262 displays the frames that make up the video recording in chronological order. The user can select one or more frames to delete unnecessary frames. The user can also extract a portion of a frame and insert it after any frame. The editing processing unit 21 edits the video recording according to the user's operation and either overwrites the edited video or saves it separately.

[0181] <Procedure for operation or processing> Next, we will explain the operations and processes performed by the recorded information display system 100 based on the above configuration.

[0182] <<Saving recorded video>> Refer to Figure 23 to explain the process of saving the recorded video. Figure 23 is an example of a sequence diagram showing the procedure by which the information recording application 41 records a panoramic image, a speaker image, and the application screen.

[0183] S1: The user launches the information recording application 41 on the terminal device 10 and connects the terminal device 10 (information recording application 41) to the information processing system 50. If the access token has expired, the display control unit 13 displays the login screen. The user enters authentication information (e.g., user ID, password) to log in to the tenant into the information recording application 41. The operation reception unit 12 of the information recording application 41 accepts the input.

[0184] S2: The communication unit 11 of the information recording application 41 sends a login request to the information processing system 50, specifying the authentication information.

[0185] S3: The communication unit 51 of the information processing system 50 receives a login request, and the authentication unit 52 authenticates the user based on the authentication information. Here, we assume that authentication was successful. The communication unit 51 of the information processing system 50 sends the access token 1 to the information recording application 41. Although not explicitly shown in the diagram, the communication unit 51 will subsequently attach the access token 1 to communications with the information processing system 50. The access token 1 is associated with the permissions of the logged-in user.

[0186] S4: Similarly, the user also logs into the storage service system 70. This is because the recorded information is stored in the storage service system 70. The user enters authentication information (e.g., user ID, password) to log into the storage service system 70. The operation reception unit 12 of the information recording application 41 receives the input.

[0187] S5: The communication unit 11 of the information recording application 41 sends a login request to the information processing system 50, specifying the authentication information.

[0188] S6: The communication unit 51 of the information processing system 50 receives a login request, and since it is a login request to the storage service system 70, it forwards the login request to the storage service system 70.

[0189] S7: The storage service system 70 authenticates the user based on the authentication information. Here, we assume that authentication was successful. The storage service system 70 sends the access token 2 to the information processing system 50.

[0190] S8: The communication unit 51 of the information processing system 50 receives the access token 2 and sends it to the information recording application 41. Although not explicitly shown in the diagram, the communication unit 51 subsequently attaches the access token 2 to its communication with the storage service system 70. The access token 2 is associated with the permissions of the logged-in user.

[0191] S21: Next, the user operates the remote conferencing application 42 to start a remote conference. Here, it is assumed that the remote conferencing applications 42 at the local site 102 and the other site 101 have started a remote conference. The remote conferencing application 42 at local site 102 transmits the image captured by the camera and the audio collected by the microphone of the terminal device 10 to the remote conferencing application 42 at the other site 101. The remote conferencing application 42 at the other site 101 displays the received image on its display and outputs the received audio from its speaker. Similarly, the remote conferencing application 42 at the other site 101 transmits the image captured by the camera and the audio collected by the microphone of the terminal device 10 to the remote conferencing application 42 at local site 102. The remote conferencing application 42 at local site 102 displays the received image on its display and outputs the received audio from its speaker. Each remote conferencing application 42 repeats this process to realize a remote conference.

[0192] S22: The user configures the recording settings on the recording settings screen 210 of the information recording application 41 shown in Figure 16. The operation reception unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that both the camera toggle button 211 and the PC screen toggle button 212 are turned on.

[0193] If a user has already scheduled a remote meeting, they can display a list of remote meetings by pressing the information acquisition button 221 on the calendar in Figure 19 and select the remote meeting to which they want to associate a recorded video. Since the user is logged into the information processing system 50, the information processing system 50 identifies the remote meetings that the logged-in user has access to. The information processing system 50 sends the list of identified remote meetings to the terminal device 10, and the user selects a remote meeting that is currently in progress or will be held soon. This determines the information related to the remote meeting, such as the meeting ID.

[0194] Furthermore, users can create a meeting when creating a recorded video, even if they haven't scheduled a remote meeting in advance. The following describes how the information recording application 41 creates a meeting when creating a recorded video and obtains the meeting ID from the information processing system 50.

[0195] S23: The user instructs the information recording application 41 to start recording (by pressing the "Start Recording Now" button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0196] S24: Since remote conferencing is not selected (because the conferencing ID has not been determined), the communication unit 11 of the information recording application 41 sends a remote conferencing creation request to the information processing system 50.

[0197] S25: The communication unit 51 of the information processing system 50 receives a remote meeting creation request, the meeting information acquisition unit 54 obtains a unique meeting ID assigned by the meeting management system 9, and the communication unit 51 transmits the meeting ID to the information recording application 41.

[0198] S26: The conference information acquisition unit 54 also transmits the destination of the recorded video (URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

[0199] S27: The communication unit 11 of the information recording application 41 receives the conference ID and the location where the recording file will be saved, causing the video storage unit 17 to determine that it is ready to record and to start recording.

[0200] S28: The application screen acquisition unit 14 of the information recording application 41 requests the application screen of the application selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In Figure 23, the application selected by the user is the remote conferencing application 42.

[0201] S29: The video storage unit 17 of the information recording application 41 notifies the meeting device 60 to start recording via the device communication unit 16. When notifying, the video storage unit 17 may also notify that the camera toggle button 211 is on (requesting panoramic images and speaker images). Regardless of whether a request is made, the meeting device 60 sends panoramic images and speaker images to the information recording application 41.

[0202] S30: When the terminal communication unit 61 of the meeting device 60 receives a recording start notification, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or obtained from the information processing system 50.

[0203] S31: The audio acquisition unit 15 of the information recording application 41 acquires the audio data output by the terminal device 10 (audio data received by the remote conferencing application 42).

[0204] S32: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and the synthesis request to the meeting device 60.

[0205] S33: The terminal communication unit 61 of the meeting device 60 receives voice data and a synthesis request, and the voice synthesis unit 65 synthesizes the received voice data with the surrounding voice data collected by the sound collection unit 64. For example, the voice synthesis unit 65 adds the two voice data together. Since clear sound from the surroundings of the meeting device 60 is recorded, the accuracy of text transcription of voices, especially those around the meeting device 60 (on the conference room side), is improved.

[0206] This voice synthesis can also be performed on the terminal device 10. However, by distributing the recording function to the terminal device 10 and the voice processing to the meeting device 60, the load on both the terminal device 10 and the meeting device 60 can be reduced. Alternatively, the recording function may be distributed to the meeting device 60 and the voice processing to the terminal device 10.

[0207] S34: In addition, the panoramic image creation unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creation unit 63 creates a speaker image.

[0208] S35: The device communication unit 16 of the information recording application 41 repeatedly acquires panoramic images and speaker images from the meeting device 60. The device communication unit 16 also repeatedly requests and acquires synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making requests to the meeting device 60. Alternatively, the meeting device 60 may automatically transmit the panoramic images and speaker images upon receiving a request to synthesize audio data. The meeting device 60 may automatically transmit the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0209] S36: The display control unit 13 of the information recording application 41 displays the application screen, panoramic image, and speaker image side by side on the recording screen 220. The video storage unit 17 of the information recording application 41 also saves the application screen, panoramic image, and speaker image acquired from the remote conferencing application 42 as separate recorded videos. In other words, the video storage unit 17 creates each recorded video by specifying the repeatedly received application screen, panoramic image, and speaker image as frames that make up the video. The video storage unit 17 also stores the audio data received from the meeting device 60.

[0210] The information recording application 41 repeats the above steps S31 to S36.

[0211] S37: When the remote meeting ends and recording is no longer needed, the user instructs the information recording application 41 to stop recording (for example, by pressing the recording stop button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0212] S38: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. The meeting device 60 continues to create panoramic images and speaker images and synthesize audio. However, the meeting device 60 may change the processing load, such as changing the resolution or fps, depending on whether recording is in progress.

[0213] S39: The video storage unit 17 of the information recording application 41 combines audio data with recorded video to create a combined video with audio. If no recorded video is saved, the audio data may remain independent. There are three types of recorded video: the application screen, a panoramic image, and a speaker image, but which recorded video to combine with the audio data is predetermined along with its priority. Furthermore, the recorded video and audio data do not necessarily have to be combined.

[0214] S40: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the recording settings screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. Specifically, the audio data processing unit 18 sends a request to the information processing system 50 via the communication unit 11 to specify the URL of the save location, along with the conference ID and recording ID, to convert the audio data combined with the recorded video.

[0215] S41: The communication unit 51 of the information processing system 50 receives a request to convert audio data, and the text conversion unit 55 converts the audio data into text data using the speech recognition service system 80. The communication unit 51 saves the text data to the same storage location (URL of the storage service system 70) as the storage location of the recorded video. In the recording information storage unit 5002, the text data is associated with the recorded video by the conference ID and recording ID. The text data may also be managed by the conference management unit 54 of the information processing system 50 and stored in the storage unit 5000. Alternatively, the terminal device 10 may request speech recognition from the speech recognition service system 80 and save the text data obtained from the speech recognition service system 80 to the storage location. The speech recognition service system 80 returns the converted text data to the information processing system 50, but may also send it directly to the storage URL. The speech recognition service system 80 may select or switch between multiple services according to the settings information set by the user in the information processing system 50.

[0216] S42: The upload unit 20 of the information recording application 41 saves the recorded video to the recording video storage location via the communication unit 11. In the recording information storage unit 5002, the recorded video is associated with the conference ID and the recording ID. Uploaded videos are recorded in the recorded video.

[0217] Since the storage location is notified to the user, the user can share the recorded video with participants by informing them of the storage location via email or other means. Even if the devices used to create the recorded video, audio data, and text data are different, they can be collected and stored in a single location, making them easily accessible to users later.

[0218] In the case of real-time speech recognition, the meeting device 60 or terminal device 10 transmits speech data to the information processing system 50 in real time. The terminal device 10 displays and saves the text data transmitted from the meeting device 60 or returned from the information processing system 50 on the recording screen 220.

[0219] Note that the processing in steps S31 to S36 does not have to be in the order shown in Figure 23, and the order in which the audio data is synthesized and the recorded video is saved may also be in order.

[0220] <<Display recorded video>> Next, with reference to Figure 24, the process by which a user displays recorded videos stored in the storage service system 70 will be explained. Figure 24 is an example of a sequence diagram illustrating the process by which the information recording application 41 downloads and displays recorded videos.

[0221] S51: Users who want to view recorded videos display the meeting list screen in the information recording application 41. To do this, the user selects the meeting list tab 231. The operation reception unit 12 of the information recording application 41 accepts the selection.

[0222] S52: The communication unit 11 of the information recording application 41 sends the meeting list to the information processing system 50, specifying the access token 1.

[0223] S53: The communication unit 51 of the information processing system 50 receives a meeting list request, and the meeting information acquisition unit 54 sends the meeting list request to the meeting management system 9.

[0224] S54, S55: The meeting information management unit 31 of the meeting management system 9 identifies meeting information that the user of access token 1 has permission to view. The meeting information management unit 31 also retrieves the application-side meeting information associated with the identified meeting information and meeting ID from the information processing system 50.

[0225] S56: The meeting information management unit 31 integrates the meeting information from the meeting management system 9 with the meeting information from the application side, and the meeting management system 9 transmits the integrated meeting list to the information processing system 50. The integration may also be performed by the information processing system 50.

[0226] S57: The communication unit 51 of the information processing system 50 receives the meeting list, and the screen generation unit 53 generates the meeting list screen.

[0227] S58: The communication unit 51 of the information processing system 50 transmits the screen information of the meeting list screen to the information recording application 41. The communication unit 11 of the information recording application 41 receives the screen information of the meeting list screen, and the display control unit 13 displays it.

[0228] S59: The user selects the meeting for which they want to view the recorded video. The operation reception unit 12 of the information recording application 41 accepts the selection (the meeting ID is identified).

[0229] S60: The communication unit 11 of the information recording application 41 specifies the conference ID and sends a request for recording information to the information processing system 50.

[0230] S61: The communication unit 51 of the information processing system 50 receives a request for recorded information, and first, the conference information acquisition unit confirms the existence of the conference with the conference management system 9. Here, it is assumed that the conference existed. Note that this confirmation process is optional.

[0231] S62: Since a meeting was found, the screen generation unit 53 retrieves recording information (meeting ID, recording ID, update date and time, title, save location, etc.) associated with the meeting ID from the recording information storage unit 5002.

[0232] S63: The screen generation unit 53 creates a recorded video display screen 240 using the recording information. At this point, since no recorded video has been acquired yet, the image display field 241 may be blank or display a title. If there is a thumbnail image of the recorded video stored in the recording information storage unit 5002, it is preferable that it be displayed in the image display field 241.

[0233] S64: The communication unit 51 of the information processing system 50 transmits the screen information of the recorded video display screen 240 to the information recording application 41. The communication unit 11 of the information recording application 41 receives the screen information of the recorded playback screen 240, and the display control unit 13 displays it.

[0234] S65: The user starts displaying the recorded video (turns on the display button 241a). The operation reception unit 12 of the information recording application 41 accepts the operation.

[0235] S66: The communication unit 11 of the information recording application 41 specifies the access token 2 and connects to the storage location stored in the information storage unit 1001.

[0236] S67: The storage service system 70 returns the URL where the storage destination (recorded information (recorded video and text data)) is stored to the information recording application 41.

[0237] S68: The communication unit 11 of the information recording application 41 receives this URL and can download the recorded information by connecting to it. The recorded video display unit 19 displays the recorded video on the recorded video display screen 240 and displays the corresponding text data.

[0238] S69: Additionally, users can delete meeting information from the information recording app 41. The user selects the meeting to be deleted on the meeting list screen. The operation reception unit 12 of the information recording app 41 receives the operation (the meeting ID is identified).

[0239] S70: The communication unit 11 of the information recording application 41 specifies the access token 2 and connects to the storage destination of the storage service system 70 stored in the information storage unit 1001, and deletes the recorded information.

[0240] S71: Next, the communication unit 11 of the information recording application 41 connects to the meeting management system 9 using the access token 1 and sends a request to delete meeting information by specifying the meeting ID.

[0241] S72: The meeting management system 9 receives a deletion request, and the meeting information management unit 31 deletes the meeting information identified by the meeting ID and the application-side meeting information.

[0242] <Details when displaying recorded video> As explained above, the recorded information includes: a. Panoramic image b. Speaker image c. Screens of one or more apps d. Text data This may include the following. The audio data is combined into one of a-c. a-c may not be present depending on the user's recording settings. d may also not be present if the audio data has not been converted to text data. However, if a-c are present, the user can switch the recorded video displayed on the recorded video display screen 240.

[0243] Figure 25 shows an example of a recorded video display screen 240 showing a panoramic image 203 and a speaker image 204. The explanation of Figure 25 mainly focuses on the differences from Figure 21. The recorded video display screen 240 displays the recorded video selection menu 249. The recorded video selection menu 249 has a video button 246, a content button 247, and a video & content button 248. The video button 246 displays the panoramic image and the speaker image, the content button 247 displays the application screen, and the video & content button 248 displays the speaker image and the application screen.

[0244] In Figure 25, the video button 246 is pressed, and the panoramic image 203 and speaker image 204 are displayed in the image display area 241.

[0245] Figure 26 shows the recorded video display screen 240, where the video & content button 248 is pressed, and the speaker image 204 and the app screen 217 are displayed. The speaker image 204 and the app screen 217 are displayed in the image display area 241. Note that the area where people are arranged vertically is a vertical arrangement of speaker images. A panoramic image is not displayed, but a panoramic image may be displayed.

[0246] Figure 27 shows the recorded video display screen 240, where the content button 247 has been pressed and the app screen 217 is displayed. The app screen 217 is displayed in the image display area 241.

[0247] Figure 28 shows the recorded videos displayed in the image display area 241, which switch depending on the operation performed on the recorded video selection menu 249. As shown in Figure 28, the user can switch the displayed recorded video by operating the recorded video selection menu 249.

[0248] <<Switching between recorded videos>> Figure 29 is an example of a sequence diagram illustrating the process by which the information recording application 41 switches the recorded video displayed in the image display area 241 in response to user operation.

[0249] S81: The user inputs an operation to switch the recorded video to be displayed in the recorded video selection menu 249. The operation reception unit 12 of the information recording application 41 receives the operation.

[0250] S82: The recorded video display unit 19 of the information recording application 41 obtains the current display time of the recorded video displayed in the image display area 241.

[0251] S83: The recorded video display unit 19 displays the selected recorded video in the image display area 241, starting from the displayed time.

[0252] In this way, even when the recorded video switches, the display time from the time before the switch to the time after the switch can be shown.

[0253] In Figure 29, the information recording application 41 processes the downloaded recorded information on the terminal device side and switches the recorded video, but the information processing system 50 may also perform the switching of recorded videos. For example, this processing may be performed when the information processing system 50 displays the recorded information as a web application.

[0254] <<Displaying the correspondence between text data and recorded video>> Next, with reference to Figure 30, we will explain the correspondence display between text data and recorded video by the information recording application 41. Figure 30 is an example of a sequence diagram that explains the process by which the information recording application 41 displays text data in correspondence with the display time of the recorded video shown in the image display area 241.

[0255] S91: The user turns on the transcription button 242 on the recorded video display screen 240. The operation reception unit 12 of the information recording application 41 accepts the operation.

[0256] S92: The recorded video display unit 19 of the information recording application 41 repeatedly acquires the current display time of the recorded video displayed in the image display area 241.

[0257] S93: The recorded video display unit 19 compares the time contained in the text data with the display time, and if there is a text string within a predetermined time from the display time, it displays this text string in the text display field 243. Therefore, depending on the display time, for example, the text strings are displayed chronologically from top to bottom.

[0258] In this way, text data is displayed in conjunction with the recorded video, allowing users to easily check what was said during the video playback.

[0259] In Figure 30, the information recording application 41 associates the recorded video with text data, but the information processing system 50 may perform the synchronization process. In this case, the information recording application 41 sends the display time to the information processing system 50, and the information processing system 50 returns the text data corresponding to the display time to the information recording application 41.

[0260] <<Search for Text Data>> Next, referring to FIGS. 31 and 32, the search for text data by the information recording application 41 and the corresponding display of the recorded video will be described.

[0261] FIG. 31 is a diagram schematically explaining the corresponding display of the search for text data and the recorded video. In FIG. 31(a), the user enters "keyword" in the search window 265 and searches for text data using this keyword. The search unit 22 searches for text data using "keyword" and displays "It keyword it", "that keyword it", and "this keyword it" as the search results 266.

[0262] In FIG. 31(b), the user is selecting "this keyword it" with a pointing device. When the operation reception unit 12 receives the selection of "this keyword it", the recorded video display unit 19 acquires the time associated with "this keyword it" from the text data. The recorded video display unit 19 currently displays the recorded video being displayed in the image display column 241 from the time of time (it may be displayed as a still image).

[0263] FIG. 32 is an example of a sequence diagram explaining the process of displaying the recorded video corresponding to the text data searched by the information recording application 41 in the image display column 241.

[0264] S101: The user enters the keyword to be searched and instructs the search. The operation reception unit 12 of the information recording application 41 receives the search.

[0265] S102: The search unit 22 of the information recording application 41 searches for text data using the keyword.

[0266] S103: The display control unit 13 displays a list of text strings as the search result.

[0267] S104: The user selects a string of text. The operation reception unit 12 of the information recording application 41 accepts the selection.

[0268] S105: The recorded video display unit 19 of the information recording application 41 displays the recorded video based on the time associated with the text string.

[0269] In this way, when you search for text data, the recorded video is displayed in conjunction with the text data, reducing the effort required for users to find the recorded video of the moment they spoke.

[0270] In Figure 32, the information recording application 41 performs text data search and displays recorded video, but this process may also be performed by the information processing system 50. In this case, the information recording application 41 sends keywords to the information processing system 50, and the information processing system 50 searches the text data using the keywords. The information processing system 50 sends the search results to the information recording application 41. The information recording application 41 sends the text string selected by the user to the information processing system 50. The information processing system 50 requests the information recording application 41 to display the time associated with the text string. If the information processing system 50 displays the recorded video, it sends the recorded video from the time associated with the text string.

[0271] <Main effects> As described above, the recording information display system of this embodiment simultaneously displays a panoramic image of the surroundings including the user, a speaker image, and the screen of an application displayed during the remote meeting, such as the remote conferencing application 42. Therefore, when a participant or non-participant of the remote meeting views the recorded video as meeting minutes, the atmosphere of the remote meeting is reproduced with a sense of presence. Furthermore, since the system displays a video created by recording both the screen information displayed by the application selected to be included in the recording information (such as the remote conferencing application) and the surrounding image information of the device within the base (such as the conference room) from the start to the end of recording, even if the content of the screen displayed by the remote conferencing application 42 changes during the meeting (during recording), it is possible to display recording information that comprehensively records the situation at the base (such as the conference room) along with the content of the remote meeting (remote communication). In addition, the information recording application 41 can display the recorded video in correspondence with text data. Furthermore, the information recording application 41 can selectively display the recorded video according to user operation. The information recording application 41 can display the recorded video in correspondence with a string that matches the search.

[0272] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0273] For example, the terminal device 10 and the meeting device 60 may be integrated into one unit. Alternatively, the meeting device 60 may be attached externally to the terminal device 10. Furthermore, the meeting device 60 may consist of a 360-degree camera, a microphone, and a speaker connected by cables.

[0274] Meeting devices 60 may also be placed at other locations 101. These other locations 101 will separately use meeting devices 60 to create combined image / video and text data. Furthermore, multiple meeting devices 60 may be placed at a single location. In this case, multiple recordings will be created for each meeting device 60.

[0275] Furthermore, the arrangement of the panoramic image 203, speaker image 204, and application screen in the recorded video used in this embodiment is merely an example. The panoramic image 203 may be at the bottom and the speaker image 204 at the top, or the user may change the arrangement, or individually switch the display of the panoramic image 203 and speaker image 204 when displaying the recorded video.

[0276] Furthermore, the configuration examples in Figure 7 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the terminal device 10, meeting device 60, and information processing system 50. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the terminal device 10, meeting device 60, and information processing system 50 can be further divided into more processing units depending on the processing content. Alternatively, one processing unit can be divided to include even more processing. For example, a system for creating recorded videos and a system for displaying recorded videos may be separate.

[0277] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0278] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figure 23, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0279] Each of the embodiments described above can be implemented by one or more processing circuits. Here, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above. <Additional notes regarding claims> [Claim 1] A recording information display system including an information recording application executed on a terminal device, The information recording application has a recorded video display unit that displays a recorded video created based on screen information displayed by the remote conferencing application selected by the information recording application and surrounding image information acquired by the device. The recorded video display unit is characterized by displaying the surrounding image information, speaker image information extracted from the surrounding image information, and the screen information. [Claim 2] The recording information display system according to claim 1, wherein the screen information is information acquired as an image by the information recording application, which displays a running application, including the remote conferencing application, as a window. [Claim 3] The recording information display system according to claim 1 or 2, wherein the image information surrounding the device is image information obtained by the device imaging a 360° area surrounding the device. [Claim 4] The recorded information display system according to any one of claims 1 to 3, wherein image information of the surroundings of the device is transmitted from the device installed at the base to the information recording application. [Claim 5] The recording video display unit further displays, on the same screen simultaneously with the recording video, text data based on the audio data collected by the device and the audio data output by the terminal device. The recording information display system according to claim 1 is characterized in that [Claim 6] The text data is audio data obtained by synthesizing the audio data collected by the device and the audio data output by the terminal device. The recording information display system according to claim 5 is characterized in that the recording video display unit displays the text data obtained by converting the synthesized audio data through voice recognition. [Claim 7] The recording information display system according to claim 5 or 6 is characterized in that the recording video display unit scrolls and displays the text data in correspondence with the display time of the recording video. [Claim 8] A search unit that searches for keywords from the text data, and a display control unit that displays a part of the text data that matches the search. The recording information display system according to claim 5 or 6 is characterized in that when a part of the text data is selected, the recording video display unit displays the recording video at the speech time associated with the part of the text data. [Claim 9] The recording video has the surrounding image information, speaker image information cut out from the surrounding image information, and the screen information. [[ID=...]] (The remaining content is translated in a similar manner as above, maintaining line breaks and preserving tags) The recording information display system according to any one of claims 1 to 8 is characterized in that the recording video display unit switches, according to a user operation, whether to display the surrounding image information and the speaker image information, whether to display the screen information, or whether to display the speaker image information and the screen information. [Claim 10] The recorded video display unit is characterized in that it displays the surrounding image information and speaker image information, the screen information, or the speaker image information and screen information from the time the user switches between displaying the surrounding image information and speaker image information, displaying the screen information, or displaying the speaker image information and screen information, after the switch. [Explanation of Symbols]

[0280] 10 Terminal devices 50 Information Processing Systems 60 Meeting Devices 100 Record Information Display System [Prior art documents] [Patent Documents]

[0281] [Patent Document 1] Japanese Patent Publication No. 2021-105688

Claims

1. A recording information display system for displaying recording information of a remote meeting, comprising: a recording information display unit that displays recording information including a recording image created based on screen information of the remote meeting displayed on a terminal device participating in the remote meeting, surrounding image information, and speaker image information cropped from the surrounding image information, and text data based on the audio data of the remote meeting; The recording information display system, wherein the recording information display unit displays the recording image associated with a time associated with the text data selected from the recording information.

2. The recording information display system according to claim 1, wherein the screen information is information displayed as a window by a remote meeting application executed on the terminal device.

3. The recording information display system according to claim 1 or 2, wherein the surrounding image information is image information obtained by imaging a 360° region.

4. The recording information display system according to claim 1, wherein the surrounding image information is transmitted from a device installed at a base to the terminal device.

5. The recording information display system according to claim 1, wherein the recording information display unit further displays the text data in a different area of the same screen simultaneously with the recording image.

6. The text data is audio data obtained by synthesizing audio data collected by a device installed at a base and audio data output by the terminal device, The recording information display system according to claim 5, wherein the recording information display unit displays the text data obtained by converting the synthesized audio data by voice recognition.

7. The recording information display system according to claim 5 or 6, wherein the recording information display unit scrolls and displays the text data in correspondence with the display time of the recording image.

8. A search unit for searching for keywords from the text data, a display control unit for displaying a part of the text data that matches the search; and The recording information display system according to claim 5 or 6, wherein the recording information display unit displays the recording image associated with a time associated with the text data selected from a part of the text data.

9. The recording image includes the surrounding image information, speaker image information cropped from the surrounding image information, and the screen information. The recording information display unit is characterized in that, in response to a user operation, it switches between displaying the surrounding image information and the speaker image information, displaying the screen information, or displaying the speaker image information and the screen information, according to the recording information display system described in claim 1.

10. The recording information display unit is characterized in that, based on the display time when the user switches between displaying the surrounding image information and the speaker image information, displaying the screen information, or displaying the speaker image information and the screen information, it displays the surrounding image information and the speaker image information, the screen information, or the speaker image information and the screen information after the switching, according to the recording information display system described in claim 9.

11. A recording information display system for displaying the recording information of a remote conference, function as a recording information display unit that displays recording information consisting of a recording image created based on the screen information of the remote conference displayed on a terminal device participating in the remote conference, the surrounding image information, and the speaker image information cut out from the surrounding image information, and text data based on the audio data of the remote conference; A program characterized in that the recording information display unit displays the recording image associated with the time associated with the text data selected from the recording information.

12. A recording information display method performed by a recording information display system for displaying the recording information of a remote conference, a process in which the recording information display unit displays recording information consisting of a recording image created based on the screen information of the remote conference displayed on a terminal device participating in the remote conference, the surrounding image information, and the speaker image information cut out from the surrounding image information, and text data based on the audio data of the remote conference; a process of displaying the recording image associated with the time associated with the text data selected from the recording information; A recording information display method characterized by performing the above.

13. A recording information playback system for playing back the recording information of a remote conference, having a recording information playback unit that plays back recording information consisting of a recording image including the screen information of the remote conference, text data based on the audio data of the remote conference, and the audio data. The recording information playback unit plays the audio data associated with the time associated with the text data selected from the recording information, and the recording information playback system is characterized by this.

14. A search unit that searches for keywords from the text data, And a display control unit that displays a part of the text data that matches the search by the search unit, The recording information playback unit plays the audio data associated with the time associated with the text data selected from a part of the text data, and the recording information playback system according to claim 13 is characterized by this.

15. The recording information further includes video information, and the recording information playback system according to claim 13 is characterized by this.

16. The recording information playback unit plays the video information and the text data in correspondence, and plays the video information associated with the time associated with the text data selected from the recording information, and the recording information playback system according to claim 15 is characterized by this.

17. The recording information playback unit plays the text data in different areas of the same screen simultaneously with the video information, and the recording information playback system according to claim 15 is characterized by this.

18. The recording information playback unit scrolls and displays the text data in the area where the text data is displayed, and the recording information playback system according to claim 17 is characterized by this.

19. The recording information playback unit accepts the selection of the text data by accepting the selection of some character strings included in the text data, and the recording information playback system according to claim 13 is characterized by this.

Images