Record information creation system, record information creation method, program

Abstract

To provide a recording information creation system that creates recording information of remote communication based on screen information obtained from a running application and image information of the device's surroundings, and that can recreate the situation when remote communication was conducted while imaging the inside of a conference room. [Solution] The present invention is a recording information creation system including an information recording application executed on a terminal device, wherein the information recording application creates recording information based on screen information displayed by a remote conferencing application selected by the information recording application and image information of the device's surroundings acquired by the device.

Description

Technical Field

[0001] The present invention relates to a recording information creation system, a recording information creation method, and a program.

Background Art

[0002] A remote conferencing system is known that transmits images and voices in real time from one base to one or more other bases and enables users at remote locations to hold meetings using images and voices.

[0003] Also known is a technique for storing conference participants as omnidirectional images (see, for example, Patent Document 1). Patent Document 1 discloses an audio recording device that images participants in a conference room with an omnidirectional camera, identifies the face regions of each participant, and determines whether to record audio according to the direction of the face.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, there is a problem that recording information for remote communication is not created based on the screen information acquired from the running application and the surrounding images. For example, just storing the information of the imaged conference room makes it difficult to reproduce the situation when remote communication is carried out while imaging the conference room.

[0005] In view of the above problems, an object of the present invention is to provide a recording information creation system that creates recording information for remote communication based on the screen information acquired from the running application and the image information around the device, and reproduces the situation when remote communication is carried out while imaging the conference room.

Means for Solving the Problems

[0006] In view of the above issues, the present invention provides a recording information creation system including an information recording application executed on a terminal device, wherein the information recording application creates recording information based on screen information displayed by a remote conferencing application selected by the information recording application and image information of the device's surroundings acquired by the device. [Effects of the Invention]

[0007] Based on screen information acquired from a running application and image information of the device's surroundings, this system can create recording information of remote communication and provide a recording information creation system that can recreate the situation when remote communication was conducted while imaging the inside of a conference room. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram outlines the creation of recording information that saves the screen of an application running during a remote meeting, along with a panoramic image of the surroundings. [Figure 2] This diagram shows an example configuration of a record information creation system. [Figure 3] This figure shows an example of the hardware configuration of an information processing system and terminal device. [Figure 4] This figure shows an example of a hardware configuration for a meeting device. [Figure 5] This diagram illustrates the imaging range of the meeting device. [Figure 6] This diagram illustrates the process of extracting images from a panoramic image and a speaker image. [Figure 7] This is an example of a functional block diagram that explains the functions of terminal devices, meeting devices, and information processing systems in a record information creation system by dividing them into blocks. [Figure 8] This figure shows an example of video recording information stored in the information storage unit. [Figure 9] This diagram shows an example of meeting information managed by the Meeting Management Department. [Figure 10]This diagram shows the recorded information stored in the recorded information storage unit. [Figure 11] This figure shows an example of the initial screen displayed after login by an information recording application running on a terminal device. [Figure 12] This diagram explains how to operate the system when the fixed display button is turned on. [Figure 13] This figure shows an example of a screen indicating that a device is not recognized. [Figure 14] This figure shows an example of a recording settings screen displayed by an information recording app. [Figure 15] This diagram shows an example of how the recording content confirmation window looks when the camera toggle button is on and the PC screen toggle button is off. [Figure 16] This diagram shows an example of how the recording content confirmation window looks when the camera toggle button and the PC screen toggle button are both turned on. [Figure 17] This figure shows an example of a recording screen displayed by an information recording app while recording. [Figure 18] This figure shows an example of a meeting list screen displayed by an information recording app. [Figure 19] This figure shows an example of the recording playback screen after selecting a combined image video displayed by an information recording app. [Figure 20] This figure shows an example of an editing screen for combined images and videos. [Figure 21] This is an example of a sequence diagram illustrating the procedure or process by which a user logs into an information processing system by operating an information recording application on a terminal device. [Figure 22] This is an example of a sequence diagram showing the procedure by which an information recording app records panoramic images, speaker images, and the app screen. [Figure 23] This is an example sequence diagram showing the procedure by which the information recording app records a panoramic image, speaker image, and app screen with the changed settings when the settings for the recording settings screen are changed while creating a combined image video. [Figure 24]This is an example of a sequence diagram showing the procedure of saving the screens of all running apps and changing the images (videos) to be included in the recording information after the remote meeting ends. [Figure 25] This is an example of a sequence diagram showing the procedure of properly ending the recording when the communication between the terminal device and the meeting device is disconnected during recording. [Figure 26] This is a diagram showing a configuration example of a recording information creation system.

Mode for Carrying Out the Invention

[0009] Hereinafter, as an example of a mode for carrying out the present invention, a recording information creation system and a recording information creation method performed by the recording information creation system will be described. [[ID=I6]]

[0010] <An Example of a Method for Creating Minutes of a Remote Meeting> First, referring to FIG. 1, an outline of a method for creating minutes using a panoramic image and an app screen will be described. FIG. 1 is a diagram for explaining an outline of creating recording information for saving the screen of an app executed during a remote meeting together with a surrounding panoramic image. As shown in FIG. 1, a user at the home base 102 shown is using the remote meeting service system 90 to hold a remote meeting with another base 101.

[0011] The recording information creation system 100 of the present embodiment creates recording information (such as minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) obtained by the imaging means capable of imaging 360° around and a meeting device 60 equipped with a microphone and a speaker by processing the information obtained by imaging the surroundings, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. For audio, the recording information creation system 100 synthesizes the audio received by the remote meeting app 42 and the audio obtained by the meeting device 60 and includes it in the recording information. Hereinafter, an outline will be described.

[0012] (1) The terminal device 10 is running the information recording application 41 and the remote conferencing application, which will be described later. Other applications, such as one for displaying documents, may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including audio received by the remote conferencing application from other locations; an example of the first audio data) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired (an example of the second audio data) with the audio from the remote conferencing application.

[0013] (2) The meeting device 60 is equipped with a microphone and, based on the direction from which the sound was acquired, processes the image to extract the speaker from the panoramic image and creates a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines the panoramic image 203 and the speaker image 204 with any application screen selected by the user (for example, the remote conferencing application screen 103). For example, the panoramic image 203, the speaker image 204, and the application screen 103 are combined so that the panoramic image 203 and the speaker image 204 are on the left and the remote conferencing application screen 103 is on the right (hereinafter referred to as the combined image 105). The application screen is an example of screen information (described later) displayed by each application such as the remote conferencing application. Since the process in (3) is executed repeatedly, the combined image 105 becomes a video (hereinafter referred to as the combined image video). The information recording application 41 also combines the audio synthesized with the combined image video to create a video with audio.

[0015] In this embodiment, an example of combining the panoramic image 203, the speaker image 204, and the application screen 103 is described, but the information recording application 41 may save these separately and arrange them on the screen during playback.

[0016] (4) The information recording application 41 accepts editing work (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0017] (5) The information recording application 41 sends the created combined image video (with audio) to the storage service system 70 for storage.

[0018] (6) The information recording application 41 also extracts only the audio from the combined image video (it may also keep the audio before combining) and sends the extracted audio to the information processing system 50. The information processing system 50 sends the audio to the speech recognition service system 80, which converts the audio into text data. The text data also includes data on how many minutes after the start of recording the speaker spoke.

[0019] In the case of real-time text conversion, the meeting device 60 transmits the audio directly to the information processing system 50.

[0020] (7) The information processing system 50 additionally stores text data in the storage service system 70 which stores the combined image video. The text data constitutes part of the recorded information.

[0021] Furthermore, the information processing system 50 can perform billing processing for users based on the services they have used. For example, billing is calculated based on the amount of text data, the file size of combined images and videos, processing time, etc.

[0022] In this way, the combined image video displays panoramic images of the surroundings including the user, as well as images of the speaker, and also shows the screen of the application used during the remote meeting, such as the remote conferencing application 42. When participants or non-participants of the remote meeting view the combined image video as meeting minutes, the atmosphere of the remote meeting is recreated with a sense of presence.

[0023] <About Terminology> An application (app) is software developed and used for a specific function or purpose. Applications can be native apps or web apps. A web app (a cloud-based app provided by a cloud service) may also work in conjunction with a native app or web browser.

[0024] A running app refers to an app in its state from the time it is launched until it is closed. The app does not need to be active (the app in the foreground); it only needs to be running in the background.

[0025] A device is a device that can capture images of its surroundings and collect sounds from its surroundings. A device may include those used in conjunction with a terminal device, those built into a terminal device, or those used by connecting to a cloud service without directly connecting to a terminal device. In this embodiment, it will be described using the term "meeting device."

[0026] The image information of the surroundings of a meeting device acquired by a meeting device is image information acquired by the meeting device by capturing the surrounding space (for example, a region or space of 180° to 360° in the horizontal direction), and refers to an image acquired by performing predetermined processing on the curved surface image information captured by the meeting device. The predetermined processing refers to various processes for creating surrounding image information from the captured information, such as a flattening process for the captured curved surface image. In addition to the process of creating the surrounding image, the predetermined processing may also include a process of cutting out the speaker image, and a merging process that combines the surrounding image and the speaker image, etc. In this embodiment, the surrounding image is described using the term panoramic image. A panoramic image is an image with a field of view of approximately 180° to 360° in the horizontal direction. The meeting device does not have to capture a panoramic image with a single unit, and multiple imaging devices with normal fields of view may be combined. Meeting devices are intended for use at locations such as tables for meetings and to monitor the surrounding environment. However, they may also be used for other purposes, such as monitoring (security, disaster prevention, etc.), supervision (childcare, elder care, etc.), and on-site situation analysis (solutions, marketing, etc.).

[0027] Recorded information refers to information recorded by the information recording application 41, which is stored and made viewable as information linked to the identification information of a particular meeting, and includes, for example, the following information: Video information created based on screen information displayed by the selected application (such as a remote conferencing application), image information of the device's surroundings acquired by the device, etc. • Audio information acquired and synthesized during a meeting using a remote conferencing application (terminal device) and a meeting device at the local office. • Text information obtained by converting the acquired audio into text. Other data and images related to the meeting. For example, document files used during the meeting, additional notes, translated text data, images and stroke data created using a cloud-based electronic whiteboard service during the meeting, etc. For example, if the information recording application 41 records the screen of a remote meeting application or the proceedings of a meeting at a local office, the recorded information may become the minutes of the meeting that took place. The minutes are just one example of the recorded information, and the name of the recorded information changes depending on the content of the remote meeting or what was done at the local office. For example, it could be called a record of communication, a record of the local office's status, etc. The recorded information also includes multiple file formats, such as video files (combined video images, etc.), audio files, text data (text data from speech recognition), document files, image files, and tabular files. Since the files are related to each other in relation to the meeting identification information, they can be viewed chronologically, either all at once or selectively.

[0028] A tenant is a group of users (companies, local governments, and some of these organizations) who have contracted to receive services from a service provider. The creation of record information and its conversion to text data in this embodiment are performed because the tenant has a contract with the service provider.

[0029] Remote communication refers to communication with someone located at a physically distant location, using software and terminal devices to communicate via voice and video. One example of remote communication is a remote conference, which may also be called a meeting, discussion, consultation, contract application, gathering, assembly, seminar, workshop, study group, seminar, training session, etc.

[0030] A base of operations refers to a place that serves as the foundation for activities. An example of a base of operations is a conference room. A conference room is a room set up primarily for the purpose of holding meetings. A base of operations can also be a home, a reception area, a store, a warehouse, or an outdoor work site, as long as there is a place or space where terminal equipment or devices can be installed.

[0031] Speech refers to human speech sounds and ambient sounds, while speech data is the data representation of speech. However, in this embodiment, we will not strictly distinguish between the two.

[0032] <Example System Configuration> Next, the system configuration of the recording information creation system 100 will be explained with reference to Figure 2. Figure 2 shows an example of the configuration of the recording information creation system 100. In Figure 2, one of several locations where remote meetings are held (local location 102) is shown, and the terminal device 10 at local location 102 communicates with the information processing system 50, the storage service system 70, and the remote meeting service system 90 via the network. A meeting device 60 is also located at local location 102, and the terminal device 10 is connected to this meeting device 60 via a USB cable or the like for communication.

[0033] Terminal device 10 operates at least an information recording application 41 and a remote conferencing application 42. The remote conferencing application 42 can communicate with terminal devices at other locations 101 via a remote conferencing service system 90 on the network, allowing users at each location to hold meetings remotely. The information recording application 41 creates recording information for remote meetings conducted by the remote conferencing application 42 using the functions of the information processing system 50 and the meeting device 60.

[0034] In this embodiment, we describe an example of creating recording information during a remote meeting, but the meeting does not have to be one that communicates with remote locations. In other words, the meeting may be one in which only participants from a single location are present. In this case, only the audio collected by the meeting device 60 is saved without synthesis, and there is no change in the processing of the information recording application 41.

[0035] The terminal device 10 has a built-in (or external) camera with a normal field of view that captures an image of the front, including the user 107 operating the terminal device 10. A normal field of view is an image that is not a panoramic image, but in this embodiment, it is mainly a planar image that is not curved like a 360-degree image. The terminal device 10 also has a built-in (or external) microphone that collects ambient sounds such as those of the user operating the terminal device 10. Therefore, the user can conduct a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42, except for the increased processing load on the terminal device 10.

[0036] The information recording application 41 is an application that records information by communicating with the meeting device 60 and creating recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capture device, a microphone, and a speaker. The camera on the terminal device 10 can only capture a limited area in front of it, but the meeting device 60 can capture the entire surrounding area (not necessarily all around) of the meeting device 60. The meeting device 60 can always keep multiple participants 106, as shown in Figure 2, within its field of view.

[0037] In addition, the meeting device 60 extracts speaker images from the panoramic image, and combines the audio acquired by the meeting device 60 with the audio output by the terminal device 10 (including the audio received by the remote conferencing application 42). The meeting device 60 is not limited to being placed on a table or stand; it can be placed anywhere within the local site 102. Since the meeting device 60 can capture a 360-degree image, it can be placed on the ceiling, for example. The meeting device 60 can also be installed at other sites or at any of the sites.

[0038] The information recording application 41 displays a list of applications running on the terminal device 10, performs image synthesis (creation of combined image videos) for the recorded information described above, plays the combined image videos, accepts editing requests, etc. The information recording application 41 also displays a list of remote meetings that have been held or are scheduled to be held. The list of remote meetings is used for information related to the recorded information, allowing the user to link remote meetings with the recorded information.

[0039] The remote conferencing application 42 is an application that enables a terminal device to communicate remotely with other terminal devices at other locations 101 through communication connections, transmission and reception of images and audio, display of images, and output of audio. The remote conferencing application can also be called a remote communication application or a remote information sharing application.

[0040] The information recording application 41 and the remote conferencing application 42 may be either web applications or native applications. A web application is an application in which a program on a web server and a program on a web browser or a native application work together to process data, and does not require installation on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used by it. In this embodiment, both will be described as native applications.

[0041] The terminal device 10 may be a general-purpose information processing device equipped with communication functions, such as a PC (Personal Computer), smartphone, or tablet device. Other examples of terminal devices 10 include electronic whiteboards, game consoles, PDAs (Personal Digital Assistants), wearable PCs, car navigation systems, industrial machinery, medical equipment, and network-connected home appliances. The terminal device 10 only needs to be a device on which at least the information recording application 41 and the remote conferencing application 42 can run.

[0042] The information processing system 50 consists of one or more information processing devices located on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and infrastructure services. These server applications manage lists of remote meetings, manage recorded information from remote meetings, manage various settings and storage paths, etc. The infrastructure services handle user authentication, contracts, billing, etc.

[0043] Furthermore, all or part of the functions of the information processing system 50 may reside in a cloud environment or in an on-premises environment. The information processing system 50 may consist of multiple server devices or a single information processing device. For example, server applications and infrastructure services may be provided by separate information processing devices, and furthermore, there may be separate information processing devices for each function within the server application. The information processing system 50 may be integrated with the storage service system 70 and the speech recognition service system 80 described below.

[0044] The storage service system 70 is a storage means on a network and provides a storage service that accepts the storage of files, etc. Examples of storage service systems 70 include OneDrive®, Google Workspace®, and Dropbox®. The storage service system 70 may also be an on-premises NAS (Network Attached Storage).

[0045] The speech recognition service system 80 provides a service that performs speech recognition on voice data and converts it into text data. The speech recognition service system 80 may be a general-purpose commercial service or it may be part of the functions of the information processing system 50. In addition, the speech recognition service system 80 may be configured and used differently for each user, tenant, or meeting.

[0046] <Example Hardware Configuration> Referring to Figure 3, the hardware configuration of the information processing system 50 and terminal device 10 according to this embodiment will be described.

[0047] <<Information Processing Systems and Terminal Devices>> Figure 3 shows the hardware configuration of an example of an information processing system 50 and terminal device 10 according to this embodiment. As shown in Figure 3, the information processing system 50 and terminal device 10 are built using a computer and include a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0048] Of these, the CPU 501 controls the operation of the entire information processing system 50 and terminal device 10. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. External devices in this case include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 3.

[0049] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0050] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be explained using Figure 4. Figure 4 is an example of a hardware configuration diagram for a meeting device 60 capable of capturing 360° video. In the following, the meeting device 60 is assumed to be a device that uses an image sensor to capture 360° video around the device at a predetermined height, but it may have one or more image sensors. Furthermore, it does not necessarily have to be a dedicated device; a 360° video capture unit can be attached to a PC, digital camera, smartphone, etc., to achieve substantially the same function.

[0051] As shown in Figure 4, the meeting device 60 consists of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection interface 616, a communication unit 617, an antenna 617a, a sound sensor 618, and a recessed terminal 621 for Micro USB.

[0052] Of these, the imaging unit 601 includes a wide-angle lens (a so-called fisheye lens) 602 with a 360° field of view for forming a hemispherical image, and an image sensor 603 provided corresponding to each wide-angle lens. The image sensor 603 has an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lens 602 into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set. The imaging unit 601 may also be a 360° camera, and is an example of an imaging means capable of capturing images of 360° around the meeting device 60. Alternatively, multiple pieces of information acquired by multiple image sensors (e.g., 180° + 180°) may be combined to create a 360° field of view.

[0053] Each image sensor 603 of the imaging unit 601 is connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensor 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via bus 610. Furthermore, ROM 612, SRAM 613, DRAM 614, operation unit 615, external device connection I / F 616, communication unit 617, and sound sensor 618 are also connected to bus 610.

[0054] The image processing unit 604 receives image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates panoramic images and speaker images from the fisheye video. Furthermore, the image processing unit 604 combines the panoramic image and speaker images, etc., to output a single video.

[0055] The imaging control unit 605 generally uses the I2C bus to set commands and other information in the registers of the image sensor 603, with the imaging control unit 605 acting as the master device and the image sensor 603 as the slave device. It receives necessary commands and other information from the CPU 611. The imaging control unit 605 also uses the I2C bus to acquire status data and other information from the registers of the image sensor 603 and send it to the CPU 611.

[0056] Furthermore, the imaging control unit 605 instructs the image sensors 603a and 603b to output image data when the image capture start button on the operation unit 615 is pressed or when an image capture start instruction is received from the PC. Depending on the meeting device 60, there may also be functions to display a preview or video on a display (for example, the display of a PC or smartphone). In this case, the image data output from the image sensor 603 is performed continuously at a predetermined frame rate (frames / minute).

[0057] Furthermore, as will be described later, the imaging control unit 605 also functions as a synchronization control means that works in cooperation with the CPU 611 to synchronize the output timing of image data from the image sensor 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0058] Microphone 608 converts sound into sound (signal) data. Sound processing unit 609 receives the sound data output from microphone 608 via the I / F bus and performs predetermined processing on the sound data.

[0059] The CPU 611 controls the overall operation of the meeting device 60 and performs necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in progress. In particular, the DRAM 614 stores image data in progress and processed equirectangular projection image data from the image processing unit 604.

[0060] The control unit 615 is a collective term for the control buttons, such as the image capture start button 615a. By operating the control unit 615, the user can start imaging and recording, turn the power on / off, establish a communication connection, and input various settings such as imaging modes and imaging conditions.

[0061] The external device connection interface 616 is an interface for connecting various external devices. In this case, external devices include, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection interface 616 may also be equipped with, for example, a USB terminal, an HDMI® terminal, etc. Video data and image data stored in the DRAM 614 are transmitted to an external terminal via this external device connection interface 616 or recorded on external media. Alternatively, multiple external device connection interfaces 616 may be used to transmit image information captured by the meeting device 60 to a PC via USB for recording, while simultaneously acquiring video (for example, screen information to be displayed in a remote conferencing application) from the PC to the meeting device 60, and then transmitting it from the meeting device 60 via HDMI to other external devices (display, projector, electronic whiteboard, etc.) for display.

[0062] The communication unit 617 may communicate with a cloud server via the internet using wireless communication technology such as Wi-Fi through the antenna 617a provided on the meeting device 60, and transmit the stored video data and image data to the cloud server. Alternatively, the communication unit 617 may communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy; registered trademark) or NFC.

[0063] The audio sensor 618 is a sensor that acquires 360° audio information in order to determine from which direction loud audio is being input within a 360° radius (horizontal plane) around the meeting device 60. The audio processing unit 609 identifies the direction of the strongest sound based on the input 360° audio parameters and outputs the audio input direction within the 360°.

[0064] Furthermore, other sensors (such as compass / accelerometers or GPS) can be used to calculate direction, position, angle, acceleration, etc., and these can be used for image correction or to add location information.

[0065] The image processing unit 604 also performs the following processing:

[0066] The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing, such as Bayer conversion (RGB interpolation processing), on the RAW data input from the image sensor that inputs a spherical image to create a fisheye image (an image consisting of curved images). Furthermore, it performs planarization processing, such as DeWarp processing (distortion correction processing), on the created fisheye image (a curved image) to create a panoramic image (an image consisting of flat images) that captures 360° around the meeting device 60.

[0067] The CPU 611 creates speaker images in the following way: The CPU 611 creates speaker images by extracting the speaker from a panoramic image (flat image) that captures the surrounding 360°. The CPU 611 uses the direction of the audio input, identified from the 360° output by the audio sensor 618 and the audio processing unit 609, as the direction of the speaker, and extracts the speaker image from the panoramic image. In this case, the method for extracting a person's image from the direction of the audio input is to cut out 30° around the audio direction identified from the 360°, and then perform face detection within that area to extract the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., 3 people) who have spoken most recently from the extracted speaker images.

[0068] The panoramic image and the images of one or more speakers may be sent individually to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the images of one or more speakers are sent individually from the meeting device 60 to the information recording application 41.

[0069] Figure 5 illustrates the imaging range of the meeting device 60. As shown in Figure 5(a), the meeting device 60 images a 360° range in the horizontal direction. As shown in Figure 5(b), the meeting device 60 has an imaging range that is 0° horizontal to the height of the meeting device 60, and a predetermined angle above and below it.

[0070] Figure 6 illustrates the extraction of panoramic and speaker images. As shown in Figure 6, the image captured by the meeting device 60 forms a part 110 of a sphere and therefore has a three-dimensional shape. As shown in Figure 5(b), the meeting device 60 divides the field of view into predetermined angles vertically and horizontally and performs perspective projection transformation. By performing perspective projection transformation seamlessly across the entire 360° horizontal direction, a predetermined number of planar images are obtained, and by stitching these predetermined number of planar images together horizontally, a panoramic image 111 is obtained. The meeting device 60 also performs face detection from the panoramic image within a predetermined range centered on the direction of sound, and creates speaker images 112 by extracting 15° to the left and right (30° in total) from the center of the face.

[0071] <About the features> Next, with reference to Figure 7, the functional configuration of the recording information creation system 100 will be described. Figure 7 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recording information creation system 100 by dividing them into blocks.

[0072] <<Terminal device>> The information recording application 41 operating on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, an image merging unit 17, an audio data processing unit 18, a recording playback unit 19, an upload unit 20, and an editing processing unit 21. Each of these units in the terminal device 10 is a function or means of function realized by any of the components shown in Figure 3 operating according to instructions from the CPU 501 following the information recording application 41 deployed from the HD 504 to the RAM 503. The terminal device 10 also has a storage unit 1000 constructed from the HD 504, etc., as shown in Figure 3. An information storage unit 1001 is constructed in the storage unit 1000.

[0073] The communication unit 11 communicates various types of information with the information processing system 50 via the network. For example, the communication unit 11 receives a list of remote meetings from the information processing system 50 and sends voice data recognition requests to the information processing system 50.

[0074] The display control unit 13 displays various screens that serve as the user interface in the information recording application 41 according to the screen transitions set in the information recording application 41. The operation reception unit 12 receives various operations on the information recording application 41.

[0075] The application screen acquisition unit 14 acquires screen information displayed by the application selected by the user, or screen information of the desktop screen, etc., from the OS (Operating System), etc. If the application selected by the user is the remote conferencing application 42, the screen generated by the remote conferencing application 42 is obtained (images of terminal device users captured by terminal device cameras at each location, images of shared documents displayed, images including participant icons and participant names, etc.). The screen information displayed by the application (application screen) is information displayed as a window by a running application, including the remote conferencing application, etc., and acquired as an image by the information recording application. The application window is drawn as an area of ​​the window within the entire desktop image and displayed on a monitor, etc. The screen information displayed by the application can be acquired by other applications (information recording applications, etc.) as an image file or a video file composed of a series of images via the OS (Operating System) API or the API of the application that displays it. Similarly, the screen information of the desktop screen is information composed of images of the desktop screen generated by the OS, and can be acquired as an image file or video file via the OS API. The format of these image files may be bitmap, PNG, or other formats. The video file format can be MP4 or other formats.

[0076] The audio acquisition unit 15 acquires audio output from the microphone or earphones of the terminal device 10 (including audio data received from the remote conferencing application 42 during a remote conference). The audio acquisition unit 15 can acquire audio even if the output audio is muted. The user does not need to perform any operations such as selecting the remote conferencing application 42 regarding audio data; the audio acquisition unit 15 can acquire the audio that the terminal device 10 can output via the OS or application's API (Application Interface). This also allows the audio data received by the remote conferencing application 42 from other locations 101 to be acquired. If the remote conferencing application 42 is not running or a remote conference is not in progress, the information recording application 41 may not be able to acquire audio data. Note that the audio acquired by the audio acquisition unit 15 may not include the audio collected by the terminal device 10, but only the output audio data. This is because the meeting device 60 separately collects audio from the locations.

[0077] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 using wireless LAN, Bluetooth (registered trademark), etc. The device communication unit 16 receives panoramic images and speaker images from the meeting device 60 and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0078] The image merging unit 17 combines the panoramic image and speaker image received by the device communication unit 16, and the application screen acquired by the application screen acquisition unit 14, to create a combined image. The image merging unit 17 also repeatedly connects the created combined images in chronological order to create a combined image video, and combines the synthesized audio data with the combined image video to create a combined image video with audio. The merging of the panoramic image and speaker image may be performed by the meeting device 60. Alternatively, videos consisting of each image, such as the panoramic image, speaker image, application screen, and the image consisting of the panoramic image and speaker image, may be stored as separate video files in the storage service system 70. In that case, the panoramic video, speaker video, application screen video, and combined video of the panoramic image and speaker image may be called up and displayed on a single display screen when viewed.

[0079] The audio data processing unit 18 either extracts the audio data combined into the combined image video, or requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

[0080] The recording and playback unit 19 plays back the combined image video. The combined image video is stored in the terminal device 10 during recording and then uploaded to the information processing system 50.

[0081] When the remote conference ends, the upload unit 20 transmits the combined image and video to the information processing system 50.

[0082] The editing processing unit 21 performs editing of the combined image video (such as deleting parts or splicing them together) in response to user operations.

[0083] Figure 8 shows the video recording information stored in the information storage unit 1001. The video recording information includes items such as meeting ID, recording ID, update date and time, title, upload, and storage location. When a user logs into the information processing system 50, the information recording application 41 downloads meeting information from the meeting information storage unit 5001 of the information processing system 50. The meeting ID and other information included in the meeting information are reflected in the video recording information. The video recording information in Figure 8 is held by a terminal device 10 operated by a particular user.

[0084] The meeting ID is identification information that identifies a remote meeting that has been held. The meeting ID is assigned when the remote meeting is scheduled in the meeting management system 9, or it is assigned by the information processing system 50 upon request from the information recording application 41. The meeting management system 9 is a system for registering meeting and remote meeting schedules, URLs (meeting links) for starting remote meetings, and reservation information for equipment used in meetings. It is a scheduler, etc., that is connected to from the terminal device 10 via the network. The meeting management system 9 can also transmit the registered schedules, etc., to the information processing system 50.

[0085] The recording ID is identification information that identifies the combined video image recorded during a remote meeting. The recording ID is assigned by the meeting device 60, but it may also be assigned by the information recording application 41 or the information processing system 50. The same meeting ID may be assigned to a different recording ID if the recording of the remote meeting was stopped midway through but resumed for some reason.

[0086] • The update date and time is the date and time when the combined image / video was updated (recording ended). If the combined image / video was edited, it will be the date and time of editing.

[0087] The title is the name of the meeting. This may be set when registering the meeting in the meeting management system 9, or it may be set arbitrarily by the user.

[0088] • Upload indicates whether or not the combined image video has been uploaded to the information processing system 50.

[0089] The storage location indicates the location (URL or file path) where the combined image / video and text data are stored in the storage service system 70. Therefore, users can view the uploaded combined image / video at their discretion. Note that the combined image / video and text data are saved with separate filenames, for example, following the URL.

[0090] <<Meeting Devices>> Let's return to Figure 7 for explanation. The meeting device 60 includes a terminal communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, and a speech synthesis unit 65. Each of these units in the meeting device 60 is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 611 in accordance with a program expanded from the ROM 612 to the DRAM 614.

[0091] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may also communicate with the terminal device 10 using wireless LAN, Bluetooth (registered trademark), or the like.

[0092] The panoramic image creation unit 62 creates a panoramic image. The speaker image creation unit 63 creates a speaker image. The methods for creating these images are explained in Figures 5 and 6.

[0093] The sound collection unit 64 converts the audio signal acquired by the microphone of the meeting device 60 into audio data (digital). This allows the content spoken by users and participants at the terminal device 10 to be collected.

[0094] The speech synthesis unit 65 synthesizes the speech transmitted from the terminal device 10 with the speech collected by the sound collection unit 64. This combines speech spoken at other locations 101 with speech spoken at the local location 102.

[0095] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a conference management unit 54, and a text conversion unit 55. Each of these units in the information processing system 50 is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 501 following a program deployed from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed from the HD 504 and other components shown in Figure 3. The storage unit 5000 includes a conference information storage unit 5001 and a recording information storage unit 5002.

[0096] The communication unit 51 sends and receives various types of information with the terminal device 10. For example, the communication unit 51 may send a list of remote meetings to the terminal device 10 or receive a voice data recognition request from the terminal device 10.

[0097] The authentication unit 52 authenticates the user operating the terminal device 10. The authentication unit 52 authenticates the user, for example, by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, biometric authentication information such as a face or fingerprint, etc. The authentication unit 52 may also authenticate using an external authentication system or an authentication method such as OAuth.

[0098] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 runs a native application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc.

[0099] The conference management unit 54 retrieves information about remote meetings from the conference management system 9 using each user's account or a system account assigned to the information processing system 50. The conference management unit 54 can retrieve a list of remote meetings that users belonging to the tenant have viewing privileges for. Since each remote meeting is assigned a meeting ID, the remote meeting and its record information are associated using the meeting ID.

[0100] The text conversion unit 55 converts the voice data requested by the terminal device 10 to be converted to text data into text data using an external voice recognition service. The text conversion unit 55 may also perform the conversion itself.

[0101] Figure 9 shows an example of meeting information stored in the meeting information storage unit 5001, which is managed by the meeting management unit 54. The meeting management unit 54 can use the above account to obtain a list of remote meetings that the user belonging to the tenant has permission to view. Note that viewing permission may also be granted directly to the meeting information managed by the meeting management unit 54 from the information recording application of the terminal device 10. Furthermore, the information of remote meetings that a user belonging to the tenant has permission to view includes information of meetings created by the user and information of meetings that the user has been granted viewing permission for by other users. In this embodiment, remote meetings are used as an example, but the list of remote meetings also includes meetings held in only one conference room.

[0102] Meeting information is managed using a meeting ID, which is associated with participants, title (meeting name), start date and time, end date and time, location, etc. These are just examples of meeting information; meeting information may include other details as well.

[0103] • Participants are those who attend the meeting.

[0104] The title should reflect the content of the meeting, such as the name of the meeting or the topics discussed abroad.

[0105] • The start date and time is the scheduled date and time for the meeting to begin.

[0106] • The end date and time is the scheduled date and time for the meeting to end.

[0107] The location is where the meeting is held, such as a conference room, branch office name, or building.

[0108] As shown in Figures 8 and 9, the combined video footage recorded during the meeting is identified by the meeting ID.

[0109] Figure 10 shows the recording information stored in the recording information storage unit 5002. The recording information contains a list of combined video recordings made by all users belonging to the tenant. The recording information includes items such as meeting ID, recording ID, update date and time, title, and storage location. These items may be the same as in Figure 8. The storage location (path information such as the URL of the cloud storage system) may be entered by the user in the user settings screen of the information recording application 41 of the terminal device 10 and stored in the recording information storage unit 5002.

[0110] <Screen transition> Next, with reference to Figures 11 to 20, several screens displayed by the terminal device 10 during a remote conference will be described. Figure 11 is the initial screen 200 displayed after login by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user enters authentication information and successfully logs in, the initial screen 200 of Figure 11 is displayed.

[0111] The initial screen 200 has a fixed display button 201, a front view change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker image 204 when not distinguished), and a recording start button 205. If the meeting device 60 is already running and capturing images of the surroundings when the user logs in, the panoramic image 203 and speaker images 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether or not to start recording while viewing these images. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker images 204 are not displayed.

[0112] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or it may display only the speaker images 204 of the N people who have spoken most recently. Figure 11 shows an example where up to three speaker images 204 are displayed. There may be no speaker images 204 until a participant speaks (one is added each time a participant speaks), or three speaker images 204 of participants facing a predetermined direction may be displayed (they are swapped depending on the participant's speech).

[0113] If no one is speaking, such as immediately after the meeting device 60 is activated, a speaker image 204 will be created in a predetermined direction within the horizontal 360° (e.g., 0°, 120°, 240°). If a fixed display setting (described later) is enabled, that setting will take precedence.

[0114] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 as the speaker image 204 and zoom in on it.

[0115] Figure 12 illustrates the operation method when the fixed display button 201 is turned on. For example, the user moves a rectangular window 206 on the panoramic image 203 using a pointing device such as a mouse or touch panel. The user aligns the window 206 with a whiteboard, podium, or other elements included in the panoramic image 203. The user's operation is sent to the meeting device 60, which creates an image of the area indicated by the window within a horizontal 360°, at the same size as the speaker image 204, and sends it to the terminal device 10. In this way, the speaker image 204 can continue to display elements other than the speaker, such as the whiteboard.

[0116] Returning to Figure 11, the front view change button 202 is a button for the user to change the front view of the panoramic image 203. The user can slide the panoramic image 203 left or right with the pointing device to determine which participant is in the front view (since the panoramic image is 360° horizontally, the right and left edges coincide in terms of direction). The user's operation is sent to the meeting device 60, which changes the angle that is considered the front view out of the 360° horizontal direction to create a panoramic image and sends it to the terminal device 10.

[0117] If the meeting device 60 is not connected or powered off when the information recording app 41 is launched, the device unrecognized screen 250 shown in Figure 13 will be displayed.

[0118] Figure 13 shows an example of the device unrecognized screen 250. The device unrecognized screen 250 displays the message 251, "The device could not be recognized. Please power on the device and connect it." The user can see this to check the power and connection status of the meeting device 60.

[0119] When the user presses the recording start button 205, the information recording application 41 displays the recording settings screen 210 shown in Figure 14.

[0120] Figure 14 shows an example of the recording settings screen 210 displayed by the information recording application 41. On the recording settings screen 210, the user can set whether to record (include in recording) the panoramic image and speaker image created by the meeting device 60, as well as the desktop screen or running application screen of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or running application screen, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) will be recorded.

[0121] The camera toggle button 211 is a button that switches the recording of panoramic images and speaker images created by the meeting device 60 on and off. The camera toggle button 211 may also be configured to record panoramic images and speaker images individually.

[0122] The PC screen toggle button 212 is a button that switches the recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10 on and off. When the PC screen toggle button 212 is on, the desktop screen is recorded.

[0123] If a user wants to record an app screen, they further select an app from the app selection field 213. The app selection field 213 displays the names of apps currently running on the terminal device 10 in a dropdown format. The user can select the app to record. This app name is obtained by the information recording app 41 from the OS. The information recording app 41 can display only apps that have a UI (screen) among the running apps. The remote conferencing app 42 may be included in the selected apps. Therefore, the information recording app 41 can also record the materials displayed by the remote conferencing app 42 and the participants at each location in video. In addition, the apps displayed in the dropdown are various apps running on the terminal device, such as presentation apps, word processing apps, spreadsheet apps, document creation and editing apps, cloud electronic whiteboard apps, and web browser apps. Therefore, the user can flexibly select the app screens to include in the combined video.

[0124] Furthermore, when recording on an app-by-app basis, users can select multiple apps. The information recording app 41 can combine the screens of all selected apps when creating a combined image.

[0125] If both the camera toggle button 211 and the PC screen toggle button 212 are turned off, the recording content confirmation window 214 will display "Only audio will be recorded." This audio consists of the audio output by the terminal device 10 (audio received by the remote conferencing application 42 from other locations 101) and the audio collected by the meeting device 60. In other words, if a remote conference is being held, the audio from the remote conferencing application 42 and the audio from the meeting device 60 will be saved regardless of image recording. However, the user may selectively stop saving the audio from the remote conferencing application 42 and the audio from the meeting device 60 in their settings.

[0126] Depending on the on / off combination of the camera toggle button 211 and the PC screen toggle button 212, a combined image video is recorded as follows. The combined image is also displayed in real time in the recording content confirmation window 214.

[0127] If the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image will be displayed in the recording content confirmation window 214.

[0128] If the camera toggle button 211 is off and the PC screen toggle button 212 is on (screen is also selected), the recording content confirmation window 214 will display the desktop screen or the screen of the selected application.

[0129] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the recording content confirmation window 214 displays the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, side by side.

[0130] Therefore, there are cases where the panoramic image, speaker image, and app screen are not combined, or where the panoramic image, speaker image, and app screen are not recorded at all. However, in this embodiment, for convenience, the image created by the information recording app 41 is referred to as a combined image or combined image video.

[0131] Figure 15 shows an example of the recording content confirmation window 214 when the camera toggle button 211 is on and the PC screen toggle button 212 is off. In Figure 15, the panoramic image 203 and the speaker image 204 are displayed prominently.

[0132] Figure 16 shows an example of the recording content confirmation window 214 when the camera toggle button 211 is ON and the PC screen toggle button 212 is ON. In Figure 16, the panoramic image 203 and speaker image 204 are displayed on the left, and the application screen 217 is displayed on the right.

[0133] Therefore, the recording content confirmation window 214 allows the user to confirm, before recording starts, what kind of combined image video (especially images from the meeting device 60) will be recorded according to the settings on the recording settings screen 210. The information recording application 41 accepts the instruction to start recording (pressing the recording start button) while displaying the images to be included in the combined image video of the recorded information. The information recording application 41 also acquires screen information displayed in the selected application (e.g., a remote conferencing application) at the time the recording start instruction is made, and image information of the device's surroundings acquired from the device from the time the recording start instruction is made, until the instruction to end recording (pressing the recording end button), and can use both to create a combined image.

[0134] Figure 16 shows an example of a combined image / video display when only one app is selected. However, if two or more apps are selected, the screens of the second and subsequent apps will be sequentially linked to the right. Alternatively, the screens of the second and subsequent apps may be arranged in a two-dimensional (vertical and horizontal) layout.

[0135] Let's return to Figure 14 for explanation. The recording settings screen 210 has a checkbox 215 along with the message, "Automatically transcribe the recording after uploading it." The recording settings screen 210 also has a "Start Recording Now" button 216. When the user checks checkbox 215, the text data of the spoken words during the remote meeting is attached to the combined video image. In this case, after the recording is finished, the information recording application 41 uploads the audio to the information processing system 50 along with a request for conversion to text data. Also, when the user presses the "Start Recording Now" button 216, the recording screen 220 in Figure 17 is displayed.

[0136] Figure 17 shows an example of the recording screen 220 displayed by the information recording application 41 during recording. The explanation of Figure 17 mainly focuses on the differences from Figure 14. The recording screen 220 is a screen that displays the combined image video being recorded in real time according to the conditions set by the user in the recording settings screen 210. The recording screen 220 can be displayed while the remote conferencing application is running. In Figure 17, the recording screen 220 is shown with the camera toggle button 211 on and the PC screen toggle button 212 off, and displays the panoramic image 203 and speaker image 204 (both videos) created by the meeting device 60. The recording screen 220 displays the recording icon 225, the pause button 226, and the recording end button 227.

[0137] Furthermore, if the user sets the PC screen toggle button 212 to ON, the recording screen 220 will display the panoramic image and speaker image alongside the desktop screen and application screen, as shown in Figure 16.

[0138] The pause button 226 is used to stop recording, and after stopping, it also accepts requests to resume recording. The recording end button 227 is used to end recording. The recording ID does not change when the pause button 226 is pressed, but it does change when the recording end button 227 is pressed. After pausing, the user can also set the recording conditions that were set on the recording settings screen 210 again when resuming recording. In that case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, when the recording end button 227 is pressed), or it may combine multiple files to form a single continuous video (for example, when the pause button 226 is pressed). Also, when the information recording application 41 plays a combined video, it may play multiple recording files continuously as a single video.

[0139] The recording screen 220 also has a "Get Information from Calendar" button 221, a meeting name field 222, a time field 223, and a location field 224. The "Get Information from Calendar" button 221 is a button for the user to retrieve meeting information from the meeting management system 9. When the "Get Information from Calendar" button 221 is pressed, the information recording application 41 retrieves and displays a list of meetings that the user has permission to view from the information processing system 50. The user selects the remote meeting to be held from the meeting list. As a result, the meeting information is reflected in the meeting name field 222, the time field 223, and the location field 224. The meeting name field 222 displays the title of the meeting information, the time field 223 displays the start and end times, and the location field 224 displays the location. In addition, the meeting information and recording information in the meeting management system are associated with each other by the meeting ID.

[0140] Once the remote meeting ends and the user stops recording, a combined video with audio is created.

[0141] Figure 18 shows an example of the meeting list screen 230 displayed by the information recording application 41. The meeting list screen 230 is a list of meetings, and can display a list of recorded information from remote meetings. It also includes meetings that were held only within a specific meeting room, regardless of whether they were remote meetings. The meeting list screen 230 displays a combined display of meeting information that the logged-in user has permission to view in the meeting information storage unit 5001 and information stored in the recording information storage unit 5002 that is associated with this remote meeting. Information from video recording information stored in the information storage unit 1001 may be further integrated.

[0142] The meeting list screen 230 is displayed when the user selects the meeting list tab 231 on the initial screen 200 in Figure 14. The meeting list screen 230 displays a list 236 of the recorded information that this user has permission to view. The meeting creator (minutes creator) can set viewing permissions for participants. The meeting list may be a list of saved recorded information, or a list of meeting schedules or meeting data.

[0143] The meeting list screen 230 has the following items: checkbox 232, update date and time 233, title 234, and status 235.

[0144] Checkbox 232 accepts the selection of recording files. Checkbox 232 is used when the user wants to delete multiple recording files at once.

[0145] • The update time 233 indicates the start and end times of the combined image video recording. If it has been edited, the edit time can be used instead.

[0146] • The title is 234, which is the title of the meeting (agenda, etc.). It can be copied from the meeting information or set by the user.

[0147] Status 235 indicates whether the combined image / video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed; if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, the upload button is displayed. If there are unuploaded combined image / videos, the information recording application 41 should automatically upload them when the user logs into the information processing system 50.

[0148] When the user selects a title or other item from the list of combined images and videos 236 using a pointing device, the information recording application 41 displays the recording playback screen 240 shown in Figure 19. The recording playback screen 240 allows playback of the combined images and videos.

[0149] Figure 19 shows an example of the recording playback screen 240 displayed by the information recording application 41 after selecting a combined image video. The recording playback screen 240 has a playback image display area 241, a transcription button 242, one or more text display areas 243, an auto-scroll button 244, and a search button 245.

[0150] The playback image display area 241 includes a play button 241a, a rewind button 241b, a fast-forward button 241c, a time indicator 241d, a playback speed button 241e, a volume button 241f, etc. The playback image display area 241 plays and displays a combined video. In Figure 19, the combined video in the playback image display area 241 has a panoramic image and a speaker image on the left, and the screen of the remote conferencing application 42 is displayed on the right. The screen of the remote conferencing application 42 changes between an image of the location and an image of the document during the remote conferencing. Therefore, the user can view the screen of the desired scene by operating various buttons.

[0151] If the audio data of the combined video displayed in the playback image display area 241 has been converted to text data, the spoken content will be displayed in text in the text display area 243.

[0152] The transcription button 242 is a button that allows the user to toggle whether or not the text data displayed in the text display field 243 is displayed in sync with the playback time of the combined image video.

[0153] The automatic scroll button 244 is a button that allows the user to toggle whether or not the text data is automatically scrolled, regardless of the playback time.

[0154] • Search button 245 is a button that allows users to search text data by specifying keywords.

[0155] Additionally, the recording playback screen (240) may allow downloading of combined images and videos.

[0156] Figure 20 shows an example of the editing screen 260 for a combined image video. The editing screen 260 is accessed automatically from the recording screen 220, or when the user performs a predetermined operation on the recording playback screen 240. The editing screen 260 has a first display area 261 and a second display area 262. The first display area 261 displays the combined image at a certain moment during playback, and the second display area 262 displays the frames that make up the combined image video in chronological order. The user can select one or more frames to delete unnecessary frames. The user can also extract a portion of a frame and insert it after any frame. The editing processing unit 21 edits the combined image video according to the user's operation and either overwrites the edited combined image video or saves it separately.

[0157] <Procedure for operation or processing> Next, we will explain the operations and processes performed by the record information creation system 100 based on the above configuration.

[0158] <<Login>> Figure 21 is an example of a sequence diagram illustrating the procedure or process by which a user logs into the information processing system 50 by operating an information recording application 41 running on a terminal device 10.

[0159] S1: The user inputs an operation to launch the information recording application 41 on the terminal device 10.

[0160] S2: In response to the operation, the terminal device 10 launches the information recording application 41.

[0161] S3: When the information recording application 41 is launched, the communication unit 11 automatically communicates with the information processing system 50 and requests a login screen.

[0162] S4: The communication unit 51 of the information processing system 50 receives a request for a login screen and sends the screen information of the login screen generated by the screen generation unit 53 to the information recording application 41.

[0163] S5: The communication unit 11 of the information recording application 41 receives screen information from the login screen, and the display control unit 13 displays the login screen.

[0164] S6: The user enters authentication information to log in to the tenant into the information recording application 41. The operation reception unit 12 of the information recording application 41 receives the input.

[0165] S7: The communication unit 11 of the information recording application 41 sends a login request to the information processing system 50, specifying the authentication information.

[0166] S8: The communication unit 51 of the information processing system 50 receives a login request, and the authentication unit 52 authenticates the user based on the authentication information. Here, we assume that the authentication was successful.

[0167] S9: The communication unit 51 of the information processing system 50 sends a login success message to the information recording application 41.

[0168] S10: The communication unit 11 of the information recording application 41 receives a login success notification, and the display control unit 13 displays the initial screen 200.

[0169] <<Saving combined images and videos>> Next, we will explain the process of saving the combined image video, referring to Figure 22. Figure 22 is an example of a sequence diagram showing the procedure by which the information recording application 41 records a panoramic image, a speaker image, and the application screen.

[0170] S21: The user starts a remote conference by operating the remote conferencing application 42. Here, it is assumed that the remote conferencing applications 42 at the local site 102 and the other site 101 have started a remote conference. The remote conferencing application 42 at local site 102 transmits the image captured by the camera and the audio collected by the microphone of the terminal device 10 to the remote conferencing application 42 at the other site 101. The remote conferencing application 42 at the other site 101 displays the received image on its display and outputs the received audio from its speaker. Similarly, the remote conferencing application 42 at the other site 101 transmits the image captured by the camera and the audio collected by the microphone of the terminal device 10 to the remote conferencing application 42 at local site 102. The remote conferencing application 42 at local site 102 displays the received image on its display and outputs the received audio from its speaker. Each remote conferencing application 42 repeats this process to realize a remote conference.

[0171] S22: The user configures the recording settings on the recording settings screen 210 of the information recording application 41 shown in Figure 14. The operation reception unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that both the camera toggle button 211 and the PC screen toggle button 212 are turned on.

[0172] If a user has already scheduled a remote meeting, they can display a list of remote meetings by pressing the information acquisition button 221 on the calendar in Figure 19 and select the remote meeting to which they want to associate a recorded video. Since the user is logged into the information processing system 50, the information processing system 50 identifies the remote meetings that the logged-in user has access to. The information processing system 50 sends the list of identified remote meetings to the terminal device 10, and the user selects a remote meeting that is currently in progress or will be held soon. This determines the information related to the remote meeting, such as the meeting ID.

[0173] Furthermore, users can create a meeting when creating a combined image video, even if they haven't scheduled a remote meeting in advance. The following describes how the information recording application 41 creates a meeting when creating a combined image video and obtains the meeting ID from the information processing system 50.

[0174] S23: The user instructs the information recording application 41 to start recording (by pressing the "Start Recording Now" button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0175] S24: Since remote conferencing is not selected (because the conferencing ID has not been determined), the communication unit 11 of the information recording application 41 sends a remote conferencing creation request to the information processing system 50.

[0176] S25: The communication unit 51 of the information processing system 50 receives a remote meeting creation request, the meeting management unit 54 obtains a unique meeting ID assigned by the meeting management system 9, and the communication unit 51 transmits the meeting ID to the information recording application 41.

[0177] S26: The conference management unit 54 also sends the destination of the combined image video (URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

[0178] S27: The communication unit 11 of the information recording application 41 receives the conference ID and the location where the recording file will be saved, causing the image merging unit 17 to determine that recording is ready and to start recording.

[0179] S28: The application screen acquisition unit 14 of the information recording application 41 requests the application screen of the application selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In Figure 22, the application selected by the user is the remote conferencing application 42.

[0180] S29: The image merging unit 17 of the information recording application 41 notifies the meeting device 60 to start recording via the device communication unit 16. When notifying, the image merging unit 17 also notifies that the camera toggle button 211 is on (requesting panoramic image and speaker image).

[0181] S30: When the terminal communication unit 61 of the meeting device 60 receives a recording start notification, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or obtained from the information processing system 50.

[0182] S31: The audio acquisition unit 15 of the information recording application 41 acquires the audio data output by the terminal device 10 (audio data received by the remote conferencing application 42).

[0183] S32: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and the synthesis request to the meeting device 60.

[0184] S33: The terminal communication unit 61 of the meeting device 60 receives voice data and a synthesis request, and the voice synthesis unit 65 synthesizes the received voice data with the surrounding voice data collected by the sound collection unit 64. For example, the voice synthesis unit 65 adds the two voice data together. Since clear sound from the surroundings of the meeting device 60 is recorded, the accuracy of text transcription of voices, especially those around the meeting device 60 (on the conference room side), is improved.

[0185] This voice synthesis can also be performed on the terminal device 10. However, by distributing the recording function to the terminal device 10 and the voice processing to the meeting device 60, the load on both the terminal device 10 and the meeting device 60 can be reduced. Alternatively, the recording function may be distributed to the meeting device 60 and the voice processing to the terminal device 10.

[0186] S34: Also, since the meeting device 60 has received that the camera toggle button 211 is on, the panoramic image creation unit 62 creates a panoramic image and the speaker image creation unit 63 creates a speaker image.

[0187] S35: The device communication unit 16 of the information recording application 41 repeatedly acquires panoramic images and speaker images from the meeting device 60. The device communication unit 16 also repeatedly requests and acquires synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making requests to the meeting device 60. Alternatively, the meeting device 60 may automatically transmit the panoramic images and speaker images upon receiving a request to synthesize audio data. The meeting device 60 may automatically transmit the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0188] S36: The image merging unit 17 of the information recording application 41 creates a combined image by arranging the application screen acquired from the remote conferencing application 42, a panoramic image, and a speaker image. The image merging unit 17 repeatedly creates combined images and creates a combined image video by specifying each combined image as a frame that makes up the video. The image merging unit 17 also stores the audio data received from the meeting device 60.

[0189] The information recording application 41 repeats the above steps S31 to S36.

[0190] S37: When the remote meeting ends and recording is no longer needed, the user instructs the information recording application 41 to stop recording (for example, by pressing the recording stop button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0191] S38: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. As a result, the meeting device 60 terminates the creation of panoramic images and speaker images and the synthesis of audio.

[0192] S39: The image merging unit 17 of the information recording application 41 combines audio data with the combined image video to create a combined image video with audio.

[0193] S40: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the recording settings screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. Specifically, the audio data processing unit 18 sends a request to the information processing system 50 via the communication unit 11 to specify the URL of the save location, along with the conference ID and recording ID, to convert the audio data combined into the combined image video.

[0194] S41: The communication unit 51 of the information processing system 50 receives a request to convert audio data, and the text conversion unit 55 converts the audio data into text data using the speech recognition service system 80. The communication unit 51 saves the text data to the same storage location (URL of the storage service system 70) as the storage location of the combined image video. In the recording information storage unit 5002, the text data is associated with the combined image video by the conference ID and recording ID. The text data may also be managed by the conference management unit 54 of the information processing system 50 and stored in the storage unit 5000. Alternatively, the terminal device 10 may request speech recognition from the speech recognition service system 80 and save the text data obtained from the speech recognition service system 80 to the storage location. The speech recognition service system 80 returns the converted text data to the information processing system 50, but may also send it directly to the storage URL. The speech recognition service system 80 may select or switch between multiple services according to the settings information set by the user in the information processing system 50.

[0195] S42: The upload unit 20 of the information recording application 41 saves the combined image video to the combined image video storage location via the communication unit 11. In the recording information storage unit 5002, the combined image video is associated with the conference ID and the recording ID. Uploaded combined image video is recorded.

[0196] Since the save location is notified to the user, the user can share the combined image / video with participants by informing them of the save location via email or other means. Even if the devices used to create the combined image / video, audio data, and text data are different, they can be collected and stored in a single location, making them easily viewable by users later.

[0197] In the case of real-time speech recognition, the meeting device 60 or terminal device 10 transmits speech data to the information processing system 50 in real time. The terminal device 10 displays and saves the text data transmitted from the meeting device 60 or returned from the information processing system 50 on the recording screen 220.

[0198] Note that the processing in steps S31 to S36 does not have to be in the order shown in Figure 22, and the order in which the audio data is synthesized and the combined image is created may also be different.

[0199] <<Settings changed while creating a combined image video>> Next, referring to Figure 23, we will explain what happens when the user changes the settings on the recording settings screen 210 while the information recording application 41 is creating a combined image video. Figure 23 is an example of a sequence diagram showing the procedure by which the information recording application 41 records a panoramic image, a speaker image, and the application screen with the changed settings when the settings on the recording settings screen 210 are changed while the combined image video is being created. Note that the explanation of Figure 23 will mainly focus on the differences from Figure 22.

[0200] In Figure 23, during the repetition of steps S31 to S36, the user changed the settings on the recording settings screen 210 in step S51. If the camera toggle button 211 and the PC screen toggle button 212 were turned on before the setting change, the following changes are possible, for example. The image merging unit 17 generates a merged image video based on the changed settings.

[0201] a. Turn off the camera toggle button 211 and the PC screen toggle button 212. b. Turn off camera toggle button 211 and turn on PC screen toggle button 212. c. The PC screen toggle button 212 is in the ON state, but the app used to save the screen has been changed. Figure 23 explains case b. S52: Since the camera toggle button 211 has been turned off, the image merging unit 17 of the information recording application 41 notifies the meeting device 60 of the end of recording via the device communication unit 16. For this reason, in Figure 23, the process of creating the panoramic image and speaker image in step S34 is omitted. Also, in step S35, only the audio data is sent from the meeting device 60 to the information recording application 41.

[0202] Furthermore, in step S36, the image merging unit 17 creates a merged image using only the application screen obtained from the remote conferencing application 42. In this case, the panoramic image and speaker image that existed when the camera toggle button 211 was on are gone, so the image merging unit 17 may either enlarge only the application screen or arrange it in the same way as when the panoramic image and speaker image were present. The image merging unit 17 may also save the merged image video as a single recording file, or it may obtain a recording ID from the meeting device 60 for each setting change and save it as a separate recording file.

[0203] In case a., step S28 becomes unnecessary, and the app screen is not displayed in step S36.

[0204] In case c, the screen of the application acquired by the information recording application 41 in step S28 is changed.

[0205] Thus, even if the recording settings are changed while creating the combined image video, the information recording app 41 can create the combined image video according to the changes. By switching the app being recorded during recording, the user can include the screen of the app that was displayed during the remote meeting in the combined image video.

[0206] <<Save the screens of all running apps>> Next, referring to Figure 24, we will explain the case where the screens of all running applications are saved, and after the remote meeting ends, the information recording application 41 creates a combined image video. Figure 24 is an example of a sequence diagram showing the procedure for saving the screens of all running applications and changing the images (videos) to be included in the recorded information after the remote meeting ends. Note that the explanation of Figure 24 will mainly focus on the differences from Figure 22.

[0207] In Figure 24, in step S28, the application screen acquisition unit 14 of the information recording application 41 acquires the screens of all running applications via the OS, etc. The terminal device 10 saves each application's screen as a video, associated with the application's identification information.

[0208] In step S36, the image merging unit 17 merges the application screen, panoramic image, and speaker image according to the recording settings, and the display control unit 13 displays the merged image.

[0209] Then, when recording ends in step S37, in step S39, the image merging unit 17 merges the panoramic image and speaker image with the screens of all apps selected on the recording settings screen 210. If the camera toggle button 211 is off, the panoramic image and speaker image are not merged.

[0210] In this way, by saving the screens of all running applications, even if the user changes the recording settings screen 210 during recording, the application screens can be included in the combined video from the start of the remote meeting. Users can change the recording settings after the remote meeting ends, and by recording all application screens, they can later decide which application screens to include in the recording information.

[0211] <Communication between terminal device and meeting device during recording> Next, with reference to Figure 25, we will explain what happens when communication between the terminal device 10 and the meeting device 60 is interrupted during recording. Figure 25 is an example of a sequence diagram showing the procedure for appropriately terminating recording when communication between the terminal device 10 and the meeting device 60 is interrupted during recording. Note that the explanation of Figure 25 will mainly focus on the differences from Figure 22.

[0212] S61: The user accidentally unplugged the USB cable connecting terminal device 10 and meeting device 60. Other examples of communication interruption include a malfunction in the wireless LAN router or the meeting device 60 being powered off.

[0213] S62: The device communication unit 16 of the information recording application 41 detects that the USB cable has been unplugged, for example, when the external device connection interface does not detect voltage. The device communication unit 16 may also detect a communication interruption due to no response from the meeting device 60, etc.

[0214] S63: If communication with the meeting device 60 is not possible, the information recording application 41 cannot acquire the synthesized audio data even if the camera toggle button 211 is off. For this reason, the information recording application 41 terminates recording. Subsequent processing can be the same as in steps S37 onwards in Figure 22.

[0215] <When each app resides in the cloud and the meeting device connects to the cloud app> Next, referring to Figure 26, we will describe the system configuration when each application resides in the cloud and the meeting device 60 and the cloud-side applications communicate. Figure 26 shows an example configuration of the recording information creation system 100. The configuration and communication connection relationship for the information recording application 41 to acquire images of the surroundings of the meeting device 60 and the screen displayed by the remote conferencing application 42 may vary as follows.

[0216] Figure 26(a) shows an example configuration where the remote conferencing application 42 and the information recording application 41 operate on separate terminal devices. In this case, the surrounding image information captured by the meeting device 60 is transmitted from the meeting device 60 installed at the base station to the information recording application 41.

[0217] Furthermore, the information recording application 41 also acquires screen information displayed by the remote conferencing application 42, and uses the image information of the surroundings of the meeting device 60 and the screen information displayed by the remote conferencing application 42 to create recorded information.

[0218] Figure 26(b) shows a configuration similar to that in Figure 2. The information recording application 41, which runs on the terminal device 10, acquires screen information displayed by the remote conferencing application 42, which also runs on the terminal device 10, as well as audio from the remote conferencing, image information of the area around the meeting device 60, and audio from the base station, and creates recorded information. The terminal device 10 and the meeting device 60 are directly connected locally (USB, BLE, Wi-Fi).

[0219] Figure 26(c) shows an example configuration in which the information recording application 41 operates on the information processing system 50. The information processing system 50 (on the cloud) executes the information recording application 41 and acquires surrounding image information from the meeting device 60 at the base and screen information from the remote conferencing application 42 from the terminal device 10. Therefore, the information recording application 41 is created by a web application that is executed on both the information processing system 50 and the terminal device 10.

[0220] The information recording application 41 of the information processing system 50 performs the main processing for creating recorded information, while the web browser application 120 of the terminal device 10 performs processing related to UI display and input. The information recording application 41 on the information processing system 50, the meeting device 60, and the terminal device 10 are connected via the Web (Internet).

[0221] The information recording application 41 on the information processing system 50 acquires screen information displayed by the remote conferencing application 42 running on the terminal device 10, as well as audio from the remote conferencing, via the Web (Internet). The information recording application 41 on the information processing system 50 also acquires image information of the surroundings of the meeting device 60 and audio from the location via the Web, and creates recorded information. Other processing is described in this embodiment.

[0222] Figure 26(d) shows an example configuration in which a remote conferencing service system 90 and an information processing system 50 communicate. The information processing system 50 is equipped with an information recording application 41, which acquires ambient image information from a meeting device 60 at a base station and screen information from a remote conferencing application 42 from another cloud service (remote conferencing service system). The information recording application 41 on the information processing system 50 side, the meeting device 60, and the cloud service side of the application (remote conferencing application 42) are connected via the Web (Internet). The information recording application 41 on the information processing system 50 side acquires screen information and audio displayed by the remote conferencing application 42 from the cloud service side of the remote conferencing application 42. The rest is the same as in Figure 26(c).

[0223] <Main effects> As described above, the recording information creation system 100 of this embodiment displays panoramic images of the surroundings including the user and images of the speaker in the combined image video, and also displays the screen of the application displayed during the remote meeting, such as the remote conferencing application 42. When participants or non-participants of the remote meeting view the combined image video as meeting minutes, the atmosphere of the remote meeting is reproduced with a sense of presence. Furthermore, since the information recording application 41 records both the screen information displayed by the application selected in the information recording application 41 (such as the remote conferencing application) and the image information of the surroundings of the device within the base (such as the conference room), even if the screen displayed by the remote conferencing application 42 changes, it is possible to create recording information that comprehensively records the state of the base (such as the conference room) along with the content of the remote meeting (remote communication).

[0224] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0225] For example, the terminal device 10 and the meeting device 60 may be integrated into one unit. Alternatively, the meeting device 60 may be attached externally to the terminal device 10. Furthermore, the meeting device 60 may consist of a 360-degree camera, a microphone, and a speaker connected by cables.

[0226] Meeting devices 60 may also be placed at other locations 101. These other locations 101 will separately use meeting devices 60 to create combined image / video and text data. Furthermore, multiple meeting devices 60 may be placed at a single location. In this case, multiple recordings will be created for each meeting device 60.

[0227] Furthermore, the arrangement of the panoramic image 203, speaker image 204, and application screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be at the bottom and the speaker image 204 at the top, or the user may change the arrangement, or individually switch the display of the panoramic image 203 and speaker image 204 during playback.

[0228] Furthermore, the configuration examples shown in Figure 7 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the terminal device 10, meeting device 60, and information processing system 50. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the terminal device 10, meeting device 60, and information processing system 50 can be further divided into more processing units depending on the processing content. Alternatively, a single processing unit can be divided to include even more processing.

[0229] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0230] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figure 22, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0231] Each of the embodiments described above can be implemented by one or more processing circuits. Here, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above. [Explanation of symbols]

[0232] 10 Terminal devices 50 Information Processing Systems 60 Meeting Devices 100 Record Information Creation System [Prior art documents] [Patent Documents]

[0233] [Patent Document 1] Japanese Patent Publication No. 2020-115609

Claims

1. A system for creating recorded information, which includes an information recording application executed on a terminal device, The information recording application is characterized by creating recording information based on screen information displayed by a remote conferencing application selected by the information recording application and image information of the device's surroundings acquired by the device.

2. The recording information creation system according to claim 1, wherein the screen information is information displayed as a window by a running application including the remote conferencing application and acquired as an image by the information recording application.

3. The recording information creation system according to claim 1 or 2, wherein the image information surrounding the device is image information obtained by imaging a 360° region surrounding the device.

4. The recording information creation system according to any one of claims 1 to 3, wherein image information of the surroundings of the device is transmitted from the device installed at the base to the information recording application.

5. The aforementioned information recording application is The image merging unit has an image merging unit that creates a combined image in which the screen information and the surrounding image information are arranged on a single screen. The image merging unit creates a combined image video consisting of a plurality of the combined images. The recording information creation system according to any one of claims 1 to 4, characterized in that the recording information includes the combined image video.

6. The recording information creation system according to claim 5, further characterized in that the image merging unit links one or more speaker images, from which the speaker has been cut out of the surrounding image information, to the surrounding image information to create the merged image.

7. A voice acquisition unit that acquires first voice data output by the terminal device, The device includes a speech synthesis unit that synthesizes the first speech data acquired by the speech acquisition unit with the second speech data collected by the device, The recording information creation system according to claim 5 or 6, characterized in that the synthesized audio data is combined with the combined image video.

8. The recording information creation system according to claim 7, characterized in that the image merging unit and the speech synthesis unit are distributed between the device and the information recording application of the terminal device.

9. An upload unit that saves the combined image video to a storage service system via a network, The system includes an audio data processing unit that stores text data converted from the synthesized audio data in association with the combined image video in the storage service system, The recording information creation system according to claim 7 or 8, characterized in that the recording information includes the combined image video and the text data.

10. The information processing system further includes a memory unit that stores user settings, The above setting includes the location where the recorded information is stored. Depending on the configured storage location, the text data and the combined image video are saved to the storage service system. The record information creation system according to claim 9.

11. The aforementioned information recording application is The unit has an operation receiving unit that accepts settings for whether or not to include the surrounding image information and the screen information displayed by the running application in the combined image. The recording information creation system according to any one of claims 7 to 9, characterized in that the image merging unit creates the combined image video including the surrounding image information and the screen information, the combined image video including either the surrounding image information or the screen information, or the synthesized audio data not including either the surrounding image information or the screen information, according to the settings received by the operation reception unit.

12. When the operation reception unit receives a setting to include the screen information displayed by the running application in the combined image, it further receives a selection of the application to be included in the combined image from among a plurality of applications, including the remote conferencing application. The recording information creation system according to claim 11, characterized in that the image merging unit includes the screen information of the application selected by the operation reception unit in the merged image video.

13. The recording information creation system according to any one of claims 1 to 12, wherein the remote conferencing application is an application that performs remote communication with other terminal devices.

14. The aforementioned information recording application is The operation receiving unit has a display control unit that displays an image to be included in the combined image selected by the operation receiving unit, The recording information creation system according to claim 11 or 12, characterized in that the display control unit displays a combined image video using the image selected by the operation reception unit before the start of recording.

15. The aforementioned information recording application is With the images to be included in the combined image displayed, the recording start command is received. The screen information displayed in the selected remote conferencing application, which is the screen information displayed when the recording start command was issued, Image information of the surroundings of the device acquired from the device, including image information acquired from the time the recording start instruction was given, The recording information creation system according to claim 14, which acquires the recording until the instruction to end recording and uses them together to create the combined image.

16. When the start of recording of the aforementioned combined image video is accepted, The recording information creation system according to any one of claims 5 to 12, characterized in that the combined image video created by the image combining unit is displayed while the remote conferencing application is running during the recording process.

17. If, after the creation of the combined image has begun, the operation reception unit receives a change in the setting for whether or not to include the surrounding image information and the screen information displayed by the running application in the combined image, The recording information creation system according to claim 11 or 12, characterized in that the image merging unit creates the merged image video according to the modified settings received by the operation reception unit.

18. The terminal device has an application screen acquisition unit that stores the screen information of all applications currently running on the terminal device. The recording information creation system according to claim 11 or 12, characterized in that when the creation of the combined image is completed, the image combining unit creates the combined image video according to the settings received by the operation receiving unit.

19. The display control unit displays an editing screen that accepts editing of the combined image video after recording has finished. The recording information creation system according to claim 14, characterized in that the operation reception unit accepts editing of the combined image video.

20. A method for creating recorded information performed by a recording information creation system that includes an information recording application executed on a terminal device, The method for creating recorded information is characterized in that the information recording application creates recorded information based on screen information displayed by a remote conferencing application selected by the information recording application and image information of the device's surroundings acquired by the device.

21. A program that runs on a terminal device, A program characterized in that it causes the terminal device to create recorded information based on screen information displayed by the remote conferencing application selected by the program and image information of the device's surroundings acquired by the device.

Images