Meeting device, image creation method, program and terminal device

Abstract

To provide a recording information creation system that creates recording information for remote communication on the basis of image information acquired from an application being executed and image information around a device and reproduces a scene at time of performing the remote communication while imaging the inside of a conference room.SOLUTION: The present invention provides a recording information creation system that comprises an information recording application which is executed on a terminal device. The information recording application is characterized by creating recording information on the basis of screen information that is displayed by a remote conference application which is selected in the information recording application and image information around a device acquired by the device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a recording information creation system, a recording information creation method, and a program.

Background Art

[0002] There is known a remote conferencing system that transmits images and voices in real time from one base to one or more other bases and enables users at remote locations to hold meetings using images and voices.

[0003] Also known is a technique for storing conference participants as omnidirectional images (see, for example, Patent Document 1). Patent Document 1 discloses an audio recording device that images participants in a conference room with an omnidirectional camera, identifies the face regions of each participant, and determines whether to record audio according to the direction of the face.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the prior art has a problem in that recording information for remote communication is not created based on screen information acquired from a running application and an image of the surroundings. For example, simply storing information obtained by imaging the inside of a conference room makes it difficult to reproduce the state during remote communication while imaging the inside of the conference room.

[0005] In view of the above problems, an object of the present invention is to provide a recording information creation system that creates recording information for remote communication based on screen information acquired from a running application and image information of the surroundings of a device, and reproduces the state during remote communication while imaging the inside of a conference room.

Means for Solving the Problems

[0006] In view of the above problems, the present invention is a recording information creation system including an information recording application executed by a terminal device, wherein the information recording application creates recording information based on screen information displayed by a remote conference application selected in the information recording application and image information of the surroundings of the device acquired by the device.

Effect of the Invention

[0007] It is possible to provide a recording information creation system that creates recording information for remote communication based on screen information acquired from a running application and image information of the surroundings of the device, and reproduces the state when remote communication is performed while imaging the inside of the conference room.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Mode for Carrying Out the Invention

[0009] Hereinafter, as an example of a mode for carrying out the present invention, a recording information creation system and a recording information creation method performed by the recording information creation system will be described.

[0010] <An Example of a Method for Creating Minutes of a Remote Meeting> First, with reference to FIG. 1, an outline of a method for creating minutes using a panoramic image and the screen of an app will be described. FIG. 1 is a diagram for explaining an outline of the creation of recording information that saves the screens of apps executed during a remote meeting together with the surrounding panoramic image. As shown in FIG. 1, a user at the home base 102 shown is using the remote meeting service system 90 to hold a remote meeting with another base 101.

[0011] The recording information creation system 100 of the present embodiment creates recording information (such as minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) obtained by processing information obtained by imaging the surroundings by an imaging means capable of imaging 360° around and a meeting device 60 equipped with a microphone and a speaker, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. Regarding the audio, the recording information creation system 100 synthesizes the audio received by the remote meeting app 42 and the audio obtained by the meeting device 60 and includes it in the recording information. Hereinafter, an outline will be described.

[0012] (1) In the terminal device 10, an information recording application 41 and a remote conferencing application described later are operating. In addition, an application for displaying materials or the like may also be operating. The information recording application 41 transmits the voice output from the terminal device 10 (including the voice received from another site by the remote conferencing application. An example of first voice data.) to the meeting device 60. The meeting device 60 mixes (combines) the voice it has acquired (an example of second voice data) with the voice of the remote conferencing application.

[0013] (2) The meeting device 60 is equipped with a microphone, and based on the direction from which the voice was acquired, it performs a process of cutting out the speaker from the panoramic image and creates a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 operating on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines an arbitrary application screen selected by the user (for example, the screen 103 of the remote conferencing application), the panoramic image 203, and the speaker image 204. For example, the panoramic image 203, the speaker image 204, and the screen 103 of the application are combined so that the panoramic image 203 and the speaker image 204 are arranged on the left side and the screen 103 of the remote conferencing application is arranged on the right side (hereinafter referred to as the combined image 105). Note that the application screen is an example of the screen information (described later) displayed by each application such as the remote conferencing application. Since the process in (3) is repeatedly executed, the combined image 105 becomes a video (hereinafter referred to as the combined image video). In addition, the information recording application 41 combines the voice synthesized with the combined image video to create a video with voice.

[0015] In this embodiment, an example of combining the panoramic image 203, the speaker image 204, and the screen 103 of the application is described, but the information recording application 41 may save these separately and arrange them on the screen during playback.

[0016] (4) The information recording application 41 accepts an editing operation (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes a part of the recording information.

[0017] (5) The information recording application 41 transmits the created combined image video (with audio) to the storage service system 70 and stores it.

[0018] (6) Also, the information recording application 41 extracts only the audio from the combined image video (it may also take the audio before combination), and transmits the extracted audio to the information processing system 50. The information processing system 50 transmits the audio to the speech recognition service system 80 that converts the audio into text data, and the audio is texturized. The text data also includes data such as how many minutes after the recording started the speech was made.

[0019] In the case of real-time texturization, the meeting device 60 directly transmits the audio to the information processing system 50.

[0020] (7) The information processing system 50 additionally stores the text data in the storage service system 70 that stores the combined image video. The text data constitutes a part of the recording information.

[0021] Note that the information processing system 50 can execute a charging process according to the services used by the user. For example, the charging is calculated based on the amount of text data, the file size of the combined image video, the processing time, etc.

[0022] In this way, in the combined image video, panoramic images and speaker images of the surroundings including the user are displayed, and furthermore, the screens of applications displayed during the remote meeting, such as the remote meeting application 42, are displayed. When participants or non-participants of the remote meeting view the combined image video as a meeting record, the situation during the remote meeting is reproduced with a sense of presence.

[0023] <Regarding terms> An application (app) is software developed and used for a specific function or purpose. Applications include native apps and web apps. Also, a web app (an app on the cloud side provided by a cloud service) may operate in cooperation with a native app or a web browser.

[0024] An app during execution refers to the app in the state from when the app is launched until it is terminated. The app does not have to be active (the foremost app), and it may operate in the background.

[0025] A device is a device that can capture images of the surroundings of the device and collect ambient sounds. The device may include those used by connecting to a terminal device, those built into a terminal device, and those used by connecting to a cloud service without directly connecting to a terminal device. In this embodiment, it is described using the term meeting device.

[0026] The image information around the meeting device acquired by the meeting device is the image information obtained by the meeting device imaging the surrounding space (for example, an area or space of 180° to 360° in the horizontal direction) surrounding the meeting device, and refers to an image obtained by performing a predetermined process on the image information of the curved surface imaged by the meeting device. The predetermined process is various processes for creating surrounding image information from the imaged information, such as a flattening process for the imaged curved surface image. The predetermined process may include, in addition to the process of creating a peripheral image, a process of cutting out a speaker image, and a combining process of combining a peripheral image, a speaker image, etc. In the present embodiment, the surrounding image is described by the term panoramic image. A panoramic image is an image having an angle of view of approximately 180° to 360° in the horizontal direction. The meeting device does not have to capture a panoramic image with one device, and a plurality of imaging devices with a normal angle of view may be combined. The meeting device is assumed to be installed and used at an installation location such as a table for meetings at a base or for grasping the surrounding situation. In addition, it may be a device used for monitoring (such as security and disaster prevention), watching over (such as childcare and nursing care), and analyzing the situation at the site (such as solutions and marketing).

[0027] The recorded information is the information recorded by the information recording application 41, and is information that is stored and saved in a browsable manner as information associated with the identification information of a certain meeting (conference), and includes, for example, the following information. · Video information created based on the screen information displayed by the selected application (such as a remote conferencing application) and the image information around the device acquired by the device. · Audio information acquired and synthesized by a remote conferencing application (terminal device) and a meeting device at the base during a meeting (conference). · Text information obtained by converting the acquired audio into text. · Other data and images that are related information related to the meeting (conference). For example, document files used during the meeting, added memos, translation data of text-converted data, images and stroke data created by a cloud electronic blackboard service during the meeting, etc. and the like. When the information recording application 41 records the screen of the remote conferencing application or the state of the meeting at the base, the recorded information may become the minutes of the meeting that was held. The minutes are an example of recorded information, and the way of referring to the recorded information changes according to the content of the remote meeting or the meeting on the base side. For example, it may be referred to as a record of communication, a record of the base situation, or the like. Also, the recorded information includes files in a plurality of formats such as, for example, video files (combined image videos, etc.), audio files, text data (text data obtained by voice recognition of voices), document files, image files, and table format files. Since the files are associated with each other with respect to the identification information of the meeting, they can be viewed collectively, selectively, or in chronological order when viewed.

[0028] A tenant is a group of users (companies, local governments, some organizations thereof, etc.) that have contracted to receive services from a service provider. The creation of the recorded information and the conversion into text data in this embodiment are executed because the tenant has a contract with the service provider.

[0029] Remote communication means communicating through voice or video by utilizing software and terminal devices with a counterpart who is at a physically distant base. An example of remote communication is a remote meeting, and the meeting may be called an assembly, a meeting, a consultation, an application for a contract, etc., a gathering, a meeting, a seminar, a training session, a study session, a seminar, a training workshop, or the like.

[0030] A base means a place where activities are centered. An example of a base is a meeting room. A meeting room is a room that is mainly installed for the purpose of using it for meetings. The base may be various places such as a home, a reception, a store, a warehouse, or an outdoor site, as long as it is a place or space where terminal devices and devices can be installed.

[0031] Voice refers to the language sounds and ambient sounds emitted by humans, and voice data is the data obtained by digitizing voice. However, in this embodiment, it will be described without strict distinction.

[0032] <System configuration example> Next, with reference to FIG. 2, the system configuration of the recording information creation system 100 will be described. FIG. 2 shows a configuration example of the recording information creation system 100. In FIG. 2, one of the multiple sites (own site 102) where a remote conference is held is shown, and the terminal device 10 at the own site 102 communicates with the information processing system 50, the storage service system 70, and the remote conference service system 90 via a network. In the own site 102, a meeting device 60 is further arranged, and the terminal device 10 is communicably connected to this meeting device 60 via a USB cable or the like.

[0033] In the terminal device 10, at least the information recording application 41 and the remote conference application 42 operate. The remote conference application 42 can communicate with the terminal devices at other sites 101 via the remote conference service system 90 on the network, enabling users at each site to hold a conference from a remote location. The information recording application 41 creates the recording information in the remote conference conducted by the remote conference application 42 using the functions of the information processing system 50 and the meeting device 60.

[0034] Note that in this embodiment, an example of creating recording information during a remote conference is described, but the conference does not have to be a conference that communicates with a remote site. That is, the conference may be a conference in which only the participants within one site participate. In this case, only the voice collected by the meeting device 60 is saved without synthesis, and there is no change in the processing of the information recording application 41.

[0035] The terminal device 10 has a built-in camera with a normal field of view (it may also be external), and is capturing an image of the front including the user 107 who operates the terminal device 10. The normal field of view refers to an image that is not a panoramic image. In this embodiment, it is mainly a non-curved planar image such as a full-sphere image. Also, the terminal device 10 has a built-in microphone (it may also be external), and is collecting ambient sounds such as those of the user who operates the terminal device 10. Therefore, the user can conduct a conventional remote meeting using the remote meeting application 42 without being conscious of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote meeting application 42 except for increasing the processing load of the terminal device 10.

[0036] The information recording application 41 is an application that records information by communicating with the meeting device 60 to create recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capturing device, a microphone, and a speaker. Although the camera of the terminal device 10 can only capture a limited range in the front, the meeting device 60 can capture the entire periphery (not necessarily the entire periphery) surrounding the meeting device 60. The meeting device 60 can always keep the plurality of participants 106 shown in FIG. 2 within the field of view.

[0037] In addition, the meeting device 60 performs operations such as extracting a speaker image from a panoramic image and synthesizing the sound acquired by the meeting device 60 and the sound output by the terminal device 10 (including the sound received by the remote meeting application 42). Note that the meeting device 60 may be placed anywhere in the self-base point 102, not limited to being on an installation location such as a desk or a stand. Since the meeting device 60 can capture a full-sphere image, it may be placed on the ceiling, for example. Also, the meeting device 60 may be installed on the other base point side or at any base point.

[0038] The information recording app 41 performs functions such as displaying a list of apps running on the terminal device 10, image synthesis for the above-described recording information (creation of combined image videos), playing back the combined image videos, and accepting editing. In addition, the information recording app 41 performs functions such as displaying a list of remote conferences that have been held or are scheduled to be held. The list of remote conferences is used for information related to the recording information, allowing the user to associate the remote conferences with the recording information.

[0039] The remote conferencing app 42 is an application that enables the terminal device to communicate remotely with other terminal devices by establishing a communication connection with other terminal devices at other bases 101 and transmitting and receiving images and audio, displaying images, and outputting audio. The remote conferencing app can also be referred to as a remote communication app, a remote information sharing app, etc.

[0040] Note that the information recording app 41 and the remote conferencing app 42 can be either web apps or native apps. A web app is an app that operates through the collaboration of a program on a web server and a program or native app on a web browser, and does not require installation on the terminal device 10. A native app is an app that is installed on the terminal device 10 and used. In this embodiment, both will be described as native apps.

[0041] The terminal device 10 can be a general-purpose information processing device equipped with a communication function, such as a PC (Personal Computer), smartphone, tablet terminal, etc. Additionally, the terminal device 10 can also be an electronic blackboard, game console, PDA (Personal Digital Assistant), wearable PC, car navigation system, industrial machine, medical device, networked home appliance, etc. The terminal device 10 only needs to be a device on which the information recording app 41 and the remote conferencing app 42 can at least operate.

[0042] The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that cooperate with the information recording application 41 to perform processing, and has basic services. These server applications manage the list of remote conferences, manage the recording information recorded in remote conferences, manage various settings and storage paths, etc. The basic services perform user authentication, contract, billing processing, etc.

[0043] Note that all or part of the functions of the information processing system 50 may exist in a cloud environment or an on-premises environment. The information processing system 50 may be composed of multiple server devices or may be composed of a single information processing device. For example, the server application and the basic service may be provided by separate information processing devices, or there may be information processing devices for each function within the server application. The information processing system 50, the storage service system 70, and the speech recognition service system 80 described below may be integrated.

[0044] The storage service system 70 is a storage means on a network and provides a storage service that accepts the storage of files and the like. Known storage service systems 70 include OneDrive (registered trademark), Google Workspace (registered trademark), DropBox (registered trademark), etc. The storage service system 70 may also be an on-premises NAS (Network Attached Storage), etc.

[0045] The speech recognition service system 80 provides a service that performs speech recognition on speech data and converts it into text data. The speech recognition service system 80 may be a general-purpose commercial service or may be part of the functions of the information processing system 50. Also, the speech recognition service system 80 may set and use different service systems for each user, tenant, and conference.

[0046] <Hardware Configuration Example> Referring to FIG. 3, the hardware configurations of the information processing system 50 and the terminal device 10 according to this embodiment will be described.

[0047] <<Information Processing System and Terminal Device>> FIG. 3 is a diagram showing an example of the hardware configurations of the information processing system 50 and the terminal device 10 according to this embodiment. As shown in FIG. 3, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0048] Among these, the CPU 501 controls the operations of the entire information processing system 50 and the terminal device 10. The ROM 502 stores programs used for driving the CPU 501 such as the IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to and from the HD 504 according to the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, characters, or an image. The external device connection I / F 508 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory, a printer, and the like. The network I / F 509 is an interface for performing data communication using a network. The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 501 shown in FIG. 3.

[0049] In addition, the keyboard 511 is a type of input means having a plurality of keys used for inputting characters, numerical values, or various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The optical drive 514 controls reading or writing of various data with respect to the optical storage medium 513 as an example of a removable recording medium. Note that the optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), or the like. The media I / F 516 controls reading or writing (storage) of data with respect to the recording medium 515 such as a flash memory.

[0050] <<Meeting Device>> Using FIG. 4, the hardware configuration of the meeting device 60 will be described. FIG. 4 is an example of a hardware configuration diagram of the meeting device 60 capable of imaging a 360° video. Hereinafter, the meeting device 60 is assumed to be a device that uses an imaging element to image a 360° video around the device at a predetermined height, but the number of imaging elements may be one or two or more. Also, it does not necessarily have to be a dedicated device, and a 360° video imaging unit may be attached to a PC, digital camera, smartphone, or the like to substantially have the same function.

[0051] As shown in FIG. 4, the meeting device 60 is composed of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, an audio processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a concave terminal 621 for Micro USB.

[0052] Among these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 having a 360° angle of view for forming a hemispherical image, and an image sensor 603 provided corresponding to each wide-angle lens. The image sensor 603 is an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image formed by the fisheye lens 602 into image data of an electrical signal and outputs it, a timing generation circuit that generates a horizontal or vertical synchronization signal and a pixel clock of this image sensor, and a register group in which various commands and parameters necessary for the operation of this image sensor are set. Note that the imaging unit 601 may be a 360° camera, and is an example of imaging means capable of imaging 360° around the meeting device 60. Note that a 360° angle of view may be obtained by synthesizing a plurality of pieces of information acquired by a plurality of image sensors (for example, 180° + 180°).

[0053] The image sensors 603 (image sensors) of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensors 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the audio processing unit 609 are connected to the CPU 611 via the bus 610. Further, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, and an audio sensor 618 are also connected to the bus 610.

[0054] The image processing unit 604 captures the image data output from the image sensor 603 through the parallel I / F bus, performs predetermined processing on each piece of image data, and creates data of a panoramic image or a speaker image from the fisheye video. Further, the image processing unit 604 performs a synthesis process on the panoramic image and the speaker image, etc., and outputs one moving image.

[0055] The imaging control unit 605 generally sets commands and the like in the register group of the imaging device 603 by using the I2C bus, with the imaging control unit 605 as the master device and the imaging device 603 as the slave device. The necessary commands and the like are received from the CPU 611. Also, the imaging control unit 605 takes in status data and the like of the register group of the imaging device 603 by using the I2C bus and sends it to the CPU 611.

[0056] In addition, the imaging control unit 605 instructs the imaging devices 603a and 603b to output image data at the timing when the imaging start button of the operation unit 615 is pressed or when an imaging start instruction is received from a PC. Depending on the meeting device 60, it may have a preview display function using a display (e.g., the display of a PC or smartphone) or a function corresponding to video display. In this case, the output of image data from the imaging device 603 is continuously performed at a predetermined frame rate (frames / minute).

[0057] Also, as will be described later, the imaging control unit 605 also functions as synchronization control means for synchronizing the output timing of the image data of the imaging device 603 in cooperation with the CPU 611. Note that in this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0058] The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 through the I / F bus and performs predetermined processing on the sound data.

[0059] The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data during processing. In particular, the DRAM 614 stores image data during processing in the image processing unit 604 and data of the processed orthographic cylindrical projection image.

[0060] The operation unit 615 is a general term for operation buttons such as the imaging start button 615a. By operating the operation unit 615, the user can start imaging and recording, execute power ON / OFF, execute communication connection, and input settings such as various imaging modes and imaging conditions.

[0061] The external device connection I / F 616 is an interface for connecting various external devices. External devices in this case are, for example, a PC (Personal Computer), a display, a projector, an electronic blackboard, etc. The external device connection I / F 616 may be provided with, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. The video data and image data stored in the DRAM 614 are transmitted to an external terminal via this external device connection I / F 616 or recorded on an external medium. Also, using a plurality of external device connection I / F 616s, for example, while transmitting and recording the image information captured and acquired by the meeting device 60 to a PC via USB, video (such as screen information displayed by a remote conferencing app, etc.) is acquired from the PC to the meeting device 60, and further transmitted from the meeting device 60 to other external devices (a display, a projector, an electronic blackboard, etc.) via HDMI and displayed.

[0062] The communication unit 617 may communicate with a cloud server via the Internet by a wireless communication technology such as Wi-Fi through the antenna 617a provided in the meeting device 60 and transmit the stored video data and image data to the cloud server. Also, the communication unit 617 may communicate with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy. Registered trademark) or NFC.

[0063] The audio sensor 618 is a sensor that acquires 360° audio information in order to identify from which direction the audio is input as a loud sound in 360° around the periphery (horizontal plane) of the meeting device 60. The audio processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction at 360°.

[0064] Note that other sensors (such as azimuth / acceleration sensors and GPS) may calculate azimuth, position, angle, acceleration, etc., and be used for image correction and adding position information.

[0065] In addition, the image processing unit 604 performs the following processing.

[0066] · The CPU 611 creates a panoramic image in the following manner. The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB complementary processing) on the RAW data input from the image sensor that inputs a spherical video to create a fish-eye video (a video consisting of an image of a curved surface). Further, planarization processing such as DeWarp processing (distortion correction processing) is performed on the created fish-eye video (curved surface video) to create a panoramic image (a video consisting of a planar image) in which 360° around the meeting device 60 is shown.

[0067] · The CPU 611 creates a speaker image in the following manner. The CPU 611 creates a speaker image obtained by cutting out the speaker from the panoramic image (planar video) in which 360° around is shown. The CPU 611 uses the voice input direction specified from the 360° output using the voice sensor 618 and the audio processing unit 609 as the direction of the speaker, and cuts out the speaker image from the above panoramic image. At this time, the method of cutting out a person's image from the voice input direction is to cut out 30° centered on the voice direction specified from 360° and perform face detection therein for cutting out. The CPU 611 further specifies the speaker images of a specific number of people (such as 3 people) who have spoken most recently among the cut-out speaker images.

[0068] The panoramic image and one or more speaker images may be individually transmitted to the information recording application 41, or the meeting device 60 may create one image from these and transmit it to the information recording application 41. In this embodiment, it is assumed that the panoramic image and one or more speaker images are individually transmitted from the meeting device 60 to the information recording application 41.

[0069] FIG. 5 is a diagram for explaining the imaging range of the meeting device 60. As shown in FIG. 5(a), the meeting device 60 images a range of 360° in the horizontal direction. As shown in FIG. 5(b), the meeting device 60 sets the horizontal direction at the height of the meeting device 60 as 0°, and the imaging range is a predetermined angle above and below.

[0070] FIG. 6 is a diagram for explaining the extraction of the panoramic image and the speaker image. As shown in FIG. 6, since the image captured by the meeting device 60 forms a part 110 of a sphere, it has a three-dimensional shape. As shown in FIG. 5(b), the meeting device 60 performs perspective projection transformation by dividing the viewing angle for each predetermined angle above and below and each predetermined angle left and right. By performing the perspective projection transformation without gaps over the entire 360° in the horizontal direction, a predetermined number of planar images can be obtained. Therefore, a panoramic image 111 can be obtained by connecting the predetermined number of planar images left and right. In addition, the meeting device 60 performs face detection within a predetermined range centered on the voice direction from the panoramic image, and creates a speaker image 112 by cutting out 15° (30° in total) to the left and right from the center of the face.

[0071] <Regarding functions> Next, with reference to FIG. 7, the functional configuration of the recording information creation system 100 will be described. FIG. 7 is an example of a functional block diagram that separately explains the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the recording information creation system 100 in blocks.

[0072] <<Terminal device>> The information recording application 41 operating on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, an image combining unit 17, an audio data processing unit 18, a recording / playback unit 19, an upload unit 20, and an editing processing unit 21. Each of these units of the terminal device 10 is a function or means realized by any of the components shown in FIG. 3 operating according to instructions from the CPU 501 in accordance with the information recording application 41 expanded from the HD 504 to the RAM 503. Further, the terminal device 10 has a storage unit 1000 constructed by the HD 504 and the like shown in FIG. 3. An information storage unit 1001 is constructed in the storage unit 1000.

[0073] The communication unit 11 communicates various information with the information processing system 50 via a network. The communication unit 11, for example, receives a list of remote conferences from the information processing system 50 or transmits a recognition request for audio data to the information processing system 50.

[0074] The display control unit 13 displays various screens that serve as user interfaces in the information recording application 41 according to the screen transitions set in the information recording application 41. The operation reception unit 12 receives various operations on the information recording application 41.

[0075] The application screen acquisition unit 14 acquires screen information displayed by the application selected by the user, or screen information of the desktop screen, etc. from the OS (Operating System) or the like. When the application selected by the user is the remote conferencing application 42, the screen generated by the remote conferencing application 42 (an image of the terminal device user captured by the terminal device camera at each site, a display image of the shared material, an image including participant icons and participant names, etc.) can be obtained. The screen information (application screen) displayed by the application is information that a running application including a remote conferencing application or the like displays as a window and that the information recording application acquires as an image. The window of the application is drawn with the area of the window as an area in the entire desktop image and is displayed on a monitor or the like. The screen information displayed by the application can be acquired by other applications (information recording applications, etc.) as an image file or a video file composed of a plurality of consecutive images via the API of the OS (Operating System) or the API of the application to be displayed. Also, the screen information of the desktop screen is information composed of an image of the desktop screen generated by the OS, and can be similarly acquired as an image file or a video file via the API of the OS. The format of these image files may be bitmap, PNG, or other formats. Also, the format of the video file may be MP4 or other formats.

[0076] The voice acquisition unit 15 acquires voice output from a microphone or earphone of the terminal device 10 (including voice data received in a remote conference from the remote conference application 42). Even when the output voice is in a muted state, the voice acquisition unit 15 can acquire the voice. For voice data, the user does not need to perform operations such as selecting the remote conference application 42, and the voice acquisition unit 15 can acquire the voice that the terminal device 10 can output via the OS or the API (Application Interface) of the application. Thereby, the voice data received by the remote conference application 42 from another site 101 is also acquired. When the remote conference application 42 is not running or not in a remote conference, the information recording application 41 may not be able to acquire the voice data. Note that the voice acquired by the voice acquisition unit 15 may include only the voice data output and not the voice collected by the terminal device 10 because the meeting device 60 separately collects the voice of the site.

[0077] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 via Wi-Fi or Bluetooth (registered trademark), etc. The device communication unit 16 receives a panoramic image and a speaker image from the meeting device 60 and transmits the voice data acquired by the voice acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the voice data synthesized by the meeting device 60.

[0078] The image combining unit 17 combines the panoramic image and the speaker image received by the device communication unit 16, and the screen of the application acquired by the application screen acquisition unit 14 to create a combined image. Further, the image combining unit 17 connects the combined images created repeatedly in time series to create a combined image video, and combines the synthesized audio data with the combined image video to create a combined image video with audio. Note that the combination of the panoramic image and the speaker image may be performed by the meeting device 60. Also, videos composed of each image such as a panoramic image, a speaker image, an application screen, and an image composed of a panoramic image and a speaker image may be stored in the storage service system 70 as separate video files. In that case, the panoramic video, the speaker video, the application screen video, and the combined video of the panoramic image and the speaker image may be called up during browsing and displayed on one display screen.

[0079] The audio data processing unit 18 extracts the audio data combined with the combined image video, or requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

[0080] The recording and playback unit 19 plays back the combined image video. The combined image video is stored in the terminal device 10 during recording and then uploaded to the information processing system 50.

[0081] The upload unit 20 transmits the combined image video to the information processing system 50 when the remote meeting ends.

[0082] The editing processing unit 21 executes editing (partial deletion, splicing, etc.) of the combined image video according to the user's operation.

[0083] Figure 8 shows the video recording information stored in the information storage unit 1001. The video recording information has items such as a meeting ID, a recording ID, an update date and time, a title, an upload status, and a storage destination. When a user logs in to the information processing system 50, the information recording application 41 downloads the meeting information from the meeting information storage unit 5001 of the information processing system 50. The meeting ID and other information included in the meeting information are reflected in the video recording information. The video recording information in Figure 8 is held by the terminal device 10 operated by a certain user.

[0084] · The meeting ID is identification information for identifying a held remote meeting. The meeting ID is numbered when the schedule of the remote meeting is registered in the meeting management system 9, or the information processing system 50 numbers it upon request from the information recording application 41. The meeting management system 9 is a system for registering the schedule of meetings and remote meetings, the URL (meeting link) for starting the remote meeting, reservation information of devices used in the meeting, etc., and is a scheduler etc. connected from the terminal device 10 via the network. Also, the meeting management system 9 can send the registered schedule etc. to the information processing system 50.

[0085] · The recording ID is identification information for identifying the combined image video recorded in the remote meeting. The recording ID is numbered by the meeting device 60, but the information recording application 41 or the information processing system 50 may also number it. Different recording IDs being assigned to the same meeting ID indicates a case where the recording ended during the remote meeting but was restarted for some reason.

[0086] · The update date and time is the date and time when the combined image video was updated (recording ended). When the combined image video is edited, it is the edited date and time.

[0087] · The title is the meeting name of the meeting. It may be set when registering the meeting in the meeting management system 9, or the user may set it arbitrarily.

[0088] · Upload indicates whether the combined image video has been uploaded to the information processing system 50.

[0089] ·The storage destination indicates the location (URL or file path) where the combined image video and text data are stored in the storage service system 70. Therefore, the user can arbitrarily view the uploaded combined image video. Note that the combined image video and text data are stored with separate file names following, for example, the URL.

[0090] <<Meeting Device>> Returning to FIG. 7 for explanation. The meeting device 60 includes a terminal communication unit 61, a panorama image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, and a voice synthesis unit 65. Each of these units included in the meeting device 60 is a function or means realized by operating according to an instruction from the CPU 611, in which any of the components shown in FIG. 4 is expanded from the ROM 612 to the DRAM 614.

[0091] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may communicate with the terminal device 10 via a wireless LAN, Bluetooth (registered trademark), or the like.

[0092] The panorama image creation unit 62 creates a panorama image. The speaker image creation unit 63 creates a speaker image. These creation methods are described in FIGS. 5 and 6.

[0093] The sound collection unit 64 converts the sound signal acquired by the microphone included in the meeting device 60 into voice data (digital). Thereby, the content spoken by the user or participants at the base on the terminal device 10 side is collected.

[0094] The voice synthesis unit 65 synthesizes the voice transmitted from the terminal device 10 and the voice collected by the sound collection unit 64. Thereby, the voice spoken at another base 101 and the speech at the own base 102 are combined.

[0095] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a conference management unit 54, and a text conversion unit 55. Each of these units included in the information processing system 50 is a function or a means for realizing the operation of any of the components shown in FIG. 3 by the instruction from the CPU 501 according to the program expanded from the HD 504 to the RAM 503. Also, the information processing system 50 has a storage unit 5000 constructed by the HD 504 etc. shown in FIG. 3. In the storage unit 5000, a conference information storage unit 5001 and a recording information storage unit 5002 are constructed.

[0096] The communication unit 51 transmits and receives various information with the terminal device 10. For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 or receives a voice data recognition request from the terminal device 10.

[0097] The authentication unit 52 authenticates the user operating the terminal device 10. For example, the authentication unit 52 authenticates the user based on whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information held in advance. Note that the authentication information may be the card number of an IC card, biometric authentication information such as a face or fingerprint, etc. Also, the authentication unit 52 may authenticate using an external authentication system or an authentication method such as OAUTH.

[0098] The screen generation unit 53 generates the screen information to be displayed on the terminal device 10. When the terminal device 10 executes a native app, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML etc. When the terminal device 10 executes a web app, the screen information is created by HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc.

[0099] The conference management department 54 acquires information regarding remote conferences using the system accounts assigned to each user's account or information processing system 50 from the conference management system 9. The conference management department 54 can obtain a list of remote conferences to which the users belonging to the tenant have viewing rights. Since a conference ID is set for each remote conference, the remote conference and the recording information are associated with each other by the conference ID.

[0100] The text conversion unit 55 converts the voice data for which conversion to text data has been requested from the terminal device 10 into text data using an external voice recognition service. The text conversion unit 55 itself may perform the conversion.

[0101] FIG. 9 is an example of the conference information stored in the conference information storage unit 5001, which is managed by the conference management department 54. The conference management department 54 can obtain a list of remote conferences to which the said user belonging to the tenant has viewing rights using the above-mentioned account. Note that the viewing rights may be directly granted to the conference information managed by the conference management department 54 from the information recording application of the terminal device 10. In addition, the information of remote conferences to which the users belonging to the tenant have viewing rights includes the information of the conferences created by the users and the information of the conferences for which the users have been granted viewing rights by other users. In the present embodiment, remote conferences are taken as an example, but the list of remote conferences also includes conferences held in only one conference room.

[0102] The conference information is managed by the conference ID and is associated with participants, title (conference name), start date and time, end date and time, location, etc. These are examples of the conference information, and the conference information may include other information as well.

[0103] · The participants are the attendees of the conference.

[0104] · The title represents the content of the conference, such as the conference name or overseas topics.

[0105] · The start date and time is the scheduled date and time when the conference starts.

[0106] · The end date and time is the date and time when the meeting is scheduled to end.

[0107] · The location is the venue where the meeting is held, such as a meeting room, a branch office name, a building, etc.

[0108] As shown in FIGS. 8 and 9, the combined image video recorded in the meeting is specified by the meeting ID.

[0109] FIG. 10 shows the recording information stored in the recording information storage unit 5002. The recording information has a list of combined image videos recorded by all users belonging to the tenant. The recording information has items such as a meeting ID, a recording ID, an update date and time, a title, a storage destination, etc. These items may be the same as those in FIG. 8. The storage destination (path information such as the URL of the cloud storage system) may be input by the user in the user setting screen of the information recording application 41 of the terminal device 10 and stored in the recording information storage unit 5002.

[0110] <Screen transition> Subsequently, referring to FIGS. 11 to 20, several screens displayed by the terminal device 10 during a remote meeting will be described. FIG. 11 is the initial screen 200 after login displayed by the information recording application 41 operating on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs authentication information and succeeds in logging in, the initial screen 200 of FIG. 11 is displayed.

[0111] The initial screen 200 has a fixed display button 201, a front view change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker image 204 when not distinguished), and a recording start button 205. When the meeting device 60 has already been activated at the time of login and is imaging the surroundings, the panoramic image 203 and the speaker image 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether to start recording while viewing these. When the meeting device 60 is not activated (not imaging), the panoramic image 203 and the speaker image 204 are not displayed.

[0112] Note that the information recording application 41 may display the speaker images 204 of all participants based on all the faces detected from the panoramic image 203, or may display only the speaker images 204 of the N people who spoke most recently. Fig. 11 shows an example in which up to three speaker images 204 are displayed. Before the participants start speaking, there may be no speaker images 204 (increasing one by one in response to speech), or three speaker images 204 of the participants in a predetermined direction may be displayed (changing in response to speech).

[0113] Note that when no one is speaking, such as immediately after the meeting device 60 is activated, speaker images 204 are created in predetermined directions (0°, 120°, 240°, etc.) out of the horizontal 360°. When the fixed display described later is set, the fixed display setting takes precedence.

[0114] The fixed display button 201 is a button for the user to perform an operation of fixedly closing up a region with the panoramic image 203 as the speaker image 204.

[0115] FIG. 12 is a diagram for explaining an operation method when the fixed display button 201 is on. For example, the user moves the rectangular window 206 on the panorama image 203 with a pointing device such as a mouse or a touch panel. The user aligns the window 206 with a whiteboard, a stage, etc. included in the panorama image 203. The operation of the user is transmitted to the meeting device 60, and the meeting device creates an image of the range indicated by the window out of 360° in the horizontal direction in the same size as the speaker image 204 and transmits it to the terminal device 10. By doing so, the speaker image 204 can continuously display a whiteboard or the like other than the speaker.

[0116] Returning to FIG. 11, the front change button 202 is a button for the user to perform an operation of changing the front of the panorama image 203. The user can slide the panorama image 203 left and right with a pointing device to determine the participants shown in the front (since the panorama image shows 360° in the horizontal direction, the right end and the left end coincide as directions). The operation of the user is transmitted to the meeting device 60, and the meeting device changes the angle to be the front out of 360° in the horizontal direction to create a panorama image and transmits it to the terminal device 10.

[0117] In addition, when the meeting device 60 is not connected or the power is not turned on at the time of starting the information recording application 41, the device unrecognized screen 250 in FIG. 13 is displayed.

[0118] FIG. 13 shows an example of the device unrecognized screen 250. The device unrecognized screen 250 displays a message 251 saying "The device cannot be recognized. Please turn on the power of the device and connect it." The user can check the power and connection status of the meeting device 60 by looking at this.

[0119] When the user presses the recording start button 205, the information recording application 41 displays the recording setting screen 210 in FIG. 14.

[0120] FIG. 14 shows an example of a recording setting screen 210 displayed by the information recording app 41. On the recording setting screen 210, the user can set (include in recording) whether to record the panoramic image and speaker images created by the meeting device 60, and the desktop screen of the terminal device 10 or the screen of the app running on it. If neither the panoramic image and speaker images nor the desktop screen or the screen of the app running on it is recorded by the information recording app 41, only the audio (the audio output by the terminal device 10 + the audio collected by the meeting device 60) is recorded.

[0121] The camera toggle button 211 is a button for switching between on and off the recording of the panoramic image and speaker images created by the meeting device 60. The camera toggle button 211 may be able to set to record the panoramic image and the speaker image separately.

[0122] The PC screen toggle button 212 is a button for switching between on and off the recording of the desktop screen of the terminal device 10 and the screen of the app running on the terminal device 10. When the PC screen toggle button 212 is on, the desktop screen is recorded.

[0123] When the user wants to record the screen of an app, the app is further selected in the app selection bar 213. The app names of the apps running on the terminal device 10 are displayed in a pull-down format in the app selection bar 213. The user can select the app to record. This app name is obtained by the information recording app 41 from the OS. The information recording app 41 can only display apps with a UI (screen) among the running apps. The remote conferencing app 42 may be included in the selected apps. Therefore, the information recording app 41 can also record the materials displayed by the remote conferencing app 42 and the participants at each site in video. In addition, the apps displayed in the pull-down are various apps running on the terminal device, such as a presentation app, a word processing app, a spreadsheet app, a document creation and editing app for materials such as documents, a cloud electronic blackboard app, and a web browser app. Therefore, the user can flexibly select the app screens to be included in the combined image video.

[0124] Also, when recording on an app-by-app basis, the user can select multiple apps. The information recording app 41 can combine the screens of all the apps selected at the time of creating the combined image.

[0125] When both the camera toggle button 211 and the PC screen toggle button 212 are off, the message "Only audio will be recorded" is displayed in the recording content confirmation window 214. This audio is the audio output by the terminal device 10 (the audio received by the remote conferencing app 42 from another site 101) and the audio collected by the meeting device 60. That is, if a remote conference is in progress, the audio of the remote conferencing app 42 and the audio of the meeting device 60 are saved regardless of image recording. However, the user may selectively stop saving the audio of the remote conferencing app 42 and the audio of the meeting device 60 in the user settings.

[0126] Depending on the on / off combination of the camera toggle button 211 and the PC screen toggle button 212, the combined image video is recorded as follows. Also, the combined image is displayed in real time in the recording content confirmation window 214.

[0127] · When the camera toggle button 211 is on and the PC screen toggle button 212 is off, a panoramic image and a speaker image captured by the meeting device 60 are displayed in the recording content confirmation window 214.

[0128] · When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen is also selected), the desktop screen or the screen of the selected app is displayed in the recording content confirmation window 214.

[0129] · When the camera toggle button 211 is on and the PC screen toggle button 212 is on, a panoramic image and a speaker image captured by the meeting device 60, and the desktop screen or the screen of the selected app are displayed side by side in the recording content confirmation window 214.

[0130] Therefore, there may be cases where the panoramic image, the speaker image, and the app screen are not combined, or where the panoramic image, the speaker image, and the app screen are not recorded at all. In this embodiment, for convenience, the image created by the information recording app 41 is referred to as a combined image or a combined image video.

[0131] FIG. 15 is a display example of the recording content confirmation window 214 when the camera toggle button 211 is on and the PC screen toggle button 212 is off. In FIG. 15, the panoramic image 203 and the speaker image 204 are largely displayed.

[0132] FIG. 16 is a display example of the recording content confirmation window 214 when the camera toggle button 211 is on and the PC screen toggle button 212 is on. In FIG. 16, the panoramic image 203 and the speaker image 204 are displayed on the left side, and the app screen 217 is displayed on the right side.

[0133] Therefore, the user can confirm, before starting the recording, what kind of content (especially the image by the meeting device 60) the combined image video will be recorded with according to the settings for the recording setting screen 210 by the recording content confirmation window 214. The information recording app 41 accepts an instruction to start recording (pressing the recording start button) while displaying the image to be included in the combined image video of the recorded information. Also, the information recording app 41 can obtain, until an instruction to end the recording (pressing the recording end button), the screen information displayed in the selected application (for example, a remote conferencing application) at the time of the recording start instruction and the image information around the device obtained from the device at the time of the recording start instruction, and use both to create a combined image.

[0134] Note that FIG. 16 is a display example of the combined image video when only one app is selected. When two or more apps are selected, the screens of the second and subsequent apps are sequentially connected to the right side. Alternatively, the screens of the second and subsequent apps may be arranged in a two-dimensional vertical and horizontal manner.

[0135] Returning to FIG. 14 for explanation. The recording settings screen 210 has a checkbox 215 together with a message saying "Automatically perform character recognition after uploading the recording". Also, the recording settings screen 210 has an immediate recording start button 216. When the user checks the checkbox 215, text data obtained by converting the speech during the remote conference is attached to the combined image video. In this case, after the recording ends, the information recording app 41 uploads the audio together with a conversion request for the text data to the information processing system 50. Also, when the user presses the immediate recording start button 216, the in-recording screen 220 of FIG. 17 is displayed.

[0136] FIG. 17 is an example of the in-recording screen 220 displayed by the information recording app 41 during recording. Note that in the description of FIG. 17, the differences from FIG. 14 will mainly be explained. The in-recording screen 220 is a screen during the recording process that displays the combined image video being recorded in real time under the conditions set by the user on the recording settings screen 210. The in-recording screen 220 can be displayed while running the remote conference application. The in-recording screen 220 of FIG. 17 shows the case where the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays the panoramic image 203 and the speaker image 204 (both are videos) created by the meeting device 60. The in-recording screen 220 displays an in-recording icon 225, a pause button 226, and a recording end button 227.

[0137] Note that when the user sets the PC screen toggle button 212 to on, as shown in FIG. 16, the desktop screen and the app screens are displayed adjacent to the panoramic image and the speaker image on the in-recording screen 220.

[0138] The pause button 226 is a button for stopping recording and will also accept resuming recording after stopping. The recording end button 227 is a button for ending recording. With the pause button 226, the recording ID does not change, while with the recording end button 227, the recording ID changes. When pausing and resuming recording, the user can also reset the recording conditions set on the recording settings screen 210. In that case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, when the recording end button 227 is pressed), or may combine multiple files so as to be continuous as one video (for example, when the pause button 226 is pressed). Also, when the information recording application 41 plays back the combined image video, the multiple recording files may be played back continuously as one video.

[0139] Also, the recording screen 220 during recording has an information acquisition button 221 from the calendar, a meeting name field 222, a time field 223, and a location field 224. The information acquisition button 221 from the calendar is a button for the user to acquire meeting information from the meeting management system 9. When the information acquisition button 221 from the calendar is pressed, the information recording application 41 acquires and displays a list of meetings for which the user has viewing permission from the information processing system 50. The user selects a remote meeting to be held from the list of meetings. Thereby, the meeting information is reflected in the meeting name field 222, the time field 223, and the location field 224. The title of the meeting information is reflected in the meeting name field 222, the start time and end time are reflected in the time field 223, and the location is reflected in the location field 224. Also, the meeting information and the recording information in the meeting management system are associated with each other by the meeting ID.

[0140] When the remote meeting ends and the user ends the recording, a combined image video with audio is created.

[0141] FIG. 18 is an example of a meeting list screen 230 displayed by the information recording application 41. The meeting list screen 230 is a list of meetings, but it can display a list of recording information recorded in a remote meeting. Also, regardless of whether it is a remote meeting, meetings held only in a certain conference room are included. The meeting list screen 230 is displayed by integrating the meeting information that the logged-in user has viewing permission for in the meeting information storage unit 5001 and the information stored in the recording information storage unit 5002 associated with this remote meeting. The information of the video recording information stored in the information storage unit 1001 may be further integrated.

[0142] The meeting list screen 230 is displayed when the user selects the meeting list tab 231 on the initial screen 200 in FIG. 14. The meeting list screen 230 displays a list 236 of recording information that this user has viewing permission for. The meeting creator (proceedings creator) can set viewing permissions for participants. Note that the meeting list may be a list of stored recording information or a list of meeting schedules and meeting data.

[0143] The meeting list screen 230 has items such as a checkbox 232, an update date and time 233, a title 234, and a status 235.

[0144] · The checkbox 232 accepts the selection of recording files. The checkbox 232 is used when the user wants to delete recording files collectively.

[0145] · The update date and time 233 indicates the start time and end time of the recording of the combined image video. If it is edited, the edit date and time may be used.

[0146] · The title 234 is the title of the meeting (such as the topic). It may be transferred from the meeting information or set by the user.

[0147] · Status 235 indicates whether the combined image video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, an upload button is displayed. If there is an un-uploaded combined image video, when the user logs in to the information processing system 50, the information recording app 41 may automatically upload it.

[0148] When the user selects an arbitrary title or the like from the combined image video list 236 using a pointing device, the information recording app 41 displays the recording / playback screen 240 in FIG. 19. On the recording / playback screen 240, playback of the combined image video and the like is possible.

[0149] FIG. 19 is an example of the recording / playback screen 240 after selection of a combined image video, which is displayed by the information recording app 41. The recording / playback screen 240 has a playback image display column 241, a speech-to-text button 242, one or more text display columns 243, an auto-scroll button 244, and a search button 245.

[0150] · The playback image display column 241 has a play button 241a, a rewind button 241b, a fast-forward button 241c, a time indicator 241d, a playback speed button 241e, a volume button 241f, etc. The playback image display column 241 plays and displays the combined image video. In FIG. 19, in the combined image video of the playback image display column 241, a panoramic image and a speaker image are arranged on the left side, and the screen of the remote conferencing app 42 is displayed on the right side. The screen of the remote conferencing app 42 becomes an image of the base or an image of materials during a remote conference. Therefore, the user can operate various buttons to view the screen of a desired scene.

[0151] · When the audio data of the combined image video being displayed in the playback image display column 241 has been converted to text data, the speech content is displayed as text in the text display column 243.

[0152] · The text conversion button 242 is a button for the user to switch whether to display the text data shown in the text display field 243 in synchronization with the playback time of the combined image video.

[0153] · The automatic scroll button 244 is a button for the user to switch whether to automatically scroll the text data regardless of the playback time.

[0154] · The search button 245 is a button for the user to search for text data by specifying a keyword.

[0155] Also, in the recording playback screen 240, it may be possible to download the combined image video.

[0156] Figure 20 is an example of an editing screen 260 for the combined image video. The editing screen 260 automatically transitions from the recording screen 220 or when the user performs a predetermined operation on the recording playback screen 240. The editing screen 260 has a first display field 261 and a second display field 262. In the first display field 261, a combined image at a certain moment during playback is displayed, and in the second display field 262, the frames constituting the combined image video are displayed in time series. The user can select one or more frames and delete unnecessary frames. Also, the user can extract a part of a frame and insert it after an arbitrary frame. The editing processing unit 21 edits the combined image video according to the user's operation and overwrites or separately saves the edited combined image video.

[0157] <Operation or processing procedure> Subsequently, based on the above configuration, the operations and processes performed by the recording information creation system 100 will be described.

[0158] <<Login>> Figure 21 is an example of a sequence diagram explaining the procedure or process for the user to operate the information recording application 41 operating on the terminal device 10 and log in to the information processing system 50.

[0159] S1: The user inputs an operation to start the information recording application 41 on the terminal device 10.

[0160] S2: In response to the operation, the terminal device 10 starts the information recording application 41.

[0161] S3: When the information recording application 41 starts, the communication unit 11 automatically communicates with the information processing system 50 and requests a login screen.

[0162] S4: The communication unit 51 of the information processing system 50 receives the request for the login screen and transmits the screen information of the login screen generated by the screen generation unit 53 to the information recording application 41.

[0163] S5: The communication unit 11 of the information recording application 41 receives the screen information of the login screen, and the display control unit 13 displays the login screen.

[0164] S6: The user inputs authentication information for logging in to the tenant in the information recording application 41. The operation reception unit 12 of the information recording application 41 receives the input.

[0165] S7: The communication unit 11 of the information recording application 41 transmits a login request to the information processing system 50 by specifying the authentication information.

[0166] S8: The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 52 authenticates the user based on the authentication information. Here, it is assumed that the authentication is successful.

[0167] S9: The communication unit 51 of the information processing system 50 transmits the login success to the information recording application 41.

[0168] S10: The communication unit 11 of the information recording application 41 receives the login success, and the display control unit 13 displays the initial screen 200.

[0169] <<Saving of combined image and video>> Next, referring to FIG. 22, the storage process of the combined image video will be described. FIG. 22 is an example of a sequence diagram showing the procedure in which the information recording application 41 records a panoramic image, a speaker image, and the screen of the application.

[0170] S21: The user operates the remote conferencing application 42 to start a remote conference. Here, it is assumed that the remote conferencing applications 42 at the home base 102 and the other base 101 have started a remote conference. The remote conferencing application 42 at the home base 102 transmits the image captured by the camera of the terminal device 10 and the audio collected by the microphone to the remote conferencing application 42 at the other base 101. The remote conferencing application 42 at the other base 101 displays the received image on the display and outputs the received audio from the speaker. Similarly, the remote conferencing application 42 at the other base 101 transmits the image captured by the camera of the terminal device 10 and the audio collected by the microphone to the remote conferencing application 42 at the home base 102. The remote conferencing application 42 at the home base 102 displays the received image on the display and outputs the received audio from the speaker. Each remote conferencing application 42 repeats this to realize a remote conference.

[0171] S22: The user makes settings related to recording for the recording setting screen 210 of the information recording application 41 shown in FIG. 14. The operation reception unit 12 of the information recording application 41 receives the settings. Here, it is assumed that both the camera toggle button 211 and the PC screen toggle button 212 are on.

[0172] If the user has previously reserved a remote conference, by pressing the information acquisition button 221 from the calendar in FIG. 19, a list of remote conferences can be displayed, and the remote conference to which the recorded video is to be associated can be selected. Since the user has logged in to the information processing system 50, the information processing system 50 identifies the remote conferences to which the logged-in user has viewing privileges. Since the information processing system 50 transmits the list of the identified remote conferences to the terminal device 10, the user selects a remote conference that is in progress or will be held in the future. Thereby, information regarding the remote conference, such as the conference ID, is determined.

[0173] In addition, even if the user has not reserved a remote meeting in advance, the user can create a meeting when creating a combined image video. Hereinafter, a case will be described in which the information recording app 41 creates a meeting when creating a combined image video and acquires a meeting ID from the information processing system 50.

[0174] S23: The user instructs the information recording app 41 to start recording (the "Record Now" button 216). The operation reception unit 12 of the information recording app 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0175] S24: Since no remote meeting is selected (because the meeting ID has not been determined), the communication unit 11 of the information recording app 41 sends a remote meeting creation request to the information processing system 50.

[0176] S25: The communication unit 51 of the information processing system 50 receives the remote meeting creation request. The meeting management unit 54 acquires a non-duplicate meeting ID numbered by the meeting management system 9, and the communication unit 51 sends the meeting ID to the information recording app 41.

[0177] S26: Also, the meeting management unit 54 sends the storage destination of the combined image video (the URL of the storage service system 70) to the information recording app 41 via the communication unit 51.

[0178] S27: When the communication unit 11 of the information recording app 41 receives the meeting ID and the storage destination of the recording file, the image combining unit 17 determines that the recording is ready and starts recording.

[0179] S28: The app screen acquisition unit 14 of the information recording app 41 requests the app for the screen of the app selected by the user (the app screen acquisition unit 14 acquires the screen of the app via the OS in more detail). In FIG. 22, the app selected by the user is the remote meeting app 42.

[0180] S29: The image combining unit 17 of the information recording application 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. At the time of notification, the image combining unit 17 also notifies that the camera toggle button 211 is on (request for panoramic image and speaker image).

[0181] S30: When the terminal communication unit 61 of the meeting device 60 receives the start of recording, it assigns a non-duplicate recording ID and returns the recording ID to the information recording application 41. Note that the recording ID may be assigned by the information recording application 41 or obtained from the information processing system 50.

[0182] S31: The voice acquisition unit 15 of the information recording application 41 acquires the voice data output by the terminal device 10 (voice data received by the remote conferencing application 42).

[0183] S32: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and the synthesis request to the meeting device 60.

[0184] S33: The terminal communication unit 61 of the meeting device 60 receives the voice data and the synthesis request, and the voice synthesis unit 65 synthesizes the ambient voice data collected by the sound collection unit 64 and the received voice data. For example, the voice synthesis unit 65 adds the two voice data together. Since the clear voice around the meeting device 60 is recorded, the text conversion accuracy of the voice around the meeting device 60 (on the conference room side) is particularly improved.

[0185] This voice synthesis is also possible on the terminal device 10. However, by dispersing the recording function to the terminal device 10 and the voice processing to the meeting device 60, the loads on the terminal device 10 and the meeting device 60 can be reduced. The recording function may be dispersed to the meeting device 60 and the voice processing to the terminal device 10.

[0186] S34: Also, since the meeting device 60 has received that the camera toggle button 211 is on, the panorama image creation unit 62 creates a panorama image, and the speaker image creation unit 63 creates a speaker image.

[0187] S35: The device communication unit 16 of the information recording application 41 repeatedly acquires the panorama image and the speaker image from the meeting device 60. Also, the device communication unit 16 repeatedly requests and acquires the synthesized voice data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 requesting the meeting device 60. Alternatively, the meeting device 60 that has received that the camera toggle button 211 is on may automatically transmit the panorama image and the speaker image. The meeting device 60 that has received the request for synthesizing voice data may automatically transmit the synthesized voice data to the information recording application 41.

[0188] S36: The image combining unit 17 of the information recording application 41 creates a combined image by arranging the screen of the application acquired from the remote conferencing application 42, the panorama image, and the speaker image. The image combining unit 17 repeatedly creates combined images, and creates a combined image video by designating each combined image for the frames constituting the video. Also, the image combining unit 17 stores the voice data received from the meeting device 60.

[0189] The information recording application 41 repeats the above steps S31 to S36.

[0190] S37: When the remote conference ends and there is no longer a need for recording, the user instructs the information recording application 41 to end the recording (for example, the recording end button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0191] S38: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 of the end of recording. As a result, the meeting device 60 ends the creation of the panorama image and the speaker image and the synthesis of the voice.

[0192] S39: The image combining section 17 of the information recording application 41 combines the audio data with the combined image video to create a combined image video with audio.

[0193] S40: Also, when the user checks the checkbox 215 associated with "Automatically perform character recognition after uploading the recording" on the recording settings screen 210, the audio data processing section 18 requests the information processing system 50 to convert the audio data into text data. Specifically, the audio data processing section 18 specifies the URL of the storage destination via the communication section 11 and transmits a conversion request for the audio data combined with the combined image video to the information processing system 50 together with the conference ID and the recording ID.

[0194] S41: The communication section 51 of the information processing system 50 receives the conversion request for the audio data, and the text conversion section 55 uses the speech recognition service system 80 to convert the audio data into text data. The communication section 51 stores the text data in the same storage destination as the storage destination of the combined image video (the URL of the storage service system 70). Note that in the recording information storage section 5002, the text data is associated with the combined image video by the conference ID and the recording ID. Note that the text data may be managed by the conference management section 54 of the information processing system 50 and stored in the storage section 5000. Also, the terminal device 10 may request speech recognition from the speech recognition service system 80 and store the text data obtained from the speech recognition service system 80 in the storage destination. Note that the speech recognition service system 80 returns the converted text data to the information processing system 50, but may also directly transmit it to the URL of the storage destination. The speech recognition service system 80 may select or switch from a plurality of services according to the setting information set by the user in the information processing system 50.

[0195] S42: Also, the upload section 20 of the information recording application 41 stores the combined image video in the storage destination of the combined image video via the communication section 11. In the recording information storage section 5002, the combined image video is associated with the conference ID and the recording ID. "Uploaded" is recorded in the combined image video.

[0196] Since the storage destination is notified to the user, the user can share the combined image video with the participants by informing the storage destination via email or the like. Even if the devices for creating the combined image video, audio data, and text data are different, they can be collected and stored in one storage location, and can be easily viewed by the user or others later.

[0197] In the case of real-time speech recognition, the meeting device 60 or the terminal device 10 transmits the speech data to the information processing system 50 in real time. The terminal device 10 displays and stores the text data transmitted from the meeting device 60 or returned from the information processing system 50 on the recording screen 220.

[0198] Note that the processes in steps S31 to S36 do not have to be in the order shown in FIG. 22, and the synthesis of audio data and the creation of combined images may be in either order.

[0199] <<Settings Change during Creation of Combined Image Video>> Subsequently, with reference to FIG. 23, a case will be described in which the user changes the settings for the recording settings screen 210 while the information recording application 41 is creating a combined image video. FIG. 23 is an example of a sequence diagram showing the procedure in which the information recording application 41 records a panoramic image, a speaker image, and the screen of the application with the settings after the change when the settings for the recording settings screen 210 are changed during the creation of the combined image video. In the description of FIG. 23, mainly the differences from FIG. 22 will be described.

[0200] In FIG. 23, during the repetition of steps S31 to S36, the user changes the settings of the recording settings screen 210 at step S51. If the camera toggle button 211 and the PC screen toggle button 212 were on before the setting change, for example, the following changes are possible. The image combining unit 17 generates a combined image video based on the changed settings.

[0201] a. Turn off the camera toggle button 211 and the PC screen toggle button 212 b. Turn off the camera toggle button 211 and turn on the PC screen toggle button 212 c. The PC screen toggle button 212 is on, but the app for saving the screen has been changed. In FIG. 23, the case of b. will be described. S52: Since the camera toggle button 211 has been changed to off, the image combining unit 17 of the information recording app 41 notifies the meeting device 60 of the end of recording via the device communication unit 16. For this reason, in FIG. 23, there is no creation process for the panoramic image and the speaker image in step S34. Also, in step S35, only the audio data is transmitted from the meeting device 60 to the information recording app 41.

[0202] Also, in step S36, the image combining unit 17 creates a combined image only from the screen of the app acquired from the remote conferencing app 42. In this case, since the panoramic image and the speaker image that existed when the camera toggle button 211 was on disappear, the image combining unit 17 may arrange only the app screen large, or may arrange it in the same way as when there were the panoramic image and the speaker image. Also, the image combining unit 17 may save the combined image video as one recording file, or may obtain a recording ID from the meeting device 60 for one setting change and save it as another recording file.

[0203] In the case of a., step S28 becomes unnecessary, and in step S36, the app screen is not displayed either.

[0204] In the case of c., the app screen acquired by the information recording app 41 in step S28 is changed.

[0205] In this way, even if the recording settings are changed during the creation of the combined image video, the information recording app 41 can create a combined image video according to the changed content. By switching the app to be recorded during recording by the user, the screen of the app displayed during the remote conference can be included in the combined image video.

[0206] <<Save the screens of all running apps>> Next, referring to FIG. 24, a case where the screen of all running apps is saved and the information recording app 41 creates a combined image video after the remote meeting ends will be described. FIG. 24 is an example of a sequence diagram showing the procedure of saving the screens of all running apps and changing the images (videos) to be included in the recorded information after the remote meeting ends. In the description of FIG. 24, the differences from FIG. 22 will be mainly described.

[0207] In FIG. 24, at step S28, the app screen acquisition unit 14 of the information recording app 41 acquires the screens of all running apps via the OS or the like. The screen of each app is saved as a video by the terminal device 10 in association with the identification information of the app.

[0208] Also, at step S36, the image combining unit 17 combines the app screen, the panorama image, and the speaker image according to the recording settings, and the display control unit 13 displays the combined image.

[0209] Then, when the recording ends at step S37, at step S39, the image combining unit 17 combines the screens of all apps selected on the recording settings screen 210, the panorama image, and the speaker image. If the camera toggle button 211 is off, the panorama image and the speaker image are not combined.

[0210] In this way, if the screens of all running apps are saved, even if the user changes the recording settings screen 210 during recording, the app screens from the start of the remote meeting can be included in the combined image video. The user can make recording settings after the remote meeting ends, and by recording all app screens, the app screens to be included in the recorded information can be determined later.

[0211] <Communication Disconnection between the Terminal Device and the Meeting Device during Recording> Next, referring to FIG. 25, a case where the communication between the terminal device 10 and the meeting device 60 is disconnected during recording will be described. FIG. 25 is an example of a sequence diagram showing a procedure for appropriately ending recording when the communication between the terminal device 10 and the meeting device 60 is disconnected during recording. In the description of FIG. 25, the differences from FIG. 22 will mainly be described.

[0212] S61: The user accidentally unplugged the USB cable connecting the terminal device 10 and the meeting device 60. Other examples of communication disconnection include an abnormality of the wireless LAN router, powering off of the meeting device 60, etc.

[0213] S62: The device communication unit 16 of the information recording application 41 detects that the USB cable has been unplugged, for example, because the external device connection I / F does not detect a voltage. The device communication unit 16 may also detect a communication interruption due to no response from the meeting device 60, etc.

[0214] S63: When communication with the meeting device 60 is not possible, even if the camera toggle button 211 is off, the information recording application 41 cannot acquire the synthesized audio data. Therefore, the information recording application 41 ends the recording. The subsequent processing may be the same as that after step S37 in FIG. 22.

[0215] <When each application exists in the cloud and the meeting device is connected to the cloud application> Next, referring to FIG. 26, the system configuration when each application exists in the cloud and the meeting device 60 communicates with the cloud-side application will be described. FIG. 26 shows a configuration example of the recording information creation system 100. The configuration and communication connection relationship for the information recording application 41 to acquire the image around the meeting device 60 and the screen displayed by the remote conferencing application 42 may be the following variations.

[0216] FIG. 26(a) is a configuration example when the remote conferencing application 42 and the information recording application 41 operate on different terminal devices. In this case, the surrounding image information captured by the meeting device 60 is transmitted from the meeting device 60 installed at the base to the information recording application 41.

[0217] Also, the screen information displayed by the remote conferencing application 42 is also acquired by the information recording application 41, and recording information is created using the image information around the meeting device 60 and the screen information displayed by the remote conferencing application 42.

[0218] FIG. 26(b) has the same configuration as FIG. 2. The information recording application 41 executed on the terminal device 10 acquires the screen information displayed by the remote conferencing application 42 executed on the terminal device 10, the audio of the remote conference, the image information around the meeting device 60, and the audio of the base, and creates recording information. The terminal device 10 and the meeting device 60 are directly connected locally (USB, BLE, Wi-Fi).

[0219] FIG. 26(c) shows a configuration example in which the information recording application 41 operates in the information processing system 50. The information processing system 50 (on the cloud) executes the information recording application 41, and acquires the surrounding image information from the meeting device 60 at the base and the screen information of the remote conferencing application 42 from the terminal device 10, respectively. Therefore, the recording information is created by a Web application that executes the information recording application 41 on the information processing system 50 and the terminal device 10.

[0220] The information recording application 41 of the information processing system 50 performs the main processing for creating recording information, and the Web browser application 120 of the terminal device 10 performs the processing related to UI display and input, etc. The information recording application 41 on the information processing system 50 side, the meeting device 60, and the terminal device 10 are connected via the Web (Internet).

[0221] The information recording app 41 on the information processing system 50 side acquires the screen information displayed by the remote conferencing app 42 executed on the terminal device 10 and the audio of the remote conference via the Web (Internet). Also, the information recording app 41 on the information processing system 50 side acquires the image information around the meeting device 60 and the audio at the base via the Web and creates recording information. Other processes have been described in this embodiment.

[0222] FIG. 26(d) shows a configuration example in which the remote conferencing service system 90 and the information processing system 50 communicate. The information processing system 50 includes the information recording app 41, and acquires the surrounding image information from the meeting device 60 at the base and the screen information of the remote conferencing app 42 from another cloud service (remote conferencing service system). The information recording app 41 on the information processing system 50 side, the meeting device 60, and the cloud service side of the app (remote conferencing app 42) are connected via the Web (Internet). The information recording app 41 on the information processing system 50 side acquires the screen information and audio displayed by the remote conferencing app 42 from the cloud service side of the remote conferencing app 42. Other aspects are the same as in FIG. 26(c).

[0223] <Main effects> As described above, in the recording information creation system 100 of this embodiment, in the combined image video, panoramic images and speaker images of the surroundings including the user are displayed, and further, the screens of apps displayed during the remote conference, such as the remote conferencing app 42, are displayed. When participants or non-participants in the remote conference view the combined image video as the minutes of the meeting, the situation during the remote conference is reproduced with a sense of presence. Also, since the information recording app 41 records both the screen information displayed by the application (such as the remote conferencing app) selected by the information recording app 41 and the image information around the devices inside the base (such as in the conference room), even if the screen displayed by the remote conferencing app 42 changes, it is possible to create recording information that accurately records the situation at the base (such as in the conference room) together with the content of the remote conference (remote communication).

[0224] <Other application examples> As described above, the best mode for carrying out the present invention has been described using examples. However, the present invention is not limited to such examples, and various modifications and substitutions can be made without departing from the gist of the present invention.

[0225] For example, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. Further, the meeting device 60 may be one in which an omnidirectional camera, a microphone, and a speaker are connected by a cable.

[0226] Also, the meeting device 60 may be arranged at another site 101. The other site 101 separately creates combined image videos and text data using the meeting device 60. Also, a plurality of meeting devices 60 may be arranged at one site. In this case, a plurality of pieces of recording information are created for each meeting device 60.

[0227] Also, the arrangement of the panoramic image 203, the speaker image 204, and the app screen in the combined image video used in the present embodiment is merely an example. The panoramic image 203 may be below and the speaker image 204 may be above, or the user may change the arrangement, or the display and non-display of the panoramic image 203 and the speaker image 204 may be individually switched during playback.

[0228] Also, the configuration example in FIG. 7 and the like is divided according to the main functions in order to facilitate the understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The present invention is not limited by the way of dividing the processing units and their names. The processing of the terminal device 10, the meeting device 60, and the information processing system 50 can be further divided into more processing units according to the processing content. Also, it can be divided so that one processing unit includes more processing.

[0229] Moreover, the device group described in the embodiments merely represents one of the multiple computing environments for implementing the embodiments disclosed in this specification. In one embodiment, the information processing system 50 includes multiple computing devices such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link including a network or a shared memory, and implement the processing disclosed in this specification.

[0230] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as FIG. 22, etc., in various combinations. For example, the process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be integrated into one server device or divided into multiple devices.

[0231] Each function of the embodiments described above can be realized by one or more processing circuits. Here, the "processing circuit" in this specification includes a processor programmed to execute each function by software like a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array) designed to execute each function described above, and devices such as conventional circuit modules.

Description of Reference Numerals

[0232] 10 Terminal device 50 Information processing system 60 Meeting device 100 Recording information creation system

Prior Art Documents

Patent Documents

[0233] [Patent Document 1] Japanese Patent Application Laid-Open No. 2020-115609

Claims

1. A recording information creation system including an information recording application to be executed on a terminal device, wherein the information recording application creates recording information based on screen information displayed by a remote conferencing application selected in the information recording application and image information of the surroundings of the device acquired by the device. A recording information creation system characterized by this.

2. The recording information creation system according to claim 1, wherein the screen information is information that is displayed as a window by a running application including the remote conferencing application and acquired as an image by the information recording application.

3. The recording information creation system according to claim 1 or 2, wherein the image information of the surroundings of the device is image information acquired by the device imaging a 360° area surrounding the device.

4. The recording information creation system according to any one of claims 1 to 3, wherein the image information of the surroundings of the device is transmitted from the device installed at the base to the information recording application.

5. The information recording application, has an image combining unit that creates a combined image in which the screen information and the surrounding image information are arranged on one screen, the image combining unit creates a combined image video composed of a plurality of the combined images, The recording information creation system according to any one of claims 1 to 4, characterized in that the recording information includes the combined image video.

6. The recording information creation system according to claim 5, wherein the image combining unit further creates the combined image by linking one or more speaker images cut out from the surrounding image information of the speaker to the surrounding image information.

7. a voice acquisition unit that acquires first voice data output by the terminal device, a voice synthesis unit that synthesizes the first voice data acquired by the voice acquisition unit and second voice data collected by the device, and The recording information creation system according to claim 5 or 6, characterized in that the synthesized voice data is combined with the combined image video.

8. The recording information creation system according to claim 7, wherein the image combining unit and the voice synthesis unit are distributed and arranged in the device and the information recording application of the terminal device.

9. an upload unit that stores the combined image video in a storage service system via a network A voice data processing unit that stores text data converted from the synthesized voice data in the storage service system in association with the combined image video, The recording information creation system according to claim 7 or 8, wherein the recording information includes the combined image video and the text data.

10. The information processing system further includes a storage unit that stores settings by a user, The setting includes a storage destination of the recording information, According to the set storage destination, the text data and the combined image video are stored in the storage service system. The recording information creation system according to claim 9.

11. The information recording application has an operation reception unit that receives a setting as to whether to include each of the surrounding image information and the screen information displayed by the application being executed in the combined image, The recording information creation system according to any one of claims 7 to 9, wherein the image combining unit creates the combined image video including the surrounding image information and the screen information, the combined image video including one of the surrounding image information and the screen information, or the synthesized voice data not including either the surrounding image information or the screen information according to the setting received by the operation reception unit.

12. When the operation reception unit receives a setting to include the screen information displayed by the application being executed in the combined image, the operation reception unit further receives a selection of an application to be included in the combined image from among a plurality of applications including the remote conference application, The recording information creation system according to claim 11, wherein the image combining unit includes the screen information of the application selected by the operation reception unit in the combined image video.

13. The remote conference application is an application that performs remote communication with another terminal device. The recording information creation system according to any one of claims 1 to 12.

14. The information recording application has a display control unit that displays an image to be included in the combined image selected by the operation reception unit, The recording information creation system according to claim 11 or 12, wherein the display control unit displays a combined image video using the image selected by the operation reception unit before the start of recording.

15. The information recording application: Receives a recording start instruction while displaying the image to be included in the combined image, The screen information to be displayed by the selected remote conference application, which is the screen information displayed when the recording start instruction is given, The image information around the device acquired from the device, which is the image information acquired since the recording start instruction is given, Acquires until an instruction to end the recording, and uses both to create the combined image, the recording information creation system according to claim 14.

16. When receiving the start of recording of the combined image video, The recording information creation system according to any one of claims 5 to 12, wherein the combined image video created by the image combining unit is displayed while the remote conference application is being executed during the recording process.

17. After starting the creation of the combined image, when the operation receiving unit receives a change in the setting of whether to include in the combined image for each of the surrounding image information and the screen information displayed by the application being executed, The image combining unit creates the combined image video according to the changed setting received by the operation receiving unit, the recording information creation system according to claim 11 or 12.

18. Having an application screen acquisition unit that stores the screen information of all applications being executed in the terminal device, When the creation of the combined image is completed, the image combining unit creates the combined image video according to the setting received by the operation receiving unit, the recording information creation system according to claim 11 or 12.

19. The display control unit displays an editing screen for receiving the editing of the combined image video that has ended recording, The operation receiving unit receives the editing of the combined image video, the recording information creation system according to claim 14.

20. A recording information creation method performed by a recording information creation system including an information recording application executed on a terminal device, The information recording application creates recording information based on the screen information displayed by the remote conference application selected in the information recording application and the image information around the device acquired by the device, the recording information creation method.

21. A program executed on a terminal device, A program characterized by causing the terminal device to create recording information based on screen information displayed by a remote conferencing application selected by the program and image information around the device acquired by the device.

Images