Information processing device, information processing method, and information processing system

Abstract

To generate an image in which a plurality of objects are displayed properly.SOLUTION: An apparatus system comprises: an imaging apparatus which captures a wide-angle image; an image generation unit which generates a first image from the wide-angle image; and a program which operates in a terminal device capable of communication with the imaging apparatus. When a plurality of objects are detected from the wide-angle image, the image generation unit generates the first image including the objects. The program causes the terminal device to function as a display control unit which displays the first image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an equipment system, an imaging device, and a display method. [Background technology]

[0002] Remote communication systems are known in which images and audio are transmitted in real time from one location to one or more other locations, allowing users in remote locations to hold conferences using images and audio. Remote communication may involve the use of devices such as electronic whiteboards.

[0003] There is known a technique for extracting a speaker from an image of a participant at a location who is participating in a conference (see, for example, Patent Document 1). Patent Document 1 discloses a system that uses face recognition to display a close-up of a speaker from a spherical image. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, when there are multiple objects to be included in an image, one of the objects may not be displayed appropriately.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide an equipment system that generates an image in which multiple objects are displayed appropriately. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides an equipment system having an imaging device that captures a wide-angle image, an image generation unit that generates a first image from the wide-angle image, and a program that runs on a terminal device that can communicate with the imaging device, wherein when multiple objects are detected from the wide-angle image, the image generation unit generates the first image that includes the multiple objects, and the program causes the terminal device to function as a display control unit that displays the first image. [Effects of the Invention]

[0007] It is possible to provide a technique for generating an image in which multiple objects are displayed appropriately. [Brief explanation of the drawings]

[0008] [Figure 1] 10A and 10B are diagrams illustrating an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. [Figure 2] FIG. 10 is a diagram illustrating an example of a generated panoramic image. [Figure 3] FIG. 1 is a diagram illustrating an example of the configuration of a record information creation system. [Figure 4] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing system and a terminal device. [Figure 5] FIG. 2 is a diagram illustrating an example of the hardware configuration of a meeting device. [Figure 6] FIG. 10 is a diagram illustrating the imaging range of the meeting device. [Figure 7] 10A and 10B are diagrams illustrating the cutting out of a panoramic image and a speaker image. [Figure 8] FIG. 2 is a diagram illustrating a hardware configuration of an example of an electronic whiteboard. [Figure 9] 1 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in a record information creation system, divided into blocks. [Figure 10] 4 is a diagram showing an example of moving image recording information stored in an information storage unit. FIG. [Figure 11] FIG. 10 is a diagram illustrating an example of conference information managed by a communication management unit. [Figure 12] 10 is a diagram showing an example of association information stored in an association information storage unit, in which conference IDs are associated with device identification information; FIG. [Figure 13] FIG. 4 is a diagram illustrating an example of account information stored in an account information storage unit. [Figure 14] FIG. 1 is an example of a functional block diagram illustrating functions of an electronic whiteboard divided into blocks. [Figure 15] 4 is a diagram showing an example of device identification information stored in a device information storage unit; FIG. [Figure 16] FIG. 2 is a diagram illustrating object information stored in an object information storage unit. [Figure 17] FIG. 10 is a diagram showing an example of an initial screen displayed by an information recording application running on the terminal device after login. [Figure 18] FIG. 10 is a diagram showing an example of a recording setting screen displayed by an information recording application. [Figure 19] FIG. 10 is a diagram showing an example of a recording screen displayed by the information recording application during recording. [Figure 20] FIG. 10 is a diagram illustrating an example of a conference list screen displayed by the information recording application. [Figure 21] 10 is an example of a sequence diagram illustrating a process of creating a panoramic image by a meeting device from the start of a meeting. [Figure 22] FIG. 10 is a diagram showing an example of the height of a panoramic image that is determined when a participant's face is detected. [Figure 23] 10A and 10B are diagrams illustrating an operation method for setting the direction of the electronic whiteboard by pressing a position registration button. [Figure 24] FIG. 10 is a diagram showing an example of a screen on which a user can confirm a direction set by the user. [Figure 25] 10 is a diagram illustrating a screen displayed on the electronic whiteboard for setting a method for detecting the orientation of the electronic whiteboard. FIG. [Figure 26] FIG. 10 is a diagram showing an example of a two-dimensional code displayed as a specific image on an electronic whiteboard. [Figure 27] 10A and 10B are diagrams illustrating a method for determining the direction of an electronic whiteboard when the electronic whiteboard outputs a specific sound. [Figure 28] 10 is an example of a sequence diagram illustrating a process in which a meeting device generates a panoramic image including an electronic whiteboard using specific video and audio. [Figure 29] FIG. 10 is a diagram illustrating an example of an automatic detection setting screen for an electronic whiteboard displayed by an information recording application. [Figure 30]10 is a diagram illustrating an example of an electronic whiteboard detected by image processing such as machine learning. [Figure 31] 10A and 10B are diagrams illustrating an example of a height of a panoramic image that is determined based on an electronic whiteboard detected by image processing. [Figure 32] FIG. 1 is a diagram illustrating a method for generating a panoramic image from a spherical image. [Figure 33] FIG. 10 is a diagram showing an example of a combined image displayed by the information recording application. [Figure 34] FIG. 10 is a flowchart illustrating an example of a process performed by a first image generating unit to determine the height of a panoramic image. [Figure 35] 1 is a diagram illustrating an example of an electronic whiteboard placed at the center of a panoramic image. [Figure 36] FIG. 10 is a diagram showing an example of a panoramic image generated when a button for fixing the display range is turned off. [Figure 37] FIG. 10 is a diagram showing an example of a panoramic image generated when a button for fixing the display range is turned on. [Figure 38] FIG. 10 is an example of a flowchart illustrating a process in which a first image generating unit generates a panoramic image when a button for fixing a display range is on or off. [Figure 39] FIG. 10 is a diagram showing an example of a panoramic image in which only a portion in the horizontal direction is cut off. [Figure 40] 10A and 10B are diagrams illustrating a process of omitting an excess gap when there is a gap between participants in a panoramic image. [Figure 41] 10 is a sequence diagram illustrating an example of a procedure in which the information recording application records a panoramic image, a speaker image, and an application screen. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a device system and a display method performed by the device system will be described as an example of an embodiment of the present invention.

[0010] <An example of how to create minutes for a remote meeting> First, an outline of a method for creating minutes using a panoramic image and an application screen will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. As shown in Fig. 1, a user at a home location 102 is using a remote conference service system 90 to hold a remote conference with another location 101. The record information creation system 100 of this embodiment creates record information (such as minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) acquired by a meeting device 60 equipped with an imaging means capable of capturing 360° images of the surroundings, a microphone, and a speaker, by processing information captured from the surroundings, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. Regarding audio, the record information creation system 100 synthesizes the audio received by the remote conference app 42 and the audio acquired by the meeting device 60, and includes the synthesized audio in the record information. An outline of this is given below.

[0011] (1) An information recording application 41 and a remote conference application, which will be described later, are running on the terminal device 10. In addition, an application for displaying materials may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including the audio received by the remote conference application from other locations) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired with the audio from the remote conference application.

[0012] (2) The meeting device 60 is equipped with a microphone and performs processing to extract the speaker from the panoramic image based on the direction from which the voice was captured, thereby creating a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0013] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines the panoramic image 203 and the speaker image 204 with any application screen selected by the user (for example, the screen 103 of a teleconferencing application). For example, the information recording application 41 combines the panoramic image 203, the speaker image 204, and the application screen 103 so that the panoramic image 203 and the speaker image 204 are arranged on the left and the teleconferencing application screen 103 is arranged on the right (hereinafter referred to as a combined image 105). The application screen is an example of screen information (described later) displayed by each application, such as a teleconferencing application. Since the process of (3) is executed repeatedly, the combined image 105 becomes a moving image (hereinafter referred to as a combined image moving image). The information recording application 41 also combines the synthesized audio with the combined image moving image to create a moving image with audio.

[0014] In this embodiment, an example in which the panoramic image 203, the speaker image 204, and the application screen 103 are combined will be described, but the information recording application 41 may store these separately and arrange them on the screen during playback.

[0015] (4) The information recording application 41 accepts editing operations (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0016] (5) The information recording application 41 transmits the created combined image video (with audio) to the storage service system 70 and stores it there.

[0017] (6) The information recording application 41 also extracts only the audio from the combined image video (the audio before combining may be kept), and transmits the extracted audio to the information processing system 50. The information processing system 50 transmits the audio to the speech recognition service system 80, which converts the audio into text data, and converts the audio into text. The text data also includes data on how many minutes after the start of recording that the speech occurred.

[0018] In the case of real-time text conversion, the meeting device 60 directly transmits the voice to the information processing system 50. The meeting device 60 transmits the text data to the information recording application 41 in real time.

[0019] (7) The information processing system 50 additionally stores the text data in the storage service system 70 that stores the combined image video. The text data constitutes part of the recorded information.

[0020] The information processing system 50 can perform billing processing for the user according to the service used. For example, the billing is calculated based on the amount of text data, the file size of the combined image video, the processing time, etc.

[0021] In this way, the combined image video displays a panoramic image of the surroundings including the user and an image of the speaker, and further displays the screen of an app that was displayed during the remote conference, such as the remote conference app 42. When participants in the remote conference or non-participants view the combined image video as minutes, the situation during the remote conference is reproduced with a sense of realism.

[0022] <Example of generating a panoramic image> Next, a method for generating a panoramic image in this embodiment will be described with reference to Fig. 2. Fig. 2 shows an example of a generated panoramic image 203. In Fig. 2, one panoramic image 203 (an example of a first image) and two speaker images 204 (an example of a second image) are arranged and displayed on one screen. The number of speaker images 204 is just an example, and there may be no speaker images 204, or there may be three or more speaker images 204.

[0023] 2(a) shows a panoramic image in which all of the participants 120 are seated. The height of the panoramic image 203 is L1, and the height of the speaker image 204 is L2.

[0024] Next, FIG. 2(b) shows a panoramic image 203 and a speaker image 204 when multiple participants 120 stand up. The meeting device 60 increases the height of the panoramic image 203 so that the faces of all participants 120 are included. For example, the meeting device 60 detects the faces of each participant 120 and determines the height so that at least all faces are included. The height of the panoramic image 203 in FIG. 2(b) is M1, and the height of the speaker image 204 is M2. Therefore, the following relationship exists between L1, L2, M1, and M2. L1<M1 、L2> M2

[0025] Next, FIG. 2(c) shows a panoramic image 203 and a speaker image 204 created to include the electronic whiteboard 2. The meeting device 60 detects the electronic whiteboard 2 using several methods described below, and increases the height of the panoramic image 203 so that the faces of all participants 120 and the electronic whiteboard 2 are included. For example, the meeting device 60 detects the faces of each participant 120 and the electronic whiteboard 2, and determines the height so that all faces and the electronic whiteboard 2 are included. The height of the panoramic image 203 in FIG. 2(c) is set to N1, and the height of the speaker image 204 is set to N2. Therefore, the following relationship exists between L1, L2, N1, and N2. L1<N1 、L2> N2

[0026] In either case of Figures 2(b) or (c), if multiple participants 120 sit down or if the electronic whiteboard 2 is no longer detected, the meeting device 60 adjusts the height of the panoramic image 203 again (in this case, makes it smaller).

[0027] In this way, the meeting device 60 of this embodiment detects multiple pre-set objects (participants' faces, devices such as the electronic whiteboard 2) and determines the height of the panoramic image so that the objects are included, thereby making it possible to display the objects in the panoramic image 203. When there are multiple objects to be included in the image, the meeting device 60 can display appropriate objects.

[0028] <Terminology> An application (app) is software developed and used for a specific function or purpose. Applications include native apps and web apps. Web apps (cloud-side apps provided by cloud services) can also operate in conjunction with native apps or web browsers.

[0029] A running app is an app that is in a state between when it is launched and when it is closed. The app does not have to be the active app (the foreground app), it just needs to be running in the background.

[0030] The image of the surroundings acquired by the meeting device is a spherical image, and a panoramic image is generated from the spherical image, captured at a wider angle of view in the horizontal direction than a normal angle of view. A spherical image is a wide-angle image capturing the surroundings over approximately 360° in both the vertical and horizontal directions. It does not necessarily have to be 360°, as long as it captures approximately the entire range of the meeting device 60. A spherical image is sometimes called a panoramic image, a 360° image, or the like. A single meeting device 60 does not have to capture a spherical image, and multiple imaging devices with normal angles of view may be combined. Furthermore, instead of a spherical image, a hemispherical image (an image with an angle of view of approximately 360° horizontally and approximately 90° vertically) may also be used.

[0031] A panoramic image is an image that captures the surroundings from a spherical image across approximately 360° in the horizontal direction. It does not necessarily have to be 360°; a wide-angle image of around 180° is also acceptable.

[0032] Recorded information is information recorded by the information recording application 41, and is stored and saved in a viewable manner as information linked to the identification information of a particular conference (meeting), and includes, for example, the following information: Video information created based on the screen information displayed by the selected app (such as a remote conferencing app) and image information of the device's surroundings acquired by the device. During a meeting, audio information is acquired and synthesized between the remote conference app (terminal device) and the meeting device at the base. -Text information converted from the captured audio. Other data and images related to the meeting, such as document files used during the meeting, notes added, translations of text data, and images and stroke data created using the cloud electronic whiteboard service during the meeting. For example, when the information recording application 41 records the screen of a remote conference application or the state of a conference at a base, the recorded information may be minutes of the conference that was held. Minutes are an example of recorded information, and the name of the recorded information changes depending on the content of the remote conference or the base, and may be called, for example, a record of communication or a record of the base situation. Furthermore, the recorded information includes files in multiple formats, such as video files (e.g., combined image videos), audio files, text data (text data obtained by voice recognition), document files, image files, and table-format files, and since the files are associated with each other based on the identification information of the conference, they can be viewed collectively or selectively in chronological order.

[0033] A tenant is a group of users (such as a company, a local government, or some of these organizations) who have signed a contract to receive services from a service provider. In this embodiment, the creation of recorded information and conversion to text data are performed because the tenant has signed a contract with the service provider.

[0034] Remote communication refers to communication through audio and video using software and terminal devices with people at physically distant locations. An example of remote communication is a remote conference, which may also be called a meeting, a conference, a consultation, a contract application, a gathering, a get-together, a seminar, a training session, a study group, a seminar, or a workshop.

[0035] A base is a place where activities are based. An example of a base is a conference room. A conference room is a room set up primarily for meetings. A base can also be a home, a reception desk, a store, a warehouse, an outdoor site, or any other location, as long as it is a place or space where terminal equipment, devices, etc. can be installed.

[0036] Although voice refers to speech sounds uttered by humans, surrounding sounds, etc., and voice data refers to voice converted into data, in the present embodiment, a strict distinction will not be made between them.

[0037] The multiple pre-set objects are objects to be displayed in the panoramic image, and in this embodiment, correspond to the faces of the participants (human faces) and the electronic whiteboard 2. The electronic whiteboard 2 may be called an electronic whiteboard, an electronic information board, or the like. A projector is known as a device equivalent to the electronic whiteboard 2. Other objects may also be electronic devices such as digital signage, televisions, displays, multifunction peripherals, and video conference terminals. The user may also be able to set the objects to be displayed in the panoramic image. In this case, the meeting device or terminal device has previously learned the shapes of objects, and detects the object selected by the user from the panoramic image. Multiple types of objects may exist simultaneously. For example, the meeting device 60 may simultaneously recognize a human face and an electronic device as objects.

[0038] The area of ​​an image is the height and width of the image, and is specified by the number of pixels, the number of pixels, the length, etc.

[0039] <System configuration example> Next, the system configuration of the record information creation system 100 will be described with reference to Fig. 3. Fig. 3 shows an example of the configuration of the record information creation system 100. Fig. 3 shows one location (own location 102) among multiple locations where a remote conference is held, and a terminal device 10 at the own location 102 communicates with an information processing system 50, a storage service system 70, and a remote conference service system 90 via a network. A meeting device 60 and an electronic whiteboard 2 are also disposed at the own location 102, and the terminal device 10 is communicatively connected to the meeting device 60 via a USB cable, an HDMI (registered trademark) cable, or the like. The terminal device 10 may also communicate with the meeting device 60 via a LAN. The meeting device 60 and the terminal device (or the information recording application 41) operate as an equipment system.

[0040] At least an information recording application 41 and a remote conference application 42 run on the terminal device 10. The remote conference application 42 can communicate with terminal devices 10 at other locations 101 via a remote conference service system 90 on the network, allowing users at each location to hold conferences from remote locations. The information recording application 41 creates record information for the remote conferences held by the remote conference application 42 using the functions of the information processing system 50 and the meeting device 60.

[0041] In this embodiment, an example of creating recorded information during a remote conference will be described, but the conference does not have to be a conference involving communication with a remote location. In other words, the conference may be a conference in which only participants from one location participate. In this case, only the audio collected by the meeting device 60 is saved without synthesis, and the processing of the information recording application 41 remains unchanged.

[0042] The terminal device 10 has a built-in camera with a normal angle of view (or it may be external), which captures a front image including the user 107 operating the terminal device 10. A normal angle of view is an image that is not a panoramic image, but in this embodiment, the normal angle of view is mainly a planar image that is not curved, such as a spherical image. The terminal device 10 also has a built-in microphone (or it may be external), which collects surrounding sounds of the user operating the terminal device 10. Therefore, the user can hold a conventional remote conference using the remote conference app 42 without being aware of the information recording app 41. The information recording app 41 and the meeting device 60 do not affect the remote conference app 42, except for increasing the processing load on the terminal device 10.

[0043] The information recording application 41 is an application that records information by communicating with the meeting device 60 and creating recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capturing device, a microphone, and a speaker. While the camera of the terminal device 10 can only capture a limited range in front of it, the meeting device 60 can capture images of the entire surroundings (not necessarily the entire periphery) of the meeting device 60. The meeting device 60 can always fit multiple participants 120 shown in FIG. 3 within its field of view.

[0044] In addition, the meeting device 60 extracts a speaker image from a panoramic image, synthesizes the audio acquired by the meeting device 60 with the audio output by the terminal device 10 (including the audio received by the remote conference application 42), and so on. The meeting device 60 is not limited to being placed on a desk, a table, or the like, and may be placed anywhere at the home base 102. Since the meeting device 60 can capture spherical images, it may be placed on the ceiling, for example. The meeting device 60 may also be installed at another base or at any base.

[0045] The information recording application 41 displays a list of applications running on the terminal device 10, synthesizes images for the recorded information (creates a combined image video), plays back the combined image video, accepts editing requests, etc. The information recording application 41 also displays a list of teleconferences that have been held or are scheduled to be held, etc. The list of teleconferences is used for information related to the recorded information, allowing the user to associate the teleconferences with the recorded information.

[0046] The remote conference application 42 is an application that enables a terminal device to remotely communicate with other terminal devices by establishing a communication connection with other terminal devices at other locations 101, transmitting and receiving images and audio, displaying images and outputting audio, etc. The remote conference application can also be called a remote communication application, a remote information common application, etc.

[0047] The information recording application 41 and the remote conference application 42 may be web applications or native applications. A web application is an application that performs processing through collaboration between a program on a web server and a program on a web browser or a native application, and does not need to be installed on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used. In this embodiment, both applications will be described as native applications.

[0048] The terminal device 10 may be, for example, a general-purpose information processing device with a communication function, such as a PC (Personal Computer), a smartphone, or a tablet terminal. The terminal device 10 may also be an electronic whiteboard 2, a game console, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, industrial machinery, medical equipment, or network home appliances. The terminal device 10 may be any device on which at least the information recording application 41 and the remote conference application 42 run. The terminal device 10 may be any device on which the information recording application 41 and the remote conference application 42 run.

[0049] The electronic whiteboard 2 displays data handwritten on a touch panel using an input means such as a pen or a finger. The electronic whiteboard 2 can communicate with a terminal device 10 or the like via wired or wireless communication, and can import screens displayed by the terminal device 10 and display them on the display. The electronic whiteboard 2 can convert handwritten data into text data and share information displayed on the display with electronic whiteboards 2 at other locations. The electronic whiteboard 2 may be a simple whiteboard (blackboard or screen) without a touch panel, in which an image is projected by a projector. The electronic whiteboard 2 may also be a tablet terminal, notebook PC (Personal Computer), PDA (Personal Digital Assistant), game console, or the like equipped with a touch panel.

[0050] The interactive whiteboard 2 can communicate with the information processing system 50. For example, after being powered on, the interactive whiteboard 2 can receive information from the information processing system 50 by polling the information processing system 50, for example.

[0051] The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and a platform service. The server applications manage remote conference lists, manage recorded information recorded in remote conferences, and manage various settings and storage paths. The platform service handles user authentication, contracts, billing, etc.

[0052] All or part of the functions of the information processing system 50 may exist in a cloud environment or an on-premise environment. The information processing system 50 may be configured with multiple server devices or may be configured with a single information processing device. For example, the server application and the infrastructure service may be provided by separate information processing devices, and each function within the server application may have its own information processing device. The information processing system 50 may be integrated with the storage service system 70 and the voice recognition service system 80 described below.

[0053] The storage service system 70 is a storage means on a network and provides a storage service that accepts the saving of files, etc. Known examples of the storage service system 70 include OneDrive (registered trademark), Google Workspace (registered trademark), and DropBox (registered trademark). The storage service system 70 may also be an on-premise NAS (Network Attached Storage), etc.

[0054] The voice recognition service system 80 provides a service that performs voice recognition on voice data and converts it into text data. The voice recognition service system 80 may be a general-purpose commercial service, or may be part of the functions of the information processing system 50. Furthermore, the voice recognition service system 80 may be set up and used as a different service system for each user, tenant, or conference.

[0055] <Hardware configuration example> The hardware configuration of the information processing system 50 and the terminal device 10 according to this embodiment will be described with reference to FIG.

[0056] <<Information processing system and terminal device>> Fig. 4 is a diagram showing an example of the hardware configuration of an information processing system 50 and a terminal device 10 according to this embodiment. As shown in Fig. 4, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0057] Of these, the CPU 501 controls the overall operation of the information processing system 50 and the terminal device 10. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 4.

[0058] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0059] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be described using Figure 5. Figure 5 is an example of a hardware configuration diagram of a meeting device 60 capable of capturing 360-degree video. In the following, the meeting device 60 is described as a device that uses an imaging element to capture 360-degree video around the device at a predetermined height, but the imaging element may be one, two, or any number of elements. Furthermore, it does not necessarily have to be a dedicated device; a PC, digital camera, smartphone, etc. may have substantially the same functionality by attaching a 360-degree video imaging unit to it after the fact.

[0060] As shown in Figure 5, the meeting device 60 is composed of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a concave terminal 621 for Micro USB.

[0061] Of these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 with a 360° angle of view for forming a hemispherical image, and imaging elements 603 (image sensors) provided corresponding to each wide-angle lens. The imaging elements 603 include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lens 602 into image data in the form of an electrical signal and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers that set various commands and parameters required for the operation of the imaging elements. The imaging unit 601 may also be a 360° camera, which is an example of imaging means capable of capturing images of the 360° surrounding the meeting device 60.

[0062] The imaging elements 603 (image sensors) of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I / F bus. Meanwhile, the imaging elements 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an audio sensor 618, and the like are also connected to the bus 610.

[0063] The image processing unit 604 takes in the image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates data for a panoramic image and a speaker image from the fisheye video. Furthermore, the image processing unit 604 synthesizes the panoramic image and the speaker image, etc., and outputs a single video.

[0064] The imaging control unit 605 generally uses the I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0065] Furthermore, the imaging control unit 605 instructs the imaging elements 603a and 603b to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from the PC. Some meeting devices 60 have a preview display function or a function that supports video display on a display (for example, a PC or smartphone display). In this case, the image data is output from the imaging element 603 continuously at a predetermined frame rate (frames per minute).

[0066] As will be described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0067] The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 via the I / F bus and performs predetermined processing on the sound data.

[0068] The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of processed equirectangular projection images.

[0069] The operation unit 615 is a general term for operation buttons such as an imaging start button 615a. By operating the operation unit 615, the user can start imaging or recording, as well as turn the power on / off, establish communication connections, and input settings such as various imaging modes and imaging conditions.

[0070] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device may be, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection I / F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. The video data and image data stored in the DRAM 614 are transmitted to an external terminal via the external device connection I / F 616 or recorded on external media. Furthermore, by using multiple external device connection I / Fs 616, for example, image information captured by the meeting device 60 may be transmitted to a PC via USB for recording, while video (e.g., screen information displayed by a remote conference app) may be acquired from the PC to the meeting device 60, and the video may then be transmitted from the meeting device 60 via HDMI to another external device (e.g., a display, a projector, an electronic whiteboard) for display.

[0071] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the meeting device 60, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also communicate with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0072] The audio sensor 618 is a sensor that acquires 360° audio information to identify from which direction a loud audio is being input within 360° around (horizontal plane) the meeting device 60. The audio processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction within 360°.

[0073] Note that other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use this information for image correction or adding location information.

[0074] The image processing unit 604 also performs the following processes.

[0075] The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB interpolation processing) on ​​the RAW data input from the image sensor that inputs spherical images to create a fisheye image (image made up of curved images).The CPU 611 then performs flattening processing such as DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image (image made up of flat images) that captures 360 degrees around the meeting device 60.

[0076] The CPU 611 creates speaker images in the following way. The CPU 611 creates speaker images by cutting out the speaker from a panoramic image (flat video) that captures 360° of the surrounding area. The CPU 611 cuts out the speaker image from the panoramic image, using the voice input direction identified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. In this case, the method for cutting out a person's image from the voice input direction is to cut out a 30° area centered on the identified voice direction from the 360°, and perform face detection within that area to cut out the image. The CPU 611 further identifies the speaker images of a specific number of people (e.g., three people) who have spoken most recently from the cut-out speaker images.

[0077] The panoramic image and the one or more speaker images may be sent separately to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the one or more speaker images are sent separately from the meeting device 60 to the information recording application 41.

[0078] 6 is a diagram illustrating the imaging range of the meeting device 60. As shown in FIG. 6(a), the meeting device 60 captures an image in a range of 360° in the horizontal direction. As shown in FIG. 6(b), the imaging range of the meeting device 60 is set to 0° in the direction horizontal to the height of the meeting device 60, and a predetermined angle (a° to b°) above and below. In this embodiment, the predetermined angle (a° to b°) above and below is variable.

[0079] FIG. 7 is a diagram illustrating the cutting out of a panoramic image and a speaker image. As shown in FIG. 7, the image captured by the meeting device 60 forms a portion 110 of a sphere and therefore has a three-dimensional shape. As shown in FIG. 6(b), the meeting device 60 performs perspective projection transformation by dividing the angle of view into predetermined angles in the vertical direction and the horizontal direction. By performing perspective projection transformation across the entire 360° horizontal direction without gaps, a predetermined number of planar images are obtained, and a panoramic image 111 is obtained by connecting the predetermined number of planar images horizontally. In addition, the meeting device 60 performs face detection within a predetermined range from the panoramic image centered on the direction of the voice, and creates a speaker image 112 by cutting out an image 15° left and right from the center of the face (30° overall).

[0080] <<Electronic Whiteboard>> Fig. 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Fig. 8, the electronic whiteboard 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a network I / F 405, and an external device connection I / F (Interface) 406.

[0081] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores the CPU 401 and programs used to drive the CPU 401, such as an IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with a communication network. The external device connection I / F 406 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 430 and external devices (a microphone 440, a speaker 450, and a camera 460).

[0082] The electronic whiteboard 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0083] Of these, the capture device 411 displays video information as still images or moving images on the display of an external PC (Personal Computer) 470. The GPU (Graphics Processing Unit) 412 is a semiconductor chip that specializes in graphics. The display controller 413 controls and manages the screen display to output the output image from the GPU 412 to a display 480 or the like. The contact sensor 414 detects contact of an electronic pen 490, a user's hand 491, or the like on the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 inputs and detects coordinates using an infrared blocking method. The method for inputting and detecting coordinates will be described below. Two light-receiving and light-emitting devices installed at both ends of the upper side of the display 480 emit multiple infrared rays parallel to the display 480. The two light-receiving and light-emitting devices receive light that is reflected by a reflecting member installed around the periphery of the display 480 and returns along the same optical path as the light emitted by the light-receiving element.

[0084] The contact sensor 414 outputs the ID of the infrared light emitted by the two light receiving and emitting devices that has been blocked by an object to the sensor controller 415, and the sensor controller 415 identifies the coordinate position of the contact position of the object. The electronic pen controller 416 communicates with the electronic pen 490 to determine whether the pen tip or the pen tail has touched the display 480. The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching the power of the electronic whiteboard 2 on and off. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 480.

[0085] Furthermore, the electronic whiteboard 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0086] The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 414 may be a resistive film type touch panel that identifies the contact position by voltage changes across two opposing resistive films. The contact sensor 414 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. The contact sensor 414 may use various other detection means. Furthermore, the electronic pen controller 416 may determine whether or not the electronic pen 490 is touched not only by the pen tip and pen tail, but also by the part of the electronic pen 490 that the user holds or other parts of the electronic pen.

[0087] <About the function> Next, the functional configuration of the record information creation system 100 will be described with reference to Fig. 9. Fig. 9 is an example of a functional block diagram that explains the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information creation system 100 by dividing them into blocks.

[0088] <<Terminal Device>> The information recording application 41 running on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a recording control unit 17, an audio data processing unit 18, a recording / playback unit 19, an upload unit 20, and an editing processing unit 21. Each of these units included in the terminal device 10 is a function or a means for performing the function, which is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 501 in accordance with the information recording application 41 loaded from the HD 504 to the RAM 503. The terminal device 10 also includes a storage unit 1000 implemented by the HD 504 and the like shown in Fig. 4. The storage unit 1000 includes an information storage unit 1001.

[0089] The communication unit 11 communicates various types of information with the information processing system 50 via a network. For example, the communication unit 11 receives a list of remote conferences from the information processing system 50 and transmits a request for recognizing voice data to the information processing system 50.

[0090] The display control unit 13 displays various screens serving as a user interface in the information recording application 41 in accordance with screen transitions set in the information recording application 41. The operation accepting unit 12 accepts various operations for the information recording application 41.

[0091] The application screen acquisition unit 14 acquires screen information displayed by an application selected by the user or screen information of the desktop screen from an operating system (OS). If the application selected by the user is a remote conference application 42, screens generated by the remote conference application 42 (images captured by the terminal device cameras at each location, images displayed with shared materials, participant icons, participant names, etc.) are acquired. The screen information displayed by the application (application screen) is information displayed as a window by the running application and acquired as an image by the information recording application. The application window is displayed on a monitor or the like, with the window area depicted as an area within the entire desktop image. The screen information displayed by the application can be acquired by other applications (such as an information recording application) as an image file or a video file consisting of multiple consecutive images via the API of the operating system (OS) or the API of the application that displays it. The screen information of the desktop screen is information composed of an image of the desktop screen generated by the OS and can be similarly acquired as an image file or a video file via the API of the OS. These image files may be in bitmap, PNG, or other formats. The video file format may be MP4 or other formats.

[0092] The audio acquisition unit 15 acquires audio output from the microphone or earphones of the terminal device 10 (including audio data received from the remote conference application 42 during the remote conference). The audio acquisition unit 15 can acquire audio even when the output audio is muted. Regarding audio data, the user does not need to perform any operation, such as selecting the remote conference application 42. The audio acquisition unit 15 can acquire audio that the terminal device 10 can output via the OS or application API (Application Interface). This also acquires audio data received by the remote conference application 42 from other locations 101. If the remote conference application 42 is not running or if a remote conference is not in progress, the information recording application 41 may not be able to acquire audio data. Note that the audio acquired by the audio acquisition unit 15 may not include audio collected by the terminal device 10, and may only include the audio data to be output. This is because the meeting device 60 separately collects audio from the locations.

[0093] The device communication unit 16 communicates with the meeting device 60 using a USB cable, an HDMI (registered trademark) cable, or the like. The device communication unit 16 may communicate with the meeting device 60 via wireless LAN, Bluetooth (registered trademark), or the like. The device communication unit 16 receives the panoramic image 203 and the speaker image 204 from the meeting device 60, and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0094] The recording control unit 17 combines the panoramic image and speaker image received by the device communication unit 16 and the app screen acquired by the app screen acquisition unit 14 to create a combined image. The recording control unit 17 also chronologically connects the repeatedly created combined images to create a combined image video, and combines the synthesized audio data with the combined image video to create a combined image video with audio. The combination of the panoramic image and speaker image may be performed by the meeting device 60. Also, videos consisting of each image, such as the panoramic image, speaker image, app screen, and image consisting of the panoramic image and speaker image, may be stored in the storage service system 70 as separate video files. In this case, the panoramic video, speaker video, video of the app screen, and combined video of the panoramic image and speaker image may be called up and displayed on a single display screen when viewed.

[0095] The audio data processing unit 18 extracts the audio data combined with the combined image video, or requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

[0096] The recording / playback unit 19 plays back the combined image moving image. The combined image moving image is stored in the terminal device 10 while being recorded, and is then uploaded to the information processing system 50.

[0097] When the remote conference ends, the upload unit 20 transmits the combined image moving image to the information processing system 50.

[0098] The editing processing unit 21 executes editing of the combined image moving image (deleting a part, splicing, etc.) in response to a user operation.

[0099] Fig. 10 shows video recording information stored in the information storage unit 1001. The video recording information includes items such as a conference ID, a recording ID, an update date and time, a title, an upload, and a save destination. When a user logs in to the information processing system 50, the information recording application 41 downloads the conference information from the conference information storage unit 5001 of the information processing system 50. The conference ID and other information included in the conference information are reflected in the video recording information. The video recording information in Fig. 10 is held by a terminal device 10 operated by a certain user.

[0100] The conference ID is identification information for identifying the remote conference that has been held. The conference ID is assigned when a remote conference schedule is registered in the conference management system 9, or is assigned by the information processing system 50 in response to a request from the information recording application 41. The conference management system 9 is a system for registering conference and remote conference schedules, a URL (conference link) for starting a remote conference, reservation information for devices to be used in the conference, and the like, and is a scheduler or the like connected from the terminal device 10 via a network. The conference management system 9 can also transmit the registered schedule, etc. to the information processing system 50.

[0101] The recording ID is identification information for identifying the combined image video recorded during a remote conference. The recording ID is assigned by the meeting device 60, but may also be assigned by the information recording application 41 or the information processing system 50. A different recording ID is assigned to the same conference ID when recording stopped during the remote conference but was resumed for some reason.

[0102] The update date and time is the date and time when the combined image video was updated (recording ended). If the combined image video was edited, this is the edit date and time.

[0103] The title is the name of the conference. It may be set when the conference is registered in the conference management system 9, or may be set arbitrarily by the user.

[0104] The "upload" indicates whether the combined image video has been uploaded to the information processing system 50.

[0105] The destination indicates the location (URL or file path) where the combined image video and text data are saved in the storage service system 70. Therefore, the user can view the uploaded combined image video at will. The combined image video and text data are saved with separate file names, for example, following the URL.

[0106] <<Meeting Devices>> Returning to Figure 9, the meeting device 60 has a terminal communication unit 61, a first image generation unit 62, a second image generation unit 63, a sound collection unit 64, a voice synthesis unit 65, a participant detection unit 66, a voice direction detection unit 67, a code analysis unit 68, and a device recognition unit 69. Each of these units in the meeting device 60 is a function or a means for performing the function, which is realized when any of the components shown in Figure 5 operates in response to an instruction from the CPU 611 in accordance with a program loaded from the ROM 612 to the DRAM 614.

[0107] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable, an HDMI (registered trademark) cable, etc. The terminal communication unit 61 may be connected not only by a wired cable but also by wireless LAN, Bluetooth (registered trademark), etc.

[0108] The first image generation unit 62 generates a panoramic image 203. The second image generation unit 63 generates a speaker image 204. The methods for generating these images have been described with reference to Figs. 6 and 7. Details will be described later. The first image generation unit 62 and / or the second image generation unit 63 may be included in the information recording application 41.

[0109] The sound collection unit 64 converts the audio signal acquired by the microphone of the meeting device 60 into audio data (digital). As a result, the content of speech made by the user or participant at the base on the terminal device 10 side is collected.

[0110] The voice synthesis unit 65 synthesizes the voice transmitted from the terminal device 10 with the voice collected by the voice collection unit 64. As a result, the voices spoken at the other locations 101 and the voices spoken at the own location 102 are combined.

[0111] The participant detection unit 66 detects participants from the spherical image. The participant detection unit 66 detects participants by performing face recognition using machine learning techniques such as deep learning and support vectors. The participant detection unit 66 may detect human faces, but may also detect bodies as well.

[0112] The voice direction detection unit 67 detects the direction of the interactive whiteboard 2 in the panoramic image by detecting a voice of a specific frequency.

[0113] The code analysis unit 68 detects two-dimensional codes and barcodes included in the panoramic image, and analyzes the two-dimensional codes and barcodes to acquire information included in the two-dimensional codes and barcodes, such as device identification information of the interactive whiteboard 2. Note that the code analysis may be performed by the terminal device 10.

[0114] The device recognition unit 69 learns the shape (circumscribed rectangle) of the electronic whiteboard 2 by machine learning and detects the electronic whiteboard 2 from the panoramic image. The device recognition unit 69 may perform recognition simply by pattern matching without using machine learning. This device recognition may also be performed by the terminal device 10.

[0115] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a communication management unit 54, a device management unit 55, and a text conversion unit 56. Each of these units included in the information processing system 50 is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The information processing system 50 also includes a storage unit 5000 configured using the HD 504 and the like shown in Fig. 4. The storage unit 5000 includes a conference information storage unit 5001, a recording information storage unit 5002, an association information storage unit 5003, and an account information storage unit 5004.

[0116] The communication unit 51 transmits and receives various types of information to and from the terminal device 10. For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 and receives a request to recognize voice data from the terminal device 10.

[0117] The authentication unit 52 authenticates the user operating the terminal device 10. For example, the authentication unit 52 authenticates the user based on whether or not authentication information (user ID and password) included in an authentication request received by the communication unit 51 matches authentication information stored in advance. Note that the authentication information may be an IC card number, biometric authentication information such as face or fingerprint, or the like. The authentication unit 52 may also perform authentication using an external authentication system or an authentication method such as OAUTH.

[0118] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 executes a native application, the screen information is stored in the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 executes a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), or the like.

[0119] The communication management unit 54 acquires information about the remote conference from the conference management system 9 using each user's account or a system account assigned to the information processing system 50. The communication management unit 54 stores the conference information of the reserved conference in the conference information storage unit in association with the conference ID. The communication management unit 54 also acquires conference information that users belonging to the tenant have permission to view. Since a conference ID is set for each conference, the remote conference and recorded information are associated with each other by the conference ID.

[0120] The device management unit 55 associates the device identification information of the electronic whiteboard 2 and the meeting device 60 with the conference ID and manages them. That is, the device management unit 55 associates the devices that participated in the same conference. One method is for the meeting device 60 to acquire the device identification information that is displayed or output as audio by the electronic whiteboard 2, and for the terminal device 10 to transmit it to the information processing system 50.

[0121] The text conversion unit 56 converts the voice data requested by the terminal device 10 to text data into text data using an external voice recognition service, or the text conversion unit 56 itself may perform the conversion.

[0122] FIG. 11 shows an example of conference information stored in the conference information storage unit 5001 managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote conferences for which the user belonging to the tenant has viewing authority. Viewing authority may be granted to the conference information managed by the conference management unit 54 directly from the information recording application of the terminal device 10. The remote conference information for which the user belonging to the tenant has viewing authority includes information about conferences created by the user and information about conferences for which the user has been granted viewing authority by other users. In this embodiment, a remote conference is used as an example, but the list of remote conferences also includes conferences held in only one conference room.

[0123] The conference information is managed by a conference ID and is associated with the organizer ID, title (conference name), start date and time, end date and time, electronic whiteboard 2, meeting device 60, etc. These are examples of conference information, and the conference information may include other information as well.

[0124] The organizer ID is the organizer (host) of the meeting.

[0125] The title item indicates the content of the conference, such as the conference name and overseas agenda.

[0126] The start date and time item is the date and time when the conference is scheduled to start.

[0127] The end date and time item is the date and time when the conference is scheduled to end.

[0128] The item of the electronic whiteboard 2 is identification information of the electronic whiteboard 2 associated with the conference.

[0129] The meeting device 60 item is the identification information of the meeting device 60 used in the conference.

[0130] As shown in FIGS. 10 and 11, the combined image video recorded at the conference is identified by the conference ID.

[0131] The recording information stored in the recording information storage unit 5002 may be the same as that shown in Fig. 10. However, the information processing system 50 has a list of combined image videos recorded by all users belonging to the tenant. The user may input desired storage destination information (path information such as a URL of a cloud storage system) on a user setting screen of the information recording application 41 of the terminal device 10, and the desired storage destination information may be stored in the recording information storage unit 5002.

[0132] 12 shows association information in which the conference ID and the device identification information (the electronic whiteboard 2 and the meeting device 60) are associated with each other, which is stored in the association information storage unit 5003. The association information is held from when the information recording application 41 transmits the device identification information to the information processing system 50 until the recording is completed.

[0133] 13 is an example of account information stored in the account information storage unit 5004. The account information includes information on not only humans but also non-human users such as the electronic whiteboard 2 and the meeting device 60.

[0134] The user ID is identification information of the users who can participate in the meeting, the electronic whiteboard 2, the meeting device 60, etc.

[0135] The type is the type of each account of the user, the electronic whiteboard 2, or the meeting device 60.

[0136] The name is the name of the user, the name of the electronic whiteboard 2, or the name of the meeting device 60.

[0137] The email address is the email address of the user, the electronic whiteboard 2, the meeting device 60, etc.

[0138] <<Electronic Whiteboard>> 14 is a functional block diagram explaining the functions of the electronic whiteboard 2 by dividing them into blocks. The electronic whiteboard 2 has a contact position detection unit 31, a drawing data generation unit 32, a data recording unit 33, a display control unit 34, a code generation unit 35, a communication unit 36, a voice data generation unit 37, and an operation detection unit 38. Each function of the electronic whiteboard 2 is a function or means realized when any of the components shown in FIG. 8 operates in response to an instruction from the CPU 401 in accordance with a program loaded from the SSD 404 onto the RAM 403.

[0139] The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 contacts the contact sensor 414. The drawing data generation unit 32 acquires the coordinates where the pen tip of the electronic pen 490 contacts from the contact position detection unit 31. The drawing data generation unit 32 generates stroke data by connecting this sequence of coordinate points by interpolating them.

[0140] The display control unit 34 displays the handwritten data, character strings converted from the handwritten data, and menus for user operation on the display.

[0141] The data recording unit 33 stores handwritten data written on the electronic whiteboard 2, shapes converted from handwritten data such as circles and triangles, stamps such as "Done", PC screens, files, etc. in the object information storage unit 3002. Handwritten data, character strings (including shapes), images such as PC screens, files, etc. are handled as objects. For handwritten data, a group of stroke data becomes one object depending on whether it is a time division due to an interruption in handwriting or a distance division due to a difference in handwriting location.

[0142] The communication unit 36 ​​is connected to Wi-Fi or a LAN and communicates with the information processing system 50. The communication unit 36 ​​transmits object information to the information processing system 50, receives object information stored in the information processing system 50 from the information processing system 50, and displays the object information on the display 480. The communication unit 36 ​​can also communicate with the terminal device 10 directly or via the information processing system 50.

[0143] The code generation unit 35 generates a two-dimensional code by encoding the device identification information of the interactive whiteboard 2 stored in the device information storage unit 3001 and information indicating that the device is usable in a conference into a two-dimensional pattern. The code generation unit 35 can also encode the device identification information of the interactive whiteboard 2 and information indicating that the device is usable in a conference into a barcode. The device identification information may be a serial number, a UUID, or the like. The device identification information may also be set by the user.

[0144] The audio data generation unit 37 generates audio data by sampling a signal of a preset frequency (a frequency indicating output from the interactive whiteboard 2) at regular intervals, similar to PCM conversion. The audio data is converted to analog by a D / A converter included in the speaker 450 and output from the speaker 450.

[0145] The operation detection unit 38 detects a user operation on the electronic whiteboard 2. The operation detection unit 38 detects the start or end of an operation, for example, when the contact position detection unit 31 detects that the electronic pen 490, the user's hand 491, or the like has come into contact with (or simply approached) the display 480 (touch panel).

[0146] The electronic whiteboard 2 also has a storage unit 3000 constructed using the SSD 404 and the like shown in Fig. 8. The storage unit 3000 includes a device information storage unit 3001 and an object information storage unit 3002.

[0147] FIG. 15 shows the device identification information and the like stored in the device information storage unit 3001. The device identification information is the identification information of the electronic whiteboard 2. The IP address is an IP address that allows other devices to connect to the electronic whiteboard 2 via a network. The password is used to authenticate other devices when they connect to the electronic whiteboard 2.

[0148] 16 is a diagram illustrating the object information stored in the object information storage unit 3002. The object information is information for managing objects displayed on the electronic whiteboard 2. The object information is transmitted to the information processing system 50 and used as minutes.

[0149] The conference ID is identification information of the conference notified by the information processing system 50.

[0150] · Object ID is identification information that identifies an object.

[0151] Type is the type of object, and can be handwritten, text, shape, image, etc. Handwritten text is stroke data (a sequence of coordinate points). Text is a string of characters (character code) converted from handwritten data. Text is sometimes called text data. Shapes are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, and TIFF imported from a PC or the Internet.

[0152] One screen of the electronic whiteboard 2 is called a page. The item on a page is its page number.

[0153] The coordinates indicate the position of the object relative to a predetermined origin on the electronic whiteboard 2. The object position is, for example, the upper left vertex of the bounding rectangle of the object. The coordinates are expressed, for example, in pixel units of the display.

[0154] The size is the width and height of the object's bounding rectangle.

[0155] <Screen transition> Next, several screens that the terminal device 10 displays during a remote conference will be described with reference to Fig. 17 to Fig. 20. Fig. 17 shows an initial screen 200 after login that is displayed by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs authentication information and successfully logs in, the initial screen 200 of Fig. 17 is displayed.

[0156] The initial screen 200 has a fixed display button 201, a change front button 202, a button 219 for fixing the display range, a position registration button 207, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter, referred to as speaker images 204 when not distinguished), and a start recording button 205. If the meeting device 60 is already running and capturing images of the surroundings at the time of login, the panoramic image 203 and speaker image 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether to start recording while viewing these. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker image 204 are not displayed.

[0157] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or may display only the speaker images 204 of the N most recently spoken participants. Fig. 17 shows an example in which speaker images 204 of up to three people are displayed. Until a participant speaks, there may be no speaker image 204 (the speaker images are added one by one as the participant speaks), or three speaker images 204 of participants in a predetermined direction may be displayed (the speaker images are replaced as the participant speaks).

[0158] When no one is speaking, such as immediately after the meeting device 60 is started up, a predetermined direction (0°, 120°, 240°, etc.) within the horizontal 360° is created as the speaker image 204. If a fixed display, which will be described later, is set, the fixed display setting takes priority.

[0159] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 in close-up as a speaker image 204 .

[0160] The front change button 202 is a button that allows the user to perform an operation to change the front of the panoramic image 203 (because a panoramic image covers 360° horizontally, the right and left edges are aligned). The user can slide the panoramic image 203 left and right with a pointing device to determine which participant is displayed in front. The user's operation is transmitted to the meeting device 60, which then changes the angle of the front within the 360° horizontal direction to create a panoramic image and transmits it to the terminal device 10.

[0161] The button 219 for fixing the display range is a button for the user to set whether or not to reduce the panoramic image 203 so that it fits within the display range of the information recording application 41 after changing the height of the panoramic image.

[0162] The position registration button 207 is a button for the user to perform an operation to set the position (direction) of a device such as the electronic whiteboard 2.

[0163] When the user presses the record start button 205, the information recording application 41 displays the recording setting screen 210 shown in FIG.

[0164] 18 is an example of a recording setting screen 210 displayed by the information recording application 41. On the recording setting screen 210, the user can set whether to record (whether to include in the recording) the panoramic image and speaker image created by the meeting device 60, and the desktop screen or the screen of a running application of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or the screen of a running application, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) is recorded.

[0165] The camera toggle button 211 is a button for switching on and off the recording of the panoramic image and the speaker image created by the meeting device 60. The camera toggle button 211 may be set to record the panoramic image and the speaker image separately.

[0166] The PC screen toggle button 212 is a button for switching on and off recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10. When the PC screen toggle button 212 is in the on state, the desktop screen is recorded.

[0167] If the user wants to record the screen of an application, the user further selects the application in the application selection field 213. The application selection field 213 displays the names of applications currently running on the terminal device 10 in a pull-down menu. The user can select the application to record. The information recording application 41 acquires this application name from the OS. The information recording application 41 can display only those applications that have a UI (screen) among the currently running applications. The selected applications may include the remote conference application 42. Therefore, the information recording application 41 can record videos of materials displayed in the remote conference application 42 and participants at each location. In addition, the applications displayed in the pull-down menu include various applications currently running on the terminal device, such as a presentation application, a word processing application, a spreadsheet application, an application for creating and editing documents or other materials, a cloud electronic whiteboard application, and a web browser application. Therefore, the user can flexibly select the application screens to include in the combined image video.

[0168] When recording on an app basis, the user can select multiple apps, and the information recording app 41 can record the screens of all the selected apps.

[0169] When both the camera toggle button 211 and the PC screen toggle button 212 are off, the recording content confirmation window 214 displays "Only audio will be recorded." This audio includes the audio output by the terminal device 10 (audio received by the remote conference app 42 from other locations 101) and the audio collected by the meeting device 60. In other words, as long as a remote conference is being held, the audio from the remote conference app 42 and the audio from the meeting device 60 are saved regardless of whether images are recorded. However, the user may be able to selectively stop saving the audio from the remote conference app 42 and the audio from the meeting device 60 using user settings.

[0170] The combined image moving image is recorded as follows according to the combination of on and off of the camera toggle button 211 and the PC screen toggle button 212. The combined image moving image is also displayed in real time in the recording content confirmation window 214.

[0171] When the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image are displayed in the recording content confirmation window 214 .

[0172] When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen has also been selected), the recording content confirmation window 214 displays the desktop screen or the screen of the selected application.

[0173] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, are displayed side by side in the recording content confirmation window 214.

[0174] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or where the panoramic image, speaker image, and app screen are not recorded at all, but in this embodiment, for convenience, the image created by the information recording application 41 is referred to as a combined image video.

[0175] Furthermore, the recording setting screen 210 has a check box 209 along with the message "Automatically transcribe recording after uploading." The recording setting screen 210 also has a Start Recording Now button 217. When the user checks the check box 209, text data obtained by converting comments made during the remote conference is attached to the recorded video. In this case, after the recording is completed, the information recording application 41 uploads the audio to the information processing system 50 along with a request to convert it into text data. Furthermore, when the user presses the Start Recording Now button 217, the recording in progress screen 220 shown in FIG. 19 is displayed.

[0176] FIG. 19 is an example of a recording screen 220 that the information recording application 41 displays during recording. Note that the explanation of FIG. 19 will mainly focus on the differences from FIG. 17. The recording screen 220 displays a combined image video being recorded in real time under the conditions set by the user on the recording setting screen 210. The recording screen 220 of FIG. 19 is displayed when the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays a panoramic image 203 and a speaker image 204 (both of which are videos) created by the meeting device 60. The recording screen 220 displays a recording icon 225, a pause button 226, and an end recording button 227.

[0177] The pause button 226 is a button for stopping recording, and after stopping, recording can also be resumed. The end recording button 227 is a button for ending recording. The pause button 226 does not change the recording ID, but the end recording button 227 changes the recording ID. After pausing, when recording is resumed, the user can reset the recording conditions set on the recording setting screen 210. In this case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, by pressing the end recording button 227), or may combine multiple files so that they form a single continuous video (for example, by pressing the pause button 226). Furthermore, when the information recording application 41 plays back a combined image video, it may play back multiple recording files continuously as a single video.

[0178] Furthermore, the recording in progress screen 220 has an Get Information from Calendar button 221, a conference name field 222, a time field 223, and a location field 224. The Get Information from Calendar button 221 is a button that the user uses to obtain conference information from the conference management system 9. When the Get Information from Calendar button 221 is pressed, the information recording application 41 obtains and displays a list of conferences that the user has permission to view from the information processing system 50. The user selects the upcoming remote conference from the conference list. This causes the conference information to be reflected in the conference name field 222, time field 223, and location field 224. The conference name field 222 reflects the title of the conference information, the time field 223 reflects the start time and end time, and the location field 224 reflects the location. Furthermore, the conference information and recording information in the conference management system are associated with each other by a conference ID.

[0179] Once the remote conference is over and the user stops recording, a combined image video with audio is created.

[0180] FIG. 20 is an example of the conference list screen 230 displayed by the information recording application 41. The conference list screen 230 is a list of conferences, and can display a list of recorded information recorded during remote conferences. It also includes conferences that were held only in a certain conference room, regardless of whether they were remote conferences. The conference list screen 230 displays conference information that the logged-in user has permission to view in the conference information storage unit 5001. Information on video recording information stored in the information storage unit 1001 may be further integrated.

[0181] The conference list screen 230 is displayed when the user selects the conference list tab 231 on the initial screen 200 in FIG. 17. The conference list screen 230 displays a list 236 of recorded information that the user has permission to view. The conference creator (minutes creator) can set viewing permissions for participants. The conference list may be a list of stored recorded information or a list of conference schedules and conference data.

[0182] The conference list screen 230 has the following items: a check box 232 , an update date and time 233 , a title 234 , and a status 235 .

[0183] The checkboxes 232 are used to select recorded files. The checkboxes 232 are used when the user wants to delete recorded files all at once.

[0184] The update date and time 233 indicates the start and end times of recording the combined image video. If the video has been edited, the edit date and time may be used.

[0185] The title 234 is the title of the meeting (e.g., agenda). It may be transcribed from the meeting information or set by the user.

[0186] The status 235 indicates whether the combined image video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, an upload button is displayed. If there is a combined image video that has not been uploaded, it is recommended that the information recording application 41 automatically uploads it when the user logs in to the information processing system 50.

[0187] When the user selects an arbitrary title or the like from the list 236 of combined image videos with a pointing device, the information recording application 41 displays a recording playback screen, but this is not shown in this embodiment. On the recording playback screen, it is possible to play back the combined image video, etc.

[0188] It is desirable that the user be able to narrow down the conferences based on update date and time, title, keywords, etc. Also, if there are a large number of conferences displayed and it is difficult to find the relevant conference, it is desirable that a search function be provided that allows the user to input a phrase to narrow down the recorded information based on words contained in the remarks made at the conference, the title, etc. The search function allows the user to find the desired recorded information in a short time even when there is a large amount of recorded information. Furthermore, the conference list screen 230 may allow the user to sort the conferences by update date and time or title.

[0189] <Action or Processing> FIG. 21 is an example of a sequence diagram illustrating a process in which the meeting device 60 creates a panoramic image 203 from the start of the conference.

[0190] S1: The user inputs a conference start command into the information recording application 41. Note that a so-called remote conference starts when the remote conference application 42 connects with another location 101. The start of the conference in step S1 means the start of recording (turning on the start recording now button 216). The detailed process of creating recorded information will be described with reference to FIG. 41.

[0191] S2: The operation reception unit 12 of the information recording application 41 receives the operation, and the device communication unit 16 notifies the meeting device 60 of the start of the conference.

[0192] S3: The terminal communication unit 61 of the meeting device 60 receives the start of the conference, the participant detection unit 66 detects the participants, and the voice direction detection unit 67, the code analysis unit 68, or the device recognition unit 69 detects the device direction of the interactive whiteboard 2. The method for detecting the device direction will be described later.

[0193] S4: The first image generation unit 62 determines the height of the panoramic image 203 so as to include the detected participants and the electronic whiteboard 2, and generates the panoramic image 203 so as to include the standing participants and the electronic whiteboard 2. Note that if there is no electronic whiteboard 2 in the conference room, the first image generation unit 62 generates a panoramic image 203 that includes only the conference participants.

[0194] S5: Furthermore, the second image generating unit 63 generates a speaker image 204 from the panoramic image 203.

[0195] S6: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image 203 and the speaker image 204 to the terminal device 10. The terminal communication unit 61 also transmits to the terminal device 10 the audio data collected by the meeting device 60 or the mixed audio data described in FIG.

[0196] S7: The device communication unit 16 of the information recording application 41 receives the panoramic image 203, the speaker image 204, and the audio data, and the recording control unit 17 generates a combined image video. The display control unit 13 displays the combined image. Upon completion of recording, the recording control unit 17 transmits the combined image video (with audio data) to the storage service system 70, and the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data, thereby transmitting the text data to the storage service system 70. The combined image video and the text data are preferably associated with each other by the conference ID and saved under the same URL, etc.

[0197] <Example of determining the height of a panoramic image> 22 shows an example of the height of the panoramic image 203 that is determined when the face of a participant 120 is detected. The first image generating unit 62 sets a margin M1 for the face at the lowest position and a margin M2 for the face at the highest position, and determines the height of the panoramic image 203. The margins M1 and M2 may be set appropriately, but may be, for example, fixed values ​​or one to three times the height of the face at the highest or lowest position.

[0198] If no participants or the electronic whiteboard 2 are detected, the first image generating unit 62 generates the panoramic image 203 at a preset initial setting height.

[0199] <Determining the orientation of an electronic whiteboard in a panoramic image> Next, we will explain how to determine the orientation of the interactive whiteboard 2 in the panoramic image 203. There are mainly four methods for determining the orientation: 1. At the start of the conference, the user selects a panoramic image 203. 2. The electronic whiteboard 2 displays a specific image (such as a two-dimensional code), which is recognized by the terminal device 10 or the meeting device 60 from the panoramic image 203 captured by the imaging unit 601 of the meeting device 60. 3. The electronic whiteboard 2 outputs a specific sound, which the meeting device 60 recognizes through the microphone 608. 4. An arbitrary information processing device learns the shape of the electronic whiteboard 2 by machine learning, and the terminal device 10 or the meeting device 60 recognizes it from the panoramic image 203 captured by the camera (imaging unit 601) of the meeting device 60.

[0200] <<1. The user selects from the panoramic images at the start of the meeting.>> 23 is a diagram illustrating an operation method for setting the orientation of the interactive whiteboard 2 by pressing the position registration button 207. Pressing the position registration button 207 causes a panoramic image 203 to pop up and be displayed. For example, the user moves a rectangular window 206 on the panoramic image 203 using a pointing device such as a mouse or a touch panel. The user aligns the window 206 with the interactive whiteboard 2, a podium, or the like included in the panoramic image 203.

[0201] 24 shows a screen for confirming the direction set by the user. When the user presses the OK button 208, the direction of the interactive whiteboard 2 relative to the panoramic image 203 is set. The direction set by the user is transmitted to the meeting device 60 and stored by the first image generation unit 62 of the meeting device 60.

[0202] <<2. The interactive whiteboard 2 displays a specific image (such as a two-dimensional code), which is recognized by the terminal device 10 or the meeting device 60 from a panoramic image 203 captured by the imaging unit of the meeting device 60. 3. The interactive whiteboard 2 outputs a specific sound, which is recognized by the meeting device 60 using a microphone.>> 25 is a diagram illustrating a screen displayed by the interactive whiteboard 2 for setting a method for detecting the orientation of the interactive whiteboard 2. FIG. 25(a) shows an example of a menu screen 130. The menu screen 130 has a camera button 131. When the camera button 131 is pressed, a detection method setting window 132 is displayed.

[0203] 25(b) shows an example of the detection method setting window 132. The detection method setting window 132 has a two-dimensional code button 133 and a voice button 134. When the two-dimensional code button 133 is pressed, the electronic whiteboard 2 displays a two-dimensional code, and when the voice button 134 is pressed, the electronic whiteboard 2 outputs a specific voice.

[0204] <<Determining direction using a two-dimensional code>> 26 shows an example of a two-dimensional code 301 displayed as a specific image by the interactive whiteboard 2. In FIG. 26, the interactive whiteboard 2 appears in the panoramic image 203, and the interactive whiteboard 2 displays one two-dimensional code 301. The code analysis unit 68 detects the two-dimensional code 301 from the panoramic image 203, adds a height 302 of the two-dimensional code from the top end of the two-dimensional code 301 upward, and determines the top end of the panoramic image 203. Note that the two-dimensional code 301 includes device identification information of the interactive whiteboard 2, so that the interactive whiteboard 2 and the meeting device 60 are associated with each other.

[0205] <<Determining direction by sound>> Fig. 27 is a diagram illustrating a method for determining the direction of the electronic whiteboard 2 when the electronic whiteboard 2 outputs a specific sound. As shown in Fig. 27, speakers 450 are installed at the left and right ends of the electronic whiteboard 2. The speakers 450 may be built into the left and right ends.

[0206] The audio data generation unit 37 outputs sound from each speaker 450. The sound collection unit 64 automatically collects sound of a specific frequency. The audio direction detection unit 67 obtains a frequency spectrum by performing a Fourier transform on the audio data, and identifies two directions from which sound of a predetermined frequency above a threshold is coming. In this way, the audio direction detection unit 67 identifies the direction from which the sound emitted from each speaker 450 is coming relative to the meeting device 60. The audio direction detection unit 67 determines the center of the speaker 450, and determines the height of the panoramic image 203 as double the height 303 of the center.

[0207] FIG. 28 is an example of a sequence diagram illustrating a process in which the meeting device 60 generates a panoramic image 203 including the interactive whiteboard 2 using specific video and audio.

[0208] S21: The user presses the two-dimensional code button 133 or the voice button 134 in the detection method setting window 132. The operation reception unit 12 receives the press.

[0209] S22: The code generation unit 35 of the interactive whiteboard 2 generates a two-dimensional code as a specific image. The display control unit 34 displays the two-dimensional code on the display 480. In addition, the audio data generation unit 37 of the interactive whiteboard 2 outputs a sound of a specific frequency from the speaker 450. Either the code generation unit 35 or the audio data generation unit 37 needs to operate, but both may also operate.

[0210] S23: The meeting device 60 repeatedly captures images of its surroundings, and if a two-dimensional code is within its field of view, the code analysis unit 68 detects the two-dimensional code. The code analysis unit 68 notifies the first image generation unit 62 of the position of the two-dimensional code. Furthermore, the sound collection unit 64 of the meeting device 60 repeatedly collects sound, and automatically collects sound of a specific frequency. The sound direction detection unit 67 obtains a frequency spectrum by performing a Fourier transform on the sound data, and identifies two directions from which sound of a predetermined frequency above a threshold is coming. The sound direction detection unit 67 converts the direction of the speaker of the interactive whiteboard 2 (latitude and longitude in the omnidirectional image) into a position in the panoramic image, and notifies the first image generation unit 62 of this. It is preferable that this sound be in the ultrasonic frequency band, as it is inaudible to the user.

[0211] S24: The first image generation unit 62 determines the height of the panoramic image 203 based on the two-dimensional code, or determines the height of the panoramic image 203 based on the direction of the speaker 450 of the interactive whiteboard 2. The first image generation unit 62 generates the panoramic image 203 at the determined height from the spherical image.

[0212] S25: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image 203, the speaker image 204, and the audio data to the terminal device 10.

[0213] S26: The device communication unit 16 of the information recording application 41 receives the panoramic image 203, the speaker image 204, and the audio data. The recording control unit 17 combines the panoramic image 203 and the speaker image 204 to generate a combined image video. The display control unit 13 displays the combined image.

[0214] <<4. An arbitrary information processing device learns the shape of the electronic whiteboard through machine learning, and the terminal device or meeting device recognizes it from a panoramic image captured by the imaging unit of the meeting device.>> 29 shows an automatic detection setting screen 140 for the electronic whiteboard 2 displayed by the information recording application 41. The automatic detection setting screen 140 has a manufacturer's machine number field 141, an operation sound toggle button 142, and an automatic detection toggle button 143. The manufacturer's machine number field 141 displays the machine number transmitted by the meeting device 60. The operation sound toggle button 142 is a button that notifies the information recording application 41 that the operation has been accepted by a user through a sound when the user operates the information recording application 41. The automatic detection toggle button 143 is a button that causes the meeting device 60 to automatically detect the electronic whiteboard 2. When the user presses the automatic detection toggle button 143, the information recording application 41 requests the meeting device 60 to automatically detect the electronic whiteboard 2. The meeting device 60 detects the electronic whiteboard 2 from the spherical image.

[0215] 30 is a diagram illustrating the electronic whiteboard 2 detected by image processing such as machine learning. The device recognition unit 69 detects the shape (circumscribed rectangle) 241 of the electronic whiteboard 2 from the spherical image using machine learning or the like.

[0216] 31 is a diagram illustrating the height of the panoramic image 203 that is determined based on the electronic whiteboard 2 detected by image processing. For example, when the electronic whiteboard 2 is detected from the spherical image, the first image generation unit 62 determines the height of the panoramic image 203 to be the height from the top edge of the electronic whiteboard 2 to half the height H of the electronic whiteboard 2. The height from the top edge of the electronic whiteboard 2 to the top edge of the panoramic image 203 may be zero or 1 / 3 to 1 / 4 of the height H of the electronic whiteboard 2, and half the height H of the electronic whiteboard 2 is merely an example.

[0217] <Example of generating panoramic images, etc.> FIG. 32 is a diagram illustrating a method for generating a panoramic image 203 from a spherical image. The first image generation unit 62 horizontally crops a panoramic image from the spherical image X so that the participants 120 and the electronic whiteboard 2 are included. Note that the spherical image X has a three-dimensional structure and is distorted when represented on a plane as in FIG. 32 , but is simplified in FIG. 32 for ease of understanding. The first image generation unit 62 changes a° and b° shown in FIG. 6( b) so that the participants 120 and the electronic whiteboard 2 are included. The cropping range in the horizontal direction may be 360°, but as will be described later, it is preferable to crop the panoramic image 203 horizontally as well so that the participants 120 and the electronic whiteboard 2 are included. In this way, the height h and width w of the panoramic image 203 are variable.

[0218] When a participant 120 or the electronic whiteboard 2 that has been detected becomes undetectable for a certain period of time, the first image generating unit 62 adjusts the range of the panoramic image 203 again so that the participant 120 and the electronic whiteboard 2 are included.

[0219] Furthermore, the second image generating unit 63 generates a speaker image 204 by cutting it out from the first image generating unit 62. In Fig. 32, a speaker image 204 including Mr. A and Mr. D is generated.

[0220] 33 is an example of a combined image displayed by the information recording application 41. A panoramic image 203 is displayed at the top of the combined image, and speaker images 204 are displayed below the panoramic image. The arrangement and the number of speaker images 204 are just examples.

[0221] FIG. 34 is an example of a flowchart illustrating a process in which the first image generating unit 62 determines the height of the panoramic image 203.

[0222] During the meeting, the meeting device 60 repeatedly captures spherical images. The participant detection unit 66 of the meeting device 60 detects the participants 120 by performing face recognition or the like on the spherical image X (S201).

[0223] If no participant 120 is detected (No in S202), the electronic whiteboard 2 does not display any objects (is not operated), and the first image generation unit 62 generates a panoramic image 203 at the default height (S206).

[0224] If a participant 120 is detected (Yes in S202), it is determined whether the voice direction detection unit 67, the code analysis unit 68, or the device recognition unit 69 of the meeting device 60 has detected the electronic whiteboard 2 from the spherical image X (S203).

[0225] It is also possible for the operation detection unit 38 to determine whether an operation on the electronic whiteboard 2 has been detected. The communication unit 36 ​​of the electronic whiteboard 2 constantly transmits information on the presence or absence of an operation to the terminal device 10. If the terminal device 10 and the electronic whiteboard 2 are within the same LAN, the terminal device 10 and the electronic whiteboard 2 can communicate with each other if the IP address (for example, included in a two-dimensional code) of the electronic whiteboard 2 is known. Furthermore, since the terminal device 10 and the electronic whiteboard 2 are participating in the same conference, the information processing system 50 may refer to the association information and transmit information on the presence or absence of an operation to the terminal device 10 based on the conference ID. In this way, only when the electronic whiteboard 2 is operated, the first image generation unit 62 can determine the height so that the electronic whiteboard 2 is included in the panoramic image 203.

[0226] If the electronic whiteboard 2 is detected (Yes in S203), the first image generation unit 62 generates a panoramic image 203 with a height that includes the electronic whiteboard 2 and all of the participants 120 (S204). For example, the first image generation unit 62 adopts the height of the panoramic image 203 determined based on the electronic whiteboard 2 or the height of the panoramic image 203 determined based on the participants 120, whichever is higher.

[0227] If the electronic whiteboard 2 is not detected (No in S203), the first image generating unit 62 generates a panoramic image 203 having a height that includes all of the participants 120 (S205).

[0228] In this way, the first image generating unit 62 can generate a panoramic image 203 that includes all of the faces of the participants 120, or, if the interactive whiteboard 2 is being operated, includes the interactive whiteboard 2 and all of the participants 120.

[0229] <Centering the electronic whiteboard> FIG. 35 shows the electronic whiteboard 2 placed at the center of the panoramic image. FIG. 35(a) shows a panoramic image 203 with the front of the meeting device 60 at the center. As shown in FIG. 35(b), when the electronic whiteboard 2 is detected, the first image generation unit 62 places the electronic whiteboard 2 at the center (width direction) of the panoramic image 203. The first image generation unit 62 moves the electronic whiteboard 2 from right to left to the center, and combines the image on the left edge with the image on the right edge by the amount of movement (the movement direction may be reversed). Since the electronic whiteboard 2 is placed at the center of the panoramic image 203, it is easier for the user to check the content displayed on the electronic whiteboard 2.

[0230] <Example of panoramic image display> Next, the function of the button 219 for fixing the display range will be described with reference to Fig. 36 and Fig. 37. Fig. 36 shows an example of a panoramic image 203 generated when the button 219 for fixing the display range is off. In Fig. 36, one panoramic image 203 and two speaker images 204 are displayed arranged on one screen. The number of speaker images 204 is just an example, and there may be no speaker images 204 or there may be three or more speaker images 204.

[0231] 36(a) shows a panoramic image 203 in which all of the participants 120 are seated. The height of the panoramic image 203 at this time is L1, and the height of the speaker image is L2.

[0232] Next, Fig. 36(b) shows the panoramic image 203 when a participant 120 stands up. The first image generation unit 62 increases the height of the panoramic image 203 so that the faces of all participants 120 are included. The height of the panoramic image 203 in Fig. 36(b) is M1, and the height of the speaker image 204 is M2. The display control unit 13 increases the size of the area (an example of the first image display unit) in which the panoramic image 203 is displayed on the information recording application 41 side to a size equal to or larger than the panoramic image 203 so that the participants 120 are included.

[0233] On the other hand, if the overall size of the combined image displayed by the information recording application 41 is fixed, the second image generation unit 63 changes the height of the speaker image 204 in accordance with the height of the panoramic image 203. That is, if the height of the combined image is L1+L2 and the height of the panoramic image 203 is M1, the height of the speaker image 204 is L1+L2-M1=M2. The second image generation unit 63 may simply trim the speaker image 204 so that the height is shorter, but may also trim the width so that the aspect ratio of the speaker image 204 remains constant. The second image generation unit 63 may reduce the size of the speaker image 204.

[0234] Therefore, the following relationship exists between L1, L2, M1, and M2: L1<M1 、L2> M2

[0235] Next, Figure 36(c) shows a panoramic image 203 created to include the electronic whiteboard 2. The first image generation unit 62 increases the height of the panoramic image 203 so that the faces of all participants 120 and the electronic whiteboard 2 are included. The first image generation unit 62 detects the faces of each participant 120 and the electronic whiteboard 2, and determines the height so that all faces and the electronic whiteboard 2 are included. The speaker image 204 is the same as in Figure 36(b). The height of the panoramic image 203 in Figure 36(c) is N1, and the height of the speaker image is N2. Therefore, the following relationship exists between L1, L2, N1, and N2. L1<N1 、L2> N2

[0236] In this way, when the button 219 for fixing the display range is turned off, the information recording application 41 can display the panoramic image 203 in a large size.

[0237] Fig. 37 shows an example of a panoramic image 203 generated when the button 219 for fixing the display range is on. Fig. 37(a) is similar to Fig. 36(a).

[0238] 37(b) shows the panoramic image 203 when the participants 120 stand up. The first image generation unit 62 increases the height of the panoramic image 203 so that it includes the faces of all of the participants 120. Then, the first image generation unit 62 reduces the panoramic image 203 while maintaining the aspect ratio after changing the height of the panoramic image 203 so that the entire panoramic image 203 fits within the height L1.

[0239] Therefore, the height of the panoramic image 203 in Fig. 37(b) remains unchanged from that in Fig. 37(a), so L1 is constant. The height L2 of the speaker image 204 is also constant. Note that this reduction may be performed by the information recording application 41.

[0240] 37(c) shows a panoramic image 203 created to include the electronic whiteboard 2. The first image generation unit 62 increases the height of the panoramic image 203 so that it includes the faces of all of the participants 120 and the electronic whiteboard 2. Then, the first image generation unit 62 reduces the panoramic image 203 while maintaining the aspect ratio of the panoramic image 203 after the height change so that the entire panoramic image 203 fits within the height L1.

[0241] Therefore, the height of the panoramic image 203 in Fig. 37(c) remains unchanged from that in Fig. 37(a), so L1 is constant. The height L2 of the speaker image 204 is also constant. Note that this reduction may be performed by the information recording application 41.

[0242] In this way, when the button 219 for fixing the display range is on, the size of the panoramic image 203 in the information recording application 41 can be kept constant.

[0243] The processes in FIGS. 36 and 37 may be performed by the terminal device 10.

[0244] <<Generating a panoramic image depending on whether the button to fix the display range is on or off>> FIG. 38 is an example of a flowchart illustrating the process of the first image generating unit 62 generating the panoramic image 203 when the button 219 for fixing the display range is on or off.

[0245] From the start of a conference (S101) until the end of the conference (S102), the participant detection unit 66 detects the participants 120 from the omnidirectional image, and the voice direction detection unit 67, the code analysis unit 68, or the device recognition unit 69 detects the electronic whiteboard 2 (S103).

[0246] As described with reference to FIG. 34, the first image generating unit 62 changes the height of the panoramic image 203 so that the faces of the participants 120 and the interactive whiteboard 2 are included, and generates the panoramic image 203 (S104).

[0247] The first image generating unit 62 determines whether the button 219 for fixing the display range in FIG. 17 is on (S105).

[0248] If the button 219 for fixing the display range is off (No in S105), the second image generating unit 63 changes the height of the speaker image 204 according to the height of the panoramic image 203 (S107).

[0249] When the button 219 for fixing the display range is on (Yes in S105), the first image generation unit 62 generates the panoramic image 203 so that it includes the faces of the participants 120 and the electronic whiteboard 2, just as when it is off. However, the first image generation unit 62 reduces the height and width of the panoramic image 203 so that the height of the panoramic image 203 is the same as the initial height, while maintaining the aspect ratio of the panoramic image 203 after the height change (S106). In this way, the panoramic image 203 can be generated so that it includes the faces of the participants 120 and the electronic whiteboard 2 without changing the size of the display area of ​​the panoramic image 203 in the combined image. Note that the second image generation unit 63 does not need to crop the speaker image 204.

[0250] The terminal communication unit 61 of the meeting device 60 transmits the panoramic image 203, the speaker image 204, and the audio data to the terminal device 10 (S108).

[0251] <Determining the width of the panoramic image> In the above embodiment, the height of the panoramic image 203 is determined so as to include the participants 120, the electronic whiteboard 2, etc., but if the panoramic image 203 created by the meeting device 60 is only a part of the 360° horizontal direction, the same inconvenience as in the height direction occurs.

[0252] Fig. 39 shows a panoramic image 203 in which only a portion in the horizontal direction is cropped. The meeting device 60 is originally capable of capturing images in a 360° range in the horizontal direction. However, in order to reduce the image processing performed by the meeting device 60 and the processing load when transmitting to the terminal device 10, the meeting device 60 can also generate only a portion in the horizontal direction (for example, approximately 180° to 200° including the front) as the panoramic image 203. As shown in Fig. 39(a), in a meeting with a small number of participants, all of the participants 120 are included in this partial horizontal panoramic image 203.

[0253] However, as shown in Fig. 39(b), in a conference with a large number of participants, not all participants 120 are included in the horizontal portion of the panoramic image 203. In Fig. 39(b), participants 120 are present in the shaded area 250, but are not included in the panoramic image 203. The information recording application 41 cannot display participants 120 that are not included in the panoramic image 203.

[0254] Therefore, when a participant 120 or the electronic whiteboard 2 is detected, the first image generation unit 62 determines the width of the panoramic image 203 so that all participants 120 and the electronic whiteboard 2 are included in the horizontal direction. For example, the first image generation unit 62 determines the width of the panoramic image 203 by setting a margin for one or two faces for the faces of the participants 120 at the left and right ends in the horizontal direction.

[0255] 39(c), the first image generation unit 62 can generate a panoramic image 203 that includes all of the participants 120 and the interactive whiteboard 2 in the horizontal direction. Furthermore, when there are few participants 120, the processing load on the meeting device 60 can be reduced.

[0256] Next, a case where there is a gap between participants in the panoramic image 203 will be described with reference to Fig. 40. Fig. 40 is a diagram illustrating a process of omitting excess space when there is a gap between participants in the panoramic image 203. Whether the width of the panoramic image 203 is fixed or variable as in Fig. 39, if the participants 120 are seated with gaps between them, there will be a gap between the participants in the panoramic image 203.

[0257] When the distance between the participants or between the participant 120 and the electronic whiteboard 2 is equal to or greater than a threshold, the first image generation unit 62 omits the excess space 251 between the participants or between the participant 120 and the electronic whiteboard 2. Omission means deleting the panoramic image 203 in the excess space 251. In FIG. 40(a), the two spaces D are equal to or greater than the threshold. In this case, the first image generation unit 62 deletes the panoramic image 203 in the remaining excess space 251, leaving a margin for one or two faces of the participant 120 in the space D, and then connects the cut panoramic image 203.

[0258] 40(b) shows the panoramic image 203 from which the extra space 251 has been omitted. By omitting the extra space 251, the panoramic image 203 is less wasted and the visibility of the participants 120 is improved. Note that the first image generating unit 62 may delete a portion of the panoramic image 203 in the horizontal direction so that the distance D between the participants or between the participant 120 and the electronic whiteboard 2 is equal to or greater than a threshold, rather than determining whether the distance D between the participants or between the participant 120 and the electronic whiteboard 2 is equal to or greater than a threshold.

[0259] Note that omission also includes shortening 1 [m] to 0.5 [m].

[0260] <Saving combined image video> Next, the process of saving the combined image moving image will be described with reference to Fig. 41. Fig. 41 is an example of a sequence diagram showing the procedure in which the information recording application 41 records the panoramic image 203, the speaker image, and the application screen.

[0261] S51: The user operates the teleconferencing application 42 to start a teleconferencing. Here, it is assumed that the teleconferencing application 42 at the home base 102 and the other base 101 have started a teleconferencing. The teleconferencing application 42 at the home base 102 transmits images captured by a camera and audio collected by a microphone of the terminal device 10 to the teleconferencing application 42 at the other base 101. The teleconferencing application 42 at the other base 101 displays the received images on a display and outputs the received audio from a speaker. Similarly, the teleconferencing application 42 at the other base 101 transmits images captured by a camera and audio collected by a microphone of the terminal device 10 to the teleconferencing application 42 at the home base 102. The teleconferencing application 42 at the home base 102 displays the received images on a display and outputs the received audio from a speaker. Each teleconferencing application 42 repeats this process to realize the teleconferencing.

[0262] S52: The user makes settings related to recording on the recording setting screen 210 of the information recording application 41 shown in Fig. 14. The operation accepting unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that the camera toggle button 211 and the PC screen toggle button 212 are both on.

[0263] If the user has reserved a remote conference in advance, the user can display a list of remote conferences by pressing the information acquisition button 221 on the calendar in Fig. 19 and select a remote conference to associate with the recorded video. Since the user has already logged in to the information processing system 50, the information processing system 50 identifies remote conferences that the logged-in user has permission to view. The information processing system 50 transmits a list of the identified remote conferences to the terminal device 10, and the user selects a remote conference that is currently being held or will be held in the future. This determines information about the remote conference, such as the conference ID.

[0264] In addition, even if the user has not booked a remote conference in advance, the user can create a conference when creating a combined image video. The following describes a case where the information recording application 41 creates a conference when creating a combined image video and acquires a conference ID from the information processing system 50.

[0265] S53: The user instructs the information recording application 41 to start recording (start recording now button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0266] S54: Since the remote conference has not been selected (the conference ID has not been determined), the communication unit 11 of the information recording application 41 transmits a remote conference creation request to the information processing system 50.

[0267] S55: The communication unit 51 of the information processing system 50 receives the remote conference creation request, the communication management unit 54 acquires a unique conference ID assigned by the conference management system 9, and the communication unit 51 transmits the conference ID to the information recording application 41.

[0268] S56: Furthermore, the communication management unit 54 transmits the destination where the combined image moving image is to be saved (the URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

[0269] S57: The communication unit 11 of the information recording application 41 receives the conference ID and the destination to save the recorded file, and transmits the conference ID to the electronic whiteboard 2. The communication unit 11 may transmit the conference ID via the information processing system 50 or directly.

[0270] S58: When the communication unit 11 of the information recording application 41 receives the conference ID and the destination to save the recorded file, the recording control unit 17 determines that preparations for recording are complete and starts recording.

[0271] S59: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In FIG. 41, the application selected by the user is the remote conference application 42.

[0272] S60: The recording control unit 17 of the information recording application 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. When notifying, the recording control unit 17 may also notify that the camera toggle button 211 is on (requesting the panoramic image 203 and the speaker image). Regardless of whether a request is made, the meeting device 60 sends the panoramic image 203 and the speaker image to the information recording application 41.

[0273] S61: When the terminal communication unit 61 of the meeting device 60 receives a recording start signal, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or may be obtained from the information processing system 50.

[0274] S62: The voice acquisition unit 15 of the information recording application 41 acquires the voice data output by the terminal device 10 (voice data received by the remote conference application 42).

[0275] S63: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and a synthesis request to the meeting device 60.

[0276] S64: The terminal communication unit 61 of the meeting device 60 receives the voice data and the synthesis request, and the voice synthesis unit 65 synthesizes the received voice data with the surrounding voice data collected by the sound collection unit 64. For example, the voice synthesis unit 65 adds the two pieces of voice data together. Because clear voices around the meeting device 60 are recorded, the accuracy of converting voices around the meeting device 60 (conference room side) into text is improved.

[0277] This audio synthesis can also be performed by the terminal device 10. However, by distributing the recording function to the terminal device 10 and the audio processing to the meeting device 60, the load on the terminal device 10 and the meeting device 60 can be reduced. The recording function may be distributed to the meeting device 60 and the audio processing may be distributed to the terminal device 10.

[0278] S65: The first image generation unit 62 of the meeting device 60 generates a panoramic image 203, and the second image generation unit 63 generates a speaker image. In this step S65, the height of the panoramic image 203 described in this embodiment is determined.

[0279] S66: The device communication unit 16 of the information recording application 41 repeatedly acquires the panoramic image 203 and the speaker image from the meeting device 60. The device communication unit 16 also repeatedly requests and acquires the synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making a request to the meeting device 60. Alternatively, the meeting device 60 may automatically transmit the panoramic image 203 and the speaker image upon receiving a request to synthesize audio data. The meeting device 60 may automatically transmit the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0280] S67: The recording control unit 17 of the information recording application 41 creates a combined image by arranging the application screen acquired from the remote conference application 42, the panoramic image 203, and the speaker image. The recording control unit 17 repeatedly creates combined images and creates a combined image video by specifying each combined image as a frame that makes up the video. The recording control unit 17 also saves the audio data received from the meeting device 60.

[0281] The information recording application 41 repeats the above steps S62 to S67.

[0282] S68: When the remote conference ends and recording is no longer necessary, the user instructs the information recording application 41 to end recording (for example, by pressing the recording end button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0283] S69: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. The meeting device 60 continues creating the panoramic image 203 and speaker images and synthesizing audio. However, the meeting device 60 may change the processing options, such as changing the resolution or fps, depending on whether recording is in progress.

[0284] S70: The recording control unit 17 of the information recording application 41 combines the audio data with the combined image moving image to create a combined image moving image with audio.

[0285] S71: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the recording setting screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. In detail, the audio data processing unit 18 specifies the URL of the save destination via the communication unit 11, and transmits a request to the information processing system 50 to convert the audio data combined into the combined image video together with the conference ID and recording ID.

[0286] S72: The communication unit 51 of the information processing system 50 receives a request to convert the voice data, and the text conversion unit 56 converts the voice data into text data using the voice recognition service system 80. The communication unit 51 saves the text data in the same storage destination (the URL of the storage service system 70) as the storage destination of the combined image video. In the recorded video information storage unit 5002, the text data is associated with the combined image video by the conference ID and the recorded video ID. The text data may be managed by the conference management unit 54 of the information processing system 50 and stored in the storage unit 5000. Alternatively, the terminal device 10 may request voice recognition from the voice recognition service system 80 and store the text data acquired from the voice recognition service system 80 in a storage destination. The voice recognition service system 80 returns the converted text data to the information processing system 50, or may directly transmit the text data to the URL of the storage destination. The voice recognition service system 80 may select or switch between multiple services depending on setting information set by the user in the information processing system 50.

[0287] S73: The upload unit 20 of the information recording application 41 also saves the combined image moving image in the destination for saving the combined image moving image via the communication unit 11. The combined image moving image is associated with the conference ID and the recording ID in the recording information storage unit 5002. The combined image moving image is recorded as having been uploaded.

[0288] S74: The user inputs the end of the meeting into the electronic whiteboard 2. The user may input the end of the meeting into the terminal device 10, and the end of the meeting may be transmitted from the terminal device 10 to the electronic whiteboard 2. In this case, the end of the meeting may be transmitted to the electronic whiteboard 2 via the information processing system 50.

[0289] S75: The communication unit 36 ​​of the electronic whiteboard 2 specifies the conference ID and transmits object data displayed (for example, handwritten) during the conference to the information processing system 50. The communication unit 36 ​​may also transmit device identification information of the electronic whiteboard 2 to the information processing system 50. In this case, the conference ID is identified by the association information.

[0290] S76: The information processing system 50 stores the object data in the same storage location as the combined image video, etc., based on the conference ID.

[0291] The user is notified of the save destination, so the user can share the combined image video with participants 120 by informing them of the save destination by email or the like. Even if the combined image video, audio data, text data, and object data are created using different devices, they can be collected and stored in a single storage location and can be easily viewed by users etc. later.

[0292] The processing of steps S62 to S67 does not have to be performed in the order shown in FIG. 41, and the synthesis of the audio data and the creation of the combined image may be performed in the reverse order.

[0293] <Major Effects> As described above, the meeting device 60 of this embodiment detects multiple pre-set objects (the faces of participants 120 and devices such as the electronic whiteboard 2) and determines the height and width of the panoramic image 203 so that the objects are included, thereby generating a panoramic image 203 that includes the objects.

[0294] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0295] For example, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. The meeting device 60 may also be an omnidirectional camera, a microphone, and a speaker connected by a cable.

[0296] Furthermore, a meeting device 60 may also be placed at another location 101. The other location 101 separately uses the meeting device 60 to create combined image video and text data. Furthermore, multiple meeting devices 60 may be placed at one location. In this case, multiple pieces of record information are created for each meeting device 60.

[0297] The layout of the panoramic image 203, the speaker image 204, and the app screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be on the bottom and the speaker image 204 on top, or the user may change the layout, or may individually switch between displaying and hiding the panoramic image 203 and the speaker image 204 during playback.

[0298] Furthermore, the configuration examples in Figure 9 and the like are divided according to main functions to facilitate understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the terminal device 10, the meeting device 60, and the information processing system 50 can be divided into even more processing units depending on the processing content. Furthermore, the processing can also be divided so that one processing unit includes even more processes.

[0299] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, information processing system 50 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0300] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in FIG. 21, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Furthermore, the information processing system 50 may be integrated into a single server device, or may be divided into multiple devices.

[0301] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a conventional circuit module designed to perform each of the above-described functions. [Explanation of symbols]

[0302] 10 Terminal Equipment 50 Information Processing Systems 60 Meeting Devices 100 Recording Information Creation System [Prior art documents] [Patent documents]

[0303] [Patent Document 1] JP 2015-19162 A

Claims

1. An image generation unit that generates a first image to be displayed in a first display area and generates second images to be displayed in each of a plurality of second display areas based on the first image; a combined image generating unit that generates a combined image in which the first image and the second image are arranged adjacent to each other in a height direction; a change unit that changes a ratio between a height of the first display area and a height of the plurality of second display areas from a first ratio to a second ratio different from the first ratio; An information processing device comprising:

2. The information processing device described in Claim 1, characterized in that the first image is a panoramic image generated from a wide-angle image captured by an imaging device.

3. An information processing device as described in Claim 1, characterized in that the first image includes multiple people.

4. An information processing device as described in Claim 1, characterized in that the second image is generated based on detected sound information.

5. An information processing device as described in Claim 3, characterized in that at least one of the second images includes a person speaking among multiple people included in the first image.

6. An information processing device as described in Claim 3, characterized in that the ratio between the height of the first display area and the heights of the multiple second display areas is changed from the first ratio to the second ratio based on the vertical movement of at least one of the multiple people included in the first image.

7. An information processing device as described in claim 1, characterized in that the first image is displayed above the second image in the height direction.

8. An information processing device as described in Claim 1, characterized in that the height of the second display area in the second ratio is smaller than the height of the second display area in the first ratio.

9. An information processing device as described in Claim 1, characterized in that the first image includes an electronic device connected to the information processing device via a network.

10. 2. The information processing apparatus according to claim 1, wherein the plurality of second images are arranged adjacent to each other in the horizontal direction.

11. An information processing device as described in Claim 1, characterized in that when the ratio between the height of the first display area and the heights of the multiple second display areas is changed from the first ratio to the second ratio, the horizontal sizes of the first display area and the multiple second display areas are maintained.

12. The information processing device according to claim 1, characterized in that it is provided with a display unit and displays the combined image on the display unit.

13. The information processing device described in Claim 1, characterized in that the information processing device is connected to a server device via a network and transmits the combined image to the server device.

14. An information processing method performed by an information processing device, comprising: an image generation unit generating a first image to be displayed in a first display area, and generating second images to be displayed in each of a plurality of second display areas based on the first image; a process in which a combined image generating unit generates a combined image in which the first image and the second image are arranged adjacent to each other in a height direction; a process in which a change unit changes a ratio between a height of the first display area and a height of the plurality of second display areas from a first ratio to a second ratio different from the first ratio; An information processing method comprising:

15. An information processing system having an information processing device, an image generation unit that generates a first image to be displayed in a first display area and generates second images to be displayed in each of a plurality of second display areas based on the first image; a combined image generating unit that generates a combined image in which the first image and the second image are arranged adjacent to each other in a height direction; a change unit that changes a ratio between a height of the first display area and a height of the plurality of second display areas from a first ratio to a second ratio different from the first ratio; An information processing system comprising:

16. A detection unit that detects one or more objects from a wide-angle image, including participants of a remote conference; If the plurality of objects are detected from the wide-angle image, a first image including the plurality of objects is generated; an image generating unit that generates a second image of one or more speaking participants of the remote conference by cutting out the second image from the first image; a display control unit that displays the first image in a first display area of a screen and the second image in a second display area of the screen adjacent to each other to display a combined image; The information processing device is characterized in that the image generation unit changes the ratio between the height of the first display area for the first image and the height of the second display area for the second image in the vertical direction on the screen so that the first image includes the multiple objects while maintaining the size of the display area of the combined image in the vertical direction on the screen.

17. The information processing device described in Claim 16, characterized in that the image generation unit changes the size of the first display area for the first image so that the first image is displayed in the display area.

18. An information processing device as described in Claim 17, characterized in that the size of the first display area for the first image changed by the image generation unit is the size in the vertical direction.

19. The information processing device described in Claim 16, characterized in that the image generation unit generates the first image in which one of the multiple objects is positioned at the horizontal center.

20. An information processing device as described in Claim 16, characterized in that if the plurality of objects set in advance are not included in the first image, the image generation unit increases the width of the first image so that the plurality of objects are included.

21. An information processing device as described in Claim 16, characterized in that when the distance between a first object and a second object among the multiple objects is equal to or greater than a threshold value, the image generation unit generates the first image in which the excess distance between the first object and the second object is omitted.

22. The information processing device according to claim 16, characterized in that the plurality of objects include a human face.

23. The information processing device according to claim 16, characterized in that the plurality of objects include electronic devices.

24. Detecting a two-dimensional code displayed by the electronic device; 24. The information processing apparatus according to claim 23, wherein the image generating unit generates the first image including the electronic device based on the detected two-dimensional code.

25. Collecting sound output by the electronic device, Detecting the direction of the collected sound, 24. The information processing apparatus according to claim 23, wherein the image generating unit generates the first image including the electronic device based on the detected direction of the electronic device.

26. The image generation unit recognizes the electronic device by image processing, 24. The information processing apparatus according to claim 23, wherein the first image is generated so as to include the recognized electronic device.

27. The information processing device described in Claim 16, characterized in that the display control unit displays a display range fixation button for fixing a first display area of the first image.

28. When the display range fix button is on, a first display area of the first image in the combined image is set to a fixed value; 28. The information processing apparatus according to claim 27, wherein when the display range fixing button is turned off, the first display area of the first image in the combined image is not set to a fixed value.

29. An apparatus using machine learning to learn the shape of each of the plurality of objects; The information processing apparatus according to claim 16 , wherein the detection unit detects the plurality of objects whose shapes have been learned.

30. The information processing device described in Claim 16, characterized in that the detection unit performs face detection from the wide-angle image centered on the direction of the sound, and detects the speaking participant by cutting out 15° to the left and right from the center of the face.

31. The information processing device described in Claim 16, characterized in that the number of imaging devices that capture the wide-angle image is one.

32. An information processing device as described in Claim 16, characterized in that the second image is a cut-out of at least one of the multiple objects contained in the first image.

33. An information processing device as described in Claim 16, characterized in that the height direction is the direction in which the multiple objects stand.

34. An information processing method performed by an information processing device, comprising: A process of detecting one or more objects from a wide-angle image, the objects including participants of the remote conference; If the plurality of objects are detected from the wide-angle image, a first image including the plurality of objects is generated; generating a second image of one or more speaking participants of the remote conference by cropping the second image from the first image; a process of displaying a combined image by arranging the first image to be displayed in a first display area of a screen and the second image to be displayed in a second display area of the screen adjacent to each other, An information processing method characterized by changing the ratio between the height of a first display area for the first image and the height of a second display area for the second image in the vertical direction on the screen so that the first image includes the multiple objects while maintaining the size of the display area of the combined image in the vertical direction on the screen.

Images