Imaging device, program, and panoramic image creation method

Abstract

To suppress the incidence of light from above, such as ceiling lighting, on an imaging device.SOLUTION: An imaging device capable of capturing an image with an angle of view of at least approximately 360° in the horizontal direction includes a plurality of lenses having an optical axis in the horizontal direction, and an upper canopy that blocks light that is incident on the plurality of lenses from directly above, and the upper canopy limits the angle of incidence of light between the direction in which light from above is incident on the lenses at the maximum angle and the optical axis of the lenses to between approximately 30° and approximately 67°.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present invention relates to an imaging device, a program, and a panoramic image creating method. [Background technology]

[0002] A remote conference service system is known in which images and audio are transmitted in real time from one location to one or more other locations, and users in remote locations hold a conference using images and audio. Also, an imaging device that captures images in all directions of 360° is known.

[0003] A technology has been devised for reducing discontinuity of color that may occur at seams when multiple images captured by an imaging device are combined (see, for example, Patent Document 1). Patent Document 1 discloses a technology that calculates a white balance evaluation value for each region constituting multiple captured images, calculates a brightness adjustment value and a white balance adjustment value based on the evaluation values ​​of regions that overlap between the captured images, and determines a smoothed balance adjustment value for each divided region by taking a weighted average of the balance adjustment values ​​for each divided region, thereby reducing discontinuity of color that may occur at seams when multiple captured images are combined. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional technology, there is a problem that light from above, such as ceiling lighting, enters the imaging device, degrading the image quality. For example, when flare or ghosting occurs in the image, the visibility of the faces of people participating in the meeting decreases. When the image quality decreases, a sense of incongruity appears at the joints of multiple images when generating a 360-degree image, or the images are misaligned. Another problem is that image processing to reduce these discontinuities that may occur at the joints requires a high processing load, making real-time processing difficult.

[0005] In view of the above problems, the present invention provides a technique for suppressing light from above, such as ceiling lighting, from entering an imaging device. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides an imaging device capable of capturing images with a field of view of at least 360° in the horizontal direction, comprising a plurality of lenses having an optical axis in the horizontal direction, and an upper canopy that blocks light that is incident on the plurality of lenses from directly above, wherein the upper canopy limits the angle of incidence of light between the direction in which light from above is incident on the lenses at the maximum angle and the optical axis of the lenses to between 30° and 67°. Effect of the Invention

[0007] The present invention can prevent light from above, such as ceiling lighting, from entering an imaging device. [Brief description of the drawings]

[0008] [Figure 1] 11A and 11B are diagrams for explaining an outline of the creation of record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surroundings. [Diagram 2] FIG. 1 is a diagram illustrating an example of the configuration of a recorded information display system. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing system and a terminal device. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a meeting device. [Diagram 5] FIG. 2 is a diagram illustrating an imaging range of a meeting device. [Figure 6] 1A and 1B are diagrams illustrating the cutting out of a panoramic image and a speaker image. [Figure 7] 1 is an example of a functional block diagram illustrating the functions of a terminal device, a meeting device, and an information processing system in a record information display system, divided into blocks. [Figure 8] 4 is a diagram showing an example of moving image recording information stored in an information storage unit; FIG. [Figure 9] 13 is an example of application-side conference information stored in an application-side conference information storage unit. [Figure 10]4 is a diagram showing recording information stored in a recording information storage unit. FIG. [Figure 11] 4 is an example of side conference information stored in a conference information storage unit. [Figure 12] 1 is a diagram illustrating a structure of text data stored in a storage service system. [Figure 13] 1 is a diagram showing an example of a camera image and a panoramic image; [Figure 14] FIG. 13 is a diagram illustrating the effect of light from the ceiling lighting in a conference room on the seam between two camera images. [Figure 15] This is a diagram explaining the adverse effect of ghosting caused by the light from the ceiling lighting in the conference room on the joint between two camera images. [Figure 16] FIG. 2 illustrates an example of a side view of a meeting device. [Figure 17] FIG. 4 is a diagram illustrating the configuration of the upper eaves. [Figure 18] 10A and 10B are diagrams illustrating the angle of incidence of light from a ceiling onto a lens. [Figure 19] FIG. 1 is a diagram illustrating the relationship between the incident angle and the amount of light in a typical conference room. [Figure 20] 1A and 1B are a perspective view and a top view of the upper eaves, respectively; [Figure 21] FIG. 13 is a diagram showing an example of a top view of an upper canopy having an elliptical shape when viewed from above. [Figure 22] 13 is an example of a top view in which the shape of the upper eave viewed from above is rectangular, and in which the shape of the upper eave viewed from above is octagonal. [Diagram 23] 13A and 13B are diagrams showing an example of a light blocking member provided on an upper eaves. [Figure 24] 13 is a diagram illustrating the shape of the upper eave as viewed from the side. FIG. [Diagram 25] 11A and 11B are diagrams showing an example of the shape of a side surface of an upper eave. [Figure 26] 11A and 11B are diagrams illustrating a display example in which the terminal device displays a panoramic image as a final output image. [Figure 27]FIG. 13 is a diagram showing an example of an initial screen displayed by an information recording application running on the terminal device after login. [Figure 28] 13A and 13B are diagrams illustrating an operation method when a fixed display button is on. [Figure 29] FIG. 13 is a diagram illustrating an example of an unrecognized device screen. [Diagram 30] FIG. 13 is a diagram showing an example of a recording setting screen displayed by an information recording application. [Diagram 31] 13 is a diagram showing an example of the display of the recorded content confirmation window when the camera toggle button is on and the PC screen toggle button is off. FIG. [Diagram 32] 13 is a diagram showing an example of the display of the recorded content confirmation window when the camera toggle button is on and the PC screen toggle button is on. FIG. [Diagram 33] FIG. 13 is a diagram showing an example of a recording screen displayed by the information recording application during recording. [Diagram 34] FIG. 13 is a diagram showing an example of a conference list screen displayed by an information recording application. [Diagram 35] FIG. 13 is a diagram showing an example of a recorded video display screen after selection of a recorded video to be displayed by the information recording application. [Diagram 36] FIG. 13 is a diagram showing an example of an editing screen for a recorded video. [Figure 37] 11 is a sequence diagram illustrating an example of a procedure in which an information recording application records a panoramic image, a speaker image, and an application screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A recorded information display system and a panoramic image creating method performed by the recorded information display system will be described below as an example of an embodiment of the present invention.

[0010] <An example of how to create minutes for a remote meeting> First, referring to FIG. 1, an overview of a method for creating minutes of a meeting using a panoramic image and an app screen will be described. FIG. 1 is a diagram for explaining an overview of the creation of recording information that saves the screen of an app executed during a remote meeting together with the surrounding panoramic image. As shown in FIG. 1, a user at the own base point 102 shown is holding a remote meeting with another base point 101 using the remote meeting service system 90.

[0011] The recording information display system 100 of the present embodiment creates recording information (minutes of a meeting) using a horizontal panoramic image (hereinafter referred to as a panoramic image) obtained by processing information obtained by imaging the surroundings by an imaging means capable of imaging 360° around and a meeting device 60 equipped with a microphone and a speaker, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. Regarding the audio, the recording information display system 100 synthesizes the audio received by the remote meeting application (hereinafter referred to as the remote meeting app 42) and the audio obtained by the meeting device 60 and includes it in the recording information. Note that in the present embodiment, it is also possible to only view and display without creating recording information. The overview will be described below.

[0012] (1) In the terminal device 10, an information recording app 41 and a remote meeting app 42, which will be described later, are operating. In addition, an app for displaying materials or the like may also be operating. The information recording app 41 transmits the audio output from the terminal device 10 (including the audio received from another base point by the remote meeting app 42) to the meeting device 60. The meeting device 60 mixes (synthesizes) the audio it has obtained and the audio of the remote meeting app 42.

[0013] (2) The meeting device 60 is equipped with a microphone and performs a process of cutting out a speaker from the panoramic image based on the direction from which the audio was obtained, and creates a speaker image (an example of speaker image information). The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 saves any application screen selected by the user (for example, the screen 103 of the remote conference application 42), the panoramic image 203, and the speaker image 204. Note that the application screen is an example of screen information (described later) displayed by each application such as the remote conference application. Since each of these is saved repeatedly, each is saved as a separate video. Hereinafter, when the application screen, the panoramic image 203, and the speaker image 204 are described without distinction, they are referred to as "recorded video."

[0015] (4) The information recording application 41 accepts editing operations (cutting out unnecessary parts by the user) and completes the recorded video. The recorded video constitutes a part of the recorded information.

[0016] (5) The information recording application 41 transmits the recorded video and audio data to the storage service system 70 and stores them.

[0017] (6) The information recording application 41 also transmits a request to convert the voice data into text to the information processing system 50. The information processing system 50 transmits the voice to the voice recognition service system 80, which converts the voice into text data, and converts the voice into text. The text data also includes data on how many minutes after the start of recording that the speech occurred.

[0018] In the case of real-time text conversion, the meeting device 60 transmits audio directly to the information processing system 50 .

[0019] (7) The information processing system 50 additionally stores the text data in the storage service system 70 that stores the recorded video. The text data constitutes a part of the recorded information.

[0020] The information processing system 50 can perform billing processing for the user according to the service used. For example, the charge is calculated based on the amount of text data, the file size of the recorded video, the processing time, and the like.

[0021] (8) The information recording application 41 can play back recorded video together with audio data. The information recording application 41 can also display text data corresponding to the recorded video. The arrangement of the panoramic image 203, the speaker image 204, and the screen 103 of the remote conference application 42 is merely an example, but for example, the panoramic image 203 and the speaker image 204 are arranged on the left side, and the screen 103 of the remote conference application 42 is arranged on the right side.

[0022] In this way, when the recorded video is displayed, a panoramic image of the surroundings including the user, an image of the speaker, and the screen of an application displayed during the remote conference, such as the remote conference application 42, are displayed. Therefore, when a participant of the remote conference or a non-participant views the recorded video as a meeting record, the panoramic image, the image of the speaker, and the screen of the application are displayed on one screen, so that the situation during the remote conference is reproduced with a sense of realism.

[0023] <Terminology> An application (app) is software developed and used for a specific function or purpose. Applications can be native apps or web apps. Web apps (cloud-side apps provided by cloud services) can also work in conjunction with native apps or web browsers.

[0024] The imaging device is a device that can capture images of the device's surroundings and collect sounds from the surroundings. In this embodiment, the imaging device is described using the term meeting device 60. The meeting device 60 may include a device that is used by connecting to a terminal device, a device that is built into a terminal device, and a device that is used by connecting to a cloud service without directly connecting to a terminal device.

[0025] The image of the surroundings of the meeting device acquired by the meeting device is image information acquired by the meeting device by capturing an image of the surrounding space (for example, an area or space of 180° to 360° in the horizontal direction) surrounding the meeting device, and refers to an image acquired by performing a predetermined process on the image information of the curved surface captured by the meeting device. The predetermined process is various processes for creating the surrounding image information from the captured information, such as flattening the captured image of the curved surface. The predetermined process may include a process for creating the surrounding image, a process for cutting out the speaker image, and a process for combining the surrounding image and the speaker image. In this embodiment, the surrounding image is described by the term panoramic image. A panoramic image is an image with a field angle of approximately 180° to 360° in the horizontal direction. A single meeting device does not need to capture a panoramic image, and multiple imaging devices with normal fields of view may be combined. The meeting device is intended to be installed and used on a table or other suitable location for meetings at a base or for grasping the surrounding situation, but it may also be used for surveillance (security, disaster prevention, etc.), monitoring (childcare, elderly care, etc.), and on-site situation analysis (solutions, marketing, etc.).

[0026] The recorded information is information recorded by the information recording application 41, and is stored and preserved in a viewable manner as information linked to the identification information of a certain conference (meeting), and includes, for example, the following information: Video information created based on the screen information displayed by the selected app (e.g., a remote conferencing app) and image information of the device's surroundings acquired by the device. During a meeting, audio information is acquired and synthesized between the remote conference app (terminal device) and the meeting device at the base. -Text information converted from the captured voice. Other data and images that are related information related to the meeting. Examples of other information include document files used during the meeting, added notes, translation data of text data, images and stroke data created using the cloud electronic whiteboard service during the meeting, etc.

[0027] When the information recording application 41 records the screen of the remote conferencing application or the situation of the meeting at the base, the recorded information may become the minutes of the meeting held. The minutes are an example of the recorded information, and the way of calling the recorded information changes according to the content of the remote meeting or the base side. For example, it may be called the recording of communication, the recording of the base situation, etc. In addition, the recorded information includes files in a plurality of formats, such as video files (combined image videos, etc.), audio files, text data (text data obtained by voice recognition of voice), document files, image files, table format files, etc. Since the files are associated with each other with respect to the identification information of the meeting, they can be browsed collectively or selectively in chronological order during browsing.

[0028] A tenant is a group of users (companies, local governments, some organizations thereof, etc.) that have contracted to receive services from a service provider. The creation of the recorded information and the conversion into text data in this embodiment are executed because the tenant has a contract with the service provider.

[0029] Remote communication means communicating through voice or video by utilizing software and terminal devices with a partner at a physically distant base. An example of remote communication is a remote meeting, and the meeting may be called an assembly, a meeting, a meeting, a consultation, an application for a contract, etc., a gathering, a meeting, a seminar, a training session, a study session, a seminar, a training session, etc.

[0030] A base means a place that serves as a place for activities. An example of a base is a meeting room. A meeting room is a room installed mainly for the purpose of using it for meetings. The base may be various places such as a home, a reception, a store, a warehouse, or an outdoor site, as long as it is a place or space where a terminal device, a device, etc. can be installed.

[0031] Voice is the language sound uttered by a human or the surrounding sound, etc., and voice data is the digitalization of voice, but in this embodiment, it will be described without strict distinction.

[0032] <System configuration example> Next, the system configuration of the record information display system 100 will be described with reference to Fig. 2. Fig. 2 shows an example of the configuration of the record information display system 100. Fig. 2 shows one base (own base 102) among multiple bases where a remote conference is held, and a terminal device 10 at the own base 102 communicates with an information processing system 50, a storage service system 70, and a remote conference service system 90 via a network. A meeting device 60 is also placed at the own base 102, and the terminal device 10 is connected to this meeting device 60 via a USB cable or the like so as to be able to communicate with each other.

[0033] At least an information recording application 41 and a remote conference application 42 run on the terminal device 10. The remote conference application 42 can communicate with terminal devices at other locations 101 via a remote conference service system 90 on the network, allowing users at each location to hold a conference from a remote location. The information recording application 41 creates record information for a remote conference held by the remote conference application 42, using the functions of an information processing system 50 and a meeting device 60.

[0034] In this embodiment, an example of creating record information during a remote conference will be described, but the conference does not have to be one in which a base communicates with a remote base. In other words, the conference may be one in which only participants at one base participate. In this case, only the voice collected by the meeting device 60 is saved without synthesis, and there is no change in the processing of the information recording application 41.

[0035] The terminal device 10 has a built-in camera with a normal angle of view (or may be external), which captures an image of the front including the user 107 who operates the terminal device 10. The normal angle of view is an image that is not a panoramic image, and in this embodiment, it is a planar image that is not curved such as a celestial sphere image. The terminal device 10 also has a built-in microphone (or may be external), which collects the surrounding sounds of the user who operates the terminal device 10. Therefore, the user can hold a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42 except for an increase in the processing load of the terminal device 10. The remote conference application 42 can also transmit the panoramic image and the speaker image captured by the meeting device 60 to the remote conference service system 90.

[0036] The information recording application 41 is an application that records information by creating record information through communication with the meeting device 60. The meeting device 60 is a meeting device equipped with a panoramic image capturing device, a microphone, and a speaker. The camera of the terminal device 10 can only capture a limited range in front of it, but the meeting device 60 can capture an image of the entire surroundings (not necessarily the entire periphery) surrounding the meeting device 60. The meeting device 60 can always include multiple participants 106 shown in FIG. 2 within its angle of view.

[0037] In addition, the meeting device 60 cuts out a speaker image from a panoramic image, synthesizes the sound acquired by the meeting device 60 with the sound output by the terminal device 10 (including the sound received by the remote conference application 42), and the like. The meeting device 60 may be placed anywhere in the home base 102, not limited to a desk or stand. Since the meeting device 60 can capture a celestial sphere image, it may be placed on the ceiling, for example. The meeting device 60 may also be installed on the other base side or at any base.

[0038] The information recording application 41 displays a list of applications running on the terminal device 10, saves the above-mentioned recorded information, displays recorded videos, accepts editing, etc. The information recording application 41 also displays a list of remote conferences that have been held or are scheduled to be held, etc. The list of remote conferences is used for information related to the recorded information, and allows the user to link remote conferences with the recorded information.

[0039] The remote conference application 42 is an application for a terminal device to perform remote communication with other terminal devices by establishing a communication connection with other terminal devices at other bases 101, transmitting and receiving images and audio, displaying images and outputting audio, etc. The remote conference application can also be called a remote communication application, a remote information common application, etc.

[0040] The information recording application 41 and the remote conference application 42 may be Web applications or native applications. A Web application is an application that performs processing by cooperation between a program on a Web server and a program on a Web browser or a native application, and does not need to be installed on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used. In this embodiment, both applications will be described as native applications.

[0041] The terminal device 10 may be, for example, a general-purpose information processing device equipped with a communication function, such as a PC (Personal Computer), a smartphone, a tablet terminal, etc. The terminal device 10 may also be an electronic whiteboard, a game machine, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, an industrial machine, a medical device, a network home appliance, etc. The terminal device 10 may be any device on which at least the information recording application 41 and the remote conference application 42 can run.

[0042] The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and a platform service. The server applications manage remote conference lists, manage recorded information recorded in remote conferences, and manage various settings and storage paths. The platform service performs user authentication, contracts, billing, and the like.

[0043] All or part of the functions of the information processing system 50 may exist in a cloud environment or an on-premise environment. The information processing system 50 may be configured with multiple server devices, or may be configured with one information processing device. For example, the server application and the platform service may be provided by different information processing devices, and further, an information processing device may exist for each function in the server application. The information processing system 50 may be integrated with the conference management system 9, storage service system 70, and voice recognition service system 80 described below.

[0044] The conference management system 9 is a system that manages information about conferences held in tenants that use the information processing system 50. The information processing system 50 acquires conference information from the conference management system 9, and manages the conference information in association with record information.

[0045] The storage service system 70 is a storage means on a network, and provides a storage service that accepts the storage of files, etc. Known examples of the storage service system 70 include One Drive (registered trademark), Google Workspace (registered trademark), DropBox (registered trademark), etc. The storage service system 70 may be an on-premise NAS (Network Attached Storage), etc.

[0046] The voice recognition service system 80 provides a service of performing voice recognition on voice data and converting it into text data. The voice recognition service system 80 may be a general-purpose commercial service, or may be part of the functions of the information processing system 50. In addition, the voice recognition service system 80 may be set to use a different service system for each user, tenant, or conference.

[0047] <Hardware configuration example> The hardware configuration of the information processing system 50 and the terminal device 10 according to this embodiment will be described with reference to FIG.

[0048] <<Information processing system and terminal device>> Fig. 3 is a diagram showing an example of a hardware configuration of an information processing system 50 and a terminal device 10 according to the present embodiment. As shown in Fig. 3, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0049] Among these, the CPU 501 controls the overall operation of the information processing system 50 and the terminal device 10. The ROM 502 stores programs such as an IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 3.

[0050] Moreover, the keyboard 511 is a type of input means having a plurality of keys used for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515 such as a flash memory.

[0051] <<Meeting Devices>> Using FIG. 4, the hardware configuration of the meeting device 60 will be described. FIG. 4 is an example of a hardware configuration diagram of the meeting device 60 capable of capturing a 360° video. Hereinafter, the meeting device 60 is an apparatus that uses an imaging device to capture a 360° video around the device at a predetermined height, but the number of imaging devices may be two or more. Also, it does not necessarily have to be a dedicated device, and a 360° video imaging unit may be attached to a PC, digital camera, smartphone, etc. to have substantially the same function.

[0052] As shown in FIG. 4, the meeting device 60 includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a speaker 619, an audio processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, and a voice sensor 618.

[0053] Among these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens, hereinafter sometimes simply referred to as lens) 602 having a 360° angle of view for forming a hemispherical image, and an imaging element 603 (image sensor) provided corresponding to each wide-angle lens. The imaging element 603 includes an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the lens 602 into image data of an electric signal and outputs it, a timing generation circuit that generates a horizontal or vertical synchronization signal or a pixel clock for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this imaging element are set. The imaging unit 601 may be a 360° camera, which is an example of an imaging means capable of capturing an image of 360° around the meeting device 60. A plurality of pieces of information acquired by a plurality of imaging elements (for example, 180°+180°) may be synthesized to obtain a 360° angle of view.

[0054] The imaging elements 603 (image sensors) of the imaging unit 601 are connected to the image processing unit 604 via a parallel I / F bus. Meanwhile, the imaging elements 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, and a sound sensor 618 are also connected to the bus 610.

[0055] The image processing unit 604 takes in image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates data on a panoramic image and a speaker image from the fisheye video. Furthermore, the image processing unit 604 synthesizes the panoramic image and the speaker image, etc., and outputs one video.

[0056] The imaging control unit 605 generally uses an I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0057] Furthermore, the imaging control unit 605 instructs the imaging elements 603a (one imaging element) and 603b (the other imaging element) to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from the PC. Some meeting devices 60 have a preview display function or a function corresponding to video display on a display (for example, a PC or smartphone display). In this case, the image data is output from the imaging element 603 continuously at a predetermined frame rate (frames / minute).

[0058] As described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0059] The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 through the I / F bus and performs a predetermined process on the sound data. The sound processing unit 609 modulates and amplifies the sound data received from the terminal device 10 and outputs it from the speaker 619.

[0060] The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories, and store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604, and data of a processed equirectangular projection image.

[0061] The operation unit 615 is a general term for operation buttons such as an imaging start button 615a. By operating the operation unit 615, the user can start imaging or recording, as well as turn the power on / off, establish communication connections, and input settings such as various imaging modes and imaging conditions.

[0062] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device is, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection I / F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. Video data and image data stored in the DRAM 614 are transmitted to an external terminal via the external device connection I / F 616, or recorded in an external medium. In addition, the meeting device 60 uses a plurality of external device connection I / Fs 616 to, for example, transmit image information captured and acquired by the meeting device 60 to a PC via USB and record it. In parallel with this, the meeting device 60 may obtain video (for example, screen information to be displayed in a remote conference application, etc.) from the PC to the meeting device 60, and further transmit the video from the meeting device 60 to another external device (display, projector, electronic whiteboard, etc.) via HDMI (registered trademark) for display.

[0063] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the meeting device 60, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also communicate with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0064] The audio sensor 618 is a sensor that acquires 360° audio information in order to identify from which direction the loudest audio is input in 360° around (horizontal plane) the meeting device 60. The sound processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction in 360°.

[0065] Other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use them for image correction and addition of location information.

[0066] In addition, the image processing unit 604 performs the following processes.

[0067] The CPU 611 creates a panoramic image in the following way. The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB complementation processing) on ​​the RAW data input from the image sensor that inputs the spherical image to create a fisheye image (image made up of curved images). It then performs flattening processing such as DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image (image made up of flat images) that shows 360° around the meeting device 60.

[0068] The CPU 611 creates speaker images in the following way. The CPU 611 creates speaker images by cutting out the speaker from a panoramic image (flat image) showing 360° of the surroundings. The CPU 611 cuts out the speaker image from the panoramic image, taking the voice input direction identified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. The method of cutting out a person's image from the voice input direction at this time is to cut out a 30° area centered on the identified voice direction from 360°, and perform face detection within that area to cut out the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., 3 people) who have spoken most recently from the cut out speaker images.

[0069] The panoramic image and the one or more speaker images may be sent separately to the information recording application 41, or the meeting device 60 may create one image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the one or more speaker images are sent separately from the meeting device 60 to the information recording application 41.

[0070] Fig. 5 is a diagram explaining the imaging range of the meeting device 60. As shown in Fig. 5(a), the meeting device 60 captures an image in a range of 360° in the horizontal direction. As shown in Fig. 5(b), the meeting device 60 takes an image in a range of 0° in the horizontal direction at the height of the meeting device 60, and takes an image in a range of a predetermined angle above and below.

[0071] FIG. 6 is a diagram for explaining the cutting out of a panoramic image and a speaker image. As shown in FIG. 5(b), the meeting device 60 performs perspective projection transformation by dividing the angle of view into predetermined angles in the vertical direction and predetermined angles in the horizontal direction. By performing perspective projection transformation across the entire 360° horizontal direction without gaps, a predetermined number of planar images are obtained, and a panoramic image 111 is obtained by connecting the predetermined number of planar images to the left and right. In addition, the meeting device 60 performs face detection from the panoramic image within a predetermined range centered on the voice direction, and creates a speaker image 112 by cutting out 15° left and right from the center of the face (30° in total).

[0072] <About the function> Next, the functional configuration of the record information display system 100 will be described with reference to Fig. 7. Fig. 7 is an example of a functional block diagram in which the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information display system 100 are divided into blocks and described.

[0073] <<Terminal Device>> The information recording application 41 that runs on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a video storage unit 17, an audio data processing unit 18, a recorded video display unit 19, an upload unit 20, an editing processing unit 21, and a search unit 22. Each of these units in the terminal device 10 is a function or a means that is realized by operating any of the components shown in FIG. 3 according to an instruction from the CPU 501 in accordance with the information recording application 41 that is expanded from the HD 504 to the RAM 503. The terminal device 10 also includes a storage unit 1000 that is constructed by the HD 504 and the like shown in FIG. 3. An information storage unit 1001 is constructed in the storage unit 1000.

[0074] The communication unit 11 communicates various information with the information processing system 50 via the network. For example, the communication unit 11 receives a list of remote conferences from the information processing system 50 and transmits a request for recognizing voice data to the information processing system 50.

[0075] The display control unit 13 displays various screens serving as a user interface in the information recording application 41 in accordance with screen transitions set in the information recording application 41. The operation accepting unit 12 accepts various operations for the information recording application 41.

[0076] The application screen acquisition unit 14 acquires screen information displayed by an application selected by the user, or screen information of a desktop screen, etc., from an OS (Operating System) or the like. When the application selected by the user is a remote conference application 42, a screen generated by the remote conference application 42 (an image captured by the terminal device camera of each site, a display image of the shared materials, and an image including participant icons and participant names, etc.) is acquired. The screen information displayed by the application (application screen) is information displayed as a window by a running application including a remote conference application, etc., and acquired as an image by an information recording application. The window of the application is drawn as an area of ​​the window as an area in the entire desktop image, and is displayed on a monitor, etc. The screen information displayed by the application can be acquired by other applications (information recording application, etc.) as an image file or a recording file consisting of multiple consecutive images via an API of the OS (Operating System) or an API of the application that displays it. Moreover, the screen information of the desktop screen is information consisting of an image of the desktop screen generated by the OS, and can be acquired as an image file or a recording file via the API of the OS in the same manner. The format of these image files may be bitmap, PNG, or other formats. The recording file format can be MP4 or other formats.

[0077] The audio acquisition unit 15 acquires audio (including audio data received from the remote conference application 42 during a remote conference) output by the terminal device 10 from a microphone or earphone. Even if the output audio is muted, the audio acquisition unit 15 can acquire the audio. The user does not need to perform an operation such as selecting the remote conference application 42 regarding audio data, and the audio acquisition unit 15 can acquire the audio that the terminal device 10 can output via an OS or an application API (Application Interface). This allows the audio data that the remote conference application 42 receives from another site 101 to be acquired. If the remote conference application 42 is not running or if a remote conference is not being held, the information recording application 41 may not be able to acquire the audio data. Note that the audio acquired by the audio acquisition unit 15 does not include the audio collected by the terminal device 10, and may be only the audio data to be output. This is because the meeting device 60 separately collects the audio of the site.

[0078] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 via wireless LAN, Bluetooth (registered trademark), or the like. The device communication unit 16 receives a panoramic image and a speaker image from the meeting device 60, and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0079] The video storage unit 17 stores the panoramic image and speaker image received by the device communication unit 16, and the application screen acquired by the application screen acquisition unit 14, as separate recorded videos. The video storage unit 17 stores the synthesized audio data separately. The video storage unit 17 may combine the audio data with the panoramic image to create a panoramic image with audio.

[0080] The voice data processing unit 18 requests the information processing system 50 to convert the synthesized voice data received from the meeting device 60 into text data.

[0081] The recorded video display unit 19 displays the recorded video. The recorded video is stored in the terminal device 10 during recording, and is then uploaded to the information processing system 50.

[0082] When the remote conference ends, the upload unit 20 transmits the recorded video and audio data to the information processing system 50.

[0083] The edit processing unit 21 executes editing of recorded moving images (deleting a part, splicing together, etc.) in response to user operations.

[0084] The search unit 22 searches the text data for the keyword entered by the user. For a portion of the text data matching the search, the time at which the comment was made from the start of the conference is also specified.

[0085] Fig. 8 shows video recording information stored in the information storage unit 1001. The video recording information has items such as a conference ID, a recording ID, an update date and time, a title, an upload, and a storage destination. When a user logs in to the information processing system 50, the information recording application 41 can download the conference information that the information processing system 50 acquires from the conference management system 9 and the application-side conference information that the information processing system 50 possesses. The conference ID and the like contained in the conference information are reflected in the video recording information. The video recording information in Fig. 8 is held by a terminal device 10 operated by a certain user.

[0086] The conference ID is identification information for identifying a remote conference that has been held. The conference ID is assigned when a remote conference schedule is registered in the conference management system 9, or the information processing system 50 assigns a conference ID in response to a request from the information recording application 41. The conference management system 9 is a system for registering conference and remote conference schedules, a URL (conference link) for starting a remote conference, reservation information for devices to be used in the conference, and the like, and is a scheduler or the like connected from the terminal device 10 via a network. The conference management system 9 is also capable of transmitting the registered schedule, etc. to the information processing system 50.

[0087] The recording ID is identification information for identifying a video recorded during a remote conference. The recording ID is assigned by the meeting device 60, but may be assigned by the information recording application 41 or the information processing system 50. A different recording ID is assigned to the same conference ID when recording was stopped in the middle of a remote conference but was resumed for some reason.

[0088] The update date and time is the date and time when the recorded video was updated (recording ended). If the recorded video was edited, it is the edit date and time. The title is the name of the conference. It may be set when the conference is registered in the conference management system 9, or may be arbitrarily set by the user. · Upload indicates whether or not the recorded video has been uploaded to the information processing system 50. The destination indicates the location (URL or file path) where the recorded video and text data are stored in the storage service system 70. Therefore, the user can view the uploaded recorded video at will. Note that the panoramic image, speaker image, app screen, and text data are stored with separate file names following the URL, for example.

[0089] <<Meeting Devices>> Returning to Fig. 7, the meeting device 60 has a terminal communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, and a voice synthesis unit 65. Each of these units of the meeting device 60 is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 611 in accordance with a program loaded from the ROM 612 to the DRAM 614.

[0090] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may communicate with the terminal device 10 via wireless LAN, Bluetooth (registered trademark), or the like.

[0091] The panoramic image creating unit 62 creates a panoramic image. The speaker image creating unit 63 creates a speaker image. The methods of creating these images have been described with reference to FIGS.

[0092] The sound collection unit 64 converts the audio signal acquired by the microphone of the meeting device 60 into audio data (digital). As a result, the contents of speech of the user or participant at the base of the terminal device 10 are collected.

[0093] The voice synthesizer 65 synthesizes the voice transmitted from the terminal device 10 and the voice collected by the voice collector 64. As a result, the voices uttered at the other locations 101 and the voices uttered at the own location 102 are combined.

[0094] <<Information Processing System>> The information processing system 50 has a communication unit 51, an authentication unit 52, a screen generation unit 53, a conference information acquisition unit 54, and a text conversion unit 55. Each of these units in the information processing system 50 is a function or a means for performing a function that is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed by the HD 504 etc. shown in Fig. 3. In the storage unit 5000, an application-side conference information storage unit 5001 and a recording information storage unit 5002 are constructed.

[0095] The communication unit 51 transmits and receives various information to and from the terminal device 10. For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 and receives a request for recognizing voice data from the terminal device 10.

[0096] The authentication unit 52 authenticates the user who operates the terminal device 10. For example, the authentication unit 52 authenticates the user based on whether or not authentication information (user ID and password) included in an authentication request received by the communication unit 51 matches authentication information stored in advance. Note that the authentication information may be an IC card number, biometric authentication information such as face or fingerprint, etc. Also, the authentication unit 52 may perform authentication using an external authentication system or an authentication method such as OAUTH.

[0097] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 executes a native application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 executes a Web application, the screen information is generated by HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), or the like.

[0098] The conference information acquisition unit 54 acquires conference information from the conference management system 9 using the account of each user or a system account assigned to the information processing system 50. The conference information acquisition unit can acquire a list of remote conferences for which a user belonging to a tenant has viewing authority. The viewing authority may be granted to the conference information managed by the conference information acquisition unit 54 directly from an information recording application of the terminal device 10. The remote conference information for which a user belonging to a tenant has viewing authority includes information on conferences created by the user and information on conferences for which the user has been given viewing authority by other users. Since a conference ID is set for a remote conference, the remote conference and recorded information are associated with each other by the conference ID.

[0099] The text conversion unit 55 converts the voice data requested by the terminal device 10 to text data into text data by using an external voice recognition service. The text conversion unit 55 may also convert the voice data itself.

[0100] 9 is an example of application-side conference information stored in the application-side conference information storage unit 5001. The application-side conference information is conference information on the information recording application 41 side that is created separately from the conference information when the information recording application 41 creates the recording information.

[0101] The conference ID is identification information of the conference. The conference ID associates the conference information managed by the conference management system 9 with the application-side conference information. The application ID is identification information for the information recording application 41. The user ID to which the access token is issued is identification information of the storage service system 70 that issues the access token. The access token issuer URL is the URL of the storage service system 70 that issues the access token. The issue date and time is the date and time when the access token was issued. The issue date and time is determined at the time of issuance. The validity date is the date and time when the access token becomes valid. The validity date and time is determined by the issuer. The expiration time is the expiration time of the access token. The expiration time is determined by the issuer. The permission information is the processing authority that the user can perform using the access token. The permission information is determined by the issuer. The display name is the display name of the user in the information recording application 41. Last Name is the user's last name. First name is the user's first name.

[0102] 10 shows the recording information stored in the recording information storage unit 5002. The recording information has a list of recorded videos recorded by all users belonging to the tenant. The recording information has items such as a meeting ID, a recording ID, an update date and time, a title, and a storage destination. These items may be the same as those in FIG. 8. The storage destination (path information such as a URL of a cloud storage system) may be stored in the recording information storage unit 5002 by a user inputting desired storage destination information on a user setting screen of the information recording application 41 of the terminal device 10 or the like.

[0103] <<Conference Management System>> Returning to Fig. 7, the explanation will be given. The conference management system 9 has a conference information management unit 31. Each of these units in the conference management system 9 is a function or a means for performing a function that is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The conference management system 9 also has a storage unit constructed by the HD 504 etc. shown in Fig. 3. A conference information storage unit 3001 is constructed in the storage unit 3000.

[0104] The conference information management unit 31 manages information about conferences held by tenants. The conference information is stored in the conference information storage unit 3001.

[0105] 11 is an example of the conference information stored in the conference information storage unit 3001. The conference information is managed by a conference ID, and has the items shown in the figure. The conference ID is identification information for the conference. The tenant ID is identification information of the tenant in which the conference is held. · The title is the subject or agenda of the meeting. The organizer is the organizer of the meeting. · Participants is the list of participants invited to the conference. · Viewing privileges are a list of users who have access to meeting resources, including recorded video. Ad-hoc participants are a list of guest participants. Location is information about the conference room, such as the name of the conference room. · Start time is the time the meeting is scheduled to begin. The end time is the time the meeting is scheduled to end. The conference creator is the user ID, etc., who registered the conference information. - Password is the password that participants use to log in to the conference.

[0106] <<Storage service system>> The storage service system 70 may be any service system that stores information. Here, the structure of text data stored in the storage service system 70 will be described.

[0107] FIG. 12 is a diagram for explaining the structure of text data stored in the storage service system 70. As shown in FIG. 12, the items ID, time, user, and text are associated with the text data. The ID is identification information that is assigned when the text data is divided into texts according to a predetermined rule. The predetermined rule is set in the voice recognition service system 80, and may be, for example, dividing the data when a silence state continues for a certain period of time, forcibly dividing the data at a certain period of time even if there is no silence state, or dividing the data into sentences detected by morphological analysis. The time is the time of the statement based on the duration from the start of recording. Since the so-called time is also saved at the start of recording, the time (absolute time) when the character string of text was uttered can also be known. The user indicates whether the statement is from the own location or another location, as determined by sound pressure or the like. The character string of text is a part of the divided text data.

[0108] In this way, since time and text are associated with each other, when text data matches the search, the information recording application 41 can display the recorded video from the display time corresponding to this time.

[0109] <How to create a full-angle panoramic image> Next, we will explain how to create an omnidirectional panoramic image (hereinafter simply referred to as a panoramic image) that captures at least approximately 360° in the horizontal direction, and also explain why an upper eaves is necessary. Approximately 360° does not have to be exactly 360°, but can be any angle range that can be determined to capture the entire horizontal direction. For example, approximately 360° can be about 350° to 360°.

[0110] First, a method for creating a panoramic image 122 from camera images 121A and 121B will be described with reference to Fig. 13. Fig. 13(a) shows camera images 121A and 121B, and Fig. 13(b) shows a panoramic image 122. Each of camera images 121A and 121B has a horizontal angle of view exceeding 180°. The panoramic image creating unit 62 can generate the panoramic image 122 by joining, for example, the vicinity of the A_1 end of camera image 121A to the vicinity of the B_1 end of camera image 121B at a seam 123, and joining the vicinity of the A_2 end of camera image 121A to the vicinity of the B_2 end of camera image 121B at a seam 124. When combining, the panoramic image creation unit 62 performs pattern matching on the area near the B_1 end with the image near the A_1 end, and pattern matching on the area near the B_2 end with the image near the A_2 end (this relationship can be reversed), thereby determining the combining portion of the two camera images 121A, 121B where the images in the respective areas are most similar. By combining the images at this combining portion, the portions with a viewing angle exceeding 180° overlap, and a 360° panoramic image 122 is generated.

[0111] For example, when the A_1 end and the B_1 end, and the A_2 end and the B_2 end are joined, respectively, there are a seam 123 and a seam 124. If there are no effects of flare or ghosting, which will be described below, the seam 123 and the seam 124 are not noticeable.

[0112] <<When affected by flare>> Next, a case where the camera image 131A or the camera image 131B is affected by flare will be described with reference to FIG. 14. FIG. 14 is a diagram for explaining the effect of light from the ceiling light in the conference room on the joints 133 and 134 between the camera image 131A and the camera image 131B. FIG. 14(a) is an example of a case where flare occurs in the camera image 131A captured by the meeting device 60. Flare is a phenomenon in which when strong light enters a lens, the light is reflected in the lens, causing fogging (light other than the intended light is irradiated on the imaging element, causing a part to become black) and uneven brightness. If a bright ceiling light is installed near the ceiling of the meeting device 60, the direct light from the ceiling light enters the lenses 602A and 602B, causing the light to be reflected in the lens, resulting in a phenomenon called flare.

[0113] Fig. 14(b) shows a panoramic image when flare occurs near the A_2 end of camera image 131A as in Fig. 14(a). Because flare occurs near the A_2 end of camera image 131A, discontinuity in color occurs due to the influence of flare at seam 134 in panoramic image 132 where the A_2 end of camera image 131A and the B_2 end of camera image 131B are joined, causing a sense of discomfort to the user.

[0114] Flare may occur not only at the ends of camera images 131A and 131B, but may occur over the entire image. In this case, panoramic image 132 becomes an unnatural panoramic image due to discontinuity in color caused by the influence of flare at both seams 133 and 134 of panoramic image 132.

[0115] In addition to ceiling lighting, flare may also occur from desk lamps. Furthermore, if the meeting device 60 is installed outdoors or next to a window, flare may also occur due to outdoor sunlight.

[0116] <<When ghosts are present>> Next, a case where a ghost occurs in camera images 141A and 141B will be described with reference to FIG. 15. FIG. 15 is a diagram for explaining the influence of a ghost caused by light from a ceiling light in a conference room on joints 143 and 144 between camera images 141A and 141B. A ghost is a phenomenon in which, when strong light enters a lens, the light reflected within the lens is imaged by an imaging element and appears as a ball or an aperture. If a bright ceiling light is installed near directly above the meeting device 60, the direct light from the ceiling light may enter lenses 602A and 602B, causing a ghost to appear on the image.

[0117] In FIG. 15(a), as an example, ghosts occur near the A_2 end of the camera image 141A and near the B_2 end of the camera image 141B. The panoramic image creation unit 62 performs pattern matching near the A_2 end of the camera image 141A and near the B_2 end of the camera image 141B to combine the camera images 141A and 141B. In this case, the pattern matching tends to cause the combining process to match the patterns of the ghosts that occur in each image. For this reason, as shown in FIG. 15(b), even if the A_2 end of the camera image 141A and the B_2 end of the camera image 141B are combined, they are combined at a position that is different from the optimal position where they would be combined if there were no ghosts. As a result, there is a possibility that the image will be misaligned at the joint portion 144, which the user will feel uncomfortable. Note that a similar inconvenience may also occur in the case of a flare.

[0118] <Meeting device structure details> Therefore, in this embodiment, the occurrence of flare and ghost is suppressed by attaching an upper canopy to the meeting device 60. The structure of the meeting device will be described in detail with reference to Figs.

[0119] Fig. 16 shows a side view of the meeting device. The meeting device 60 has an imaging unit 633A and an imaging unit 633B whose optical axes face in opposite directions. The imaging units 633A and 633B each have a lens 602A, 602B having a horizontal angle of view exceeding 180°, and as described in Fig. 13, the images from each lens are projected onto the imaging elements 603a and 603b to obtain the camera image 121A and the camera image 121B. The camera image 121A and the camera image 121B are combined into a 360° horizontal panoramic image by image processing by joining them at the seams 123 and 124.

[0120] The meeting device 60 of this embodiment has the following features. That is, the upper eaves 632 is provided on the upper parts of the two imaging units 633A and 633B, and the acute angle (hereinafter referred to as the incident angle) between the straight line connecting the lens apex and the upper eaves protruding part (described later) that hits the end of the upper eaves 632 when viewed from the lenses 602A and 602B and the optical axis (horizontal line) is 30° or more and 67° or less. Also, the incident angle θ is smaller in the lens end direction than in the lens front direction (optical axis direction). In short, one of the features is that the upper eaves 632 is provided on the upper parts of the two imaging units 633A and 633B, and the upper eaves 632 can appropriately limit the incidence of direct light from the ceiling lighting on the lenses 602A and 602B.

[0121] Although detailed explanations are given below, the structure and functions of the meeting device 60 are not limited to the embodiments described below.

[0122] The meeting device 60 includes an omnidirectional camera unit 631 capable of capturing images in all directions of 360° at least in the horizontal direction. The omnidirectional camera unit 631 includes an imaging unit 633A, an imaging unit 633B, and lenses 602A and 602B. The omnidirectional camera unit 631 is located at the top of the meeting device 60 and includes imaging units 633A and 633B whose optical axes face in opposite directions. The optical axes face in any horizontal direction. The imaging units 633A and 633B include lenses 602A and 602B, which are super-wide-angle lenses having an angle of view exceeding 180°, and the imaging elements 603a and 603b described above. The omnidirectional camera unit 631 may have a mechanism for rotating horizontally relative to the meeting device 60.

[0123] The meeting device 60 includes an upper eaves 632 above the imaging units 633A and 633B. The upper eaves 632 is located at the top of the vertical upper side of the meeting device 60, and is positioned to block light incident on the lenses 602A and 602B from directly above (to cover the lenses 602A and 602B). The imaging units 633A and 633B project images from the lenses 602A and 602B onto the image sensors 603a and 603b, respectively, to obtain the camera images 121A and 121B. The meeting device 60 generates a 360° horizontal panoramic image by combining the camera images 121A and 121B at the seams 123 and 124 by image processing.

[0124] Furthermore, the meeting device 60 includes a plurality of external device connection I / Fs 616 for connecting external devices or power sources. The external device connection I / Fs 616 allow the meeting device 60 to receive power from an external power source such as an AC adapter.

[0125] Also, the external device connection I / F 616 allows connection to an external device such as a computer via a connection cable such as a USB cable, and data can be sent and received from the external device. Furthermore, the meeting device 60 includes a speaker 619 and a microphone 608 inside the main body. The microphone 608 allows the terminal device 10 to transmit not only video from its own base to multiple bases during a remote conference, but also audio from other bases, and the speaker 619 built into the side of the main body allows audio from other bases to be played back in real time at the own base.

[0126] <<The structure of the upper eaves and its effect>> Fig. 17 is a diagram for explaining the configuration of upper eaves 632. Fig. 18 is a diagram for explaining the angle of incidence of light from the ceiling onto lenses 602A, 602B. By providing upper eaves 632, meeting device 60 can reduce direct light from the ceiling lighting in the conference room from entering lenses 602A, 602B. Therefore, meeting device 60 can reduce adverse effects on seams of a panoramic image and provide a user with a panoramic image that is easy to view and does not look strange.

[0127] The maximum angle at which light from the ceiling can enter the lens vertex 635 of the lenses 602A and 602B is the incident angle θ. The point where the light incident at the incident angle θ meets the upper eaves 632 is called the upper eaves protrusion 636. The straight line connecting the lens vertex 635 and the upper eaves protrusion 636 is called the "shielding line 637," and the area surrounded by the shielding line 637 is called the "non-incident area 638." Light from the ceiling lighting in the non-incident area 638 does not directly enter the lenses 602A and 602B. This allows the meeting device 60 to reduce adverse effects on the seams of the panoramic image and provide the user with a panoramic image that is easy to view and does not look strange.

[0128] As the incidence angle θ, which is the angle between the shielding line 637 and the optical axis, becomes smaller, it becomes more difficult for the light from the ceiling lighting to directly enter the lenses 602A and 602B, and the non-incident area 638 becomes larger. In addition, the shortest distance between the lens apex 635 and the ceiling lighting when the shielding line 637 is extended to the ceiling (hereinafter referred to as the incident distance L) becomes longer. The incident distance L is the shortest distance at which the light from the ceiling lighting can directly enter the lenses 602A and 602B. Since the intensity of light attenuates in inverse proportion to the square of the distance, even if the light from the office ceiling lighting is directly incident, it is possible to reduce the adverse effects on the joints by reducing the incidence angle θ and lengthening the incident distance L.

[0129] Fig. 19 is a diagram for explaining the relationship between the incident angle θ and the amount of light in a typical conference room. As shown in Fig. 19, the larger the incident angle θ, the greater the amount of light incident on lenses 602A and 603B. The amount of light can also be called the light remaining rate, which indicates the attenuation of light.

[0130] The following description will be given with reference to Fig. 18. For example, the distance from lens vertex 635 of omnidirectional camera unit 631 to the ceiling in the direction directly above lens vertex 635 is denoted as h. The position where perpendicular line 641 drawn from lens vertex 635 to the ceiling intersects with the ceiling is denoted as A, and the position where shielding line 637 intersects with the ceiling is denoted as B. The distance from lens vertex 635 to position B is incident distance L, Injection distance L= h / SINθ For h, an approximate value based on the ceiling height and desk height of a typical office may be set, or an actual measured value may be set.

[0131] When ceiling lights of the same brightness are installed at positions A and B, light attenuates inversely proportional to the square of the distance, so the ratio of the amount of light from the ceiling light that directly enters lens 602 when a ceiling light is installed at position B to that when a ceiling light is installed at position A (light attenuation rate) is calculated using equation (1).

number

[0132] As shown in Equation (1), the amount of light from the ceiling lighting directly incident on the lenses 602A and 602B decreases as the incident angle θ decreases. Therefore, the amount of light incident from the ceiling lighting at position B is suppressed more than the amount of light incident from the ceiling lighting at position A. According to the brightness of the lighting in a general conference room in an office, if the amount of incident light can be suppressed to 85% or less with respect to position A where the light is most strongly incident on the lenses 602A and 602B, it is considered that the occurrence of flare and ghosting can be prevented. That is, if the amount of light at position B is suppressed to 85% or less with respect to position A, the meeting device 60 can reduce the adverse effect on the seam portion of the panoramic image. According to Equation (1) or FIG. 18, in order to suppress the amount of light to 85% or less, the adverse effect can be achieved by setting the incident angle to 67° or less.

[0133] On the other hand, it is preferable that the angle of view required to photograph a person in a meeting can secure 30° or more above the horizontal line, and it is also preferable that the incident angle θ is 30° or more. From the above, by setting the incident angle to about 30° to about 67°, it is possible to secure the angle of view required to photograph a person while reducing the influence (degradation of image quality) of the ceiling lighting on the seam portion of the panoramic image. The phrase "about 30° to about 67°" means that it does not have to be exactly from 30° to 67°, and it may be generally within this range.

[0134] <<Projection amount of the upper visor>> Subsequently, based on FIG. 20, the adjustment of the incident angle by the projection amount of the upper visor 632 will be described. FIG. 20(a) shows a perspective view of the upper visor 632, and FIG. 20(b) shows a top view of the upper visor 632 viewed from above. The incident angle θ can be adjusted by changing the projection amount d1 in the front direction of the lens. The projection amount d1 in the front direction of the lens is the distance from the upper visor projection portion 636 to the lens apex 635 (the distance projected onto the horizontal plane).

[0135] In FIG. 20(b), as an example, the shape of the upper eaves 632 is circular when the omnidirectional camera unit 631 is viewed from above. The optical axis of the lens passing through the lens apex 635 (the direction from the lens apex 635 toward a point in space that is the center of the captured image) is defined as the lens front direction. The direction with the widest angle in the horizontal direction based on the lens apex 635 (the direction toward a point in space that is the horizontal end of the captured image) is defined as the lens end direction. Since the protrusion amount d1 in the lens front direction is shorter than the protrusion amount d2 in the lens end direction, the incidence angle θ in the lens end direction is limited to be smaller than the incidence angle θ in the lens front direction. The lens end direction coincides with the direction that becomes the seam in the panoramic image.

[0136] Therefore, by limiting the angle of incidence θ toward the end of the lens to be smaller than the angle of incidence θ toward the front of the lens, it is possible to reduce the adverse effects of direct light falling on the seam while maintaining overall brightness.

[0137] Incidentally, the incidence angle θ in the direction of the lens end is preferably about 30° or more and about 45° or less. By making the incidence angle in the direction of the lens end 45° or less, flare and ghosting occurring at the lens end can be further suppressed, and color changes and misalignment of the image joints at the seams of the panoramic image can be suppressed. Also, by ensuring an incidence angle of 30° or more, a viewing angle and brightness sufficient for capturing images of people can be ensured. About 30° to about 45° means that it does not have to be exactly 30° to exactly 45°, and it is sufficient as long as it is generally within this range.

[0138] Furthermore, as shown in formula (2), the ratio R of the incident angle θ in the lens front direction to the incident angle θ in the lens end direction θ It is preferable to set it to 2 or less.

number

[0139] Fig. 21 shows a top view of the upper eaves 632, which is elliptical when viewed from above. As shown in Fig. 21, the shape of the upper eaves 632 when viewed from above the meeting device 60 is not limited to a circle, but may be elliptical. By making it elliptical, the incident angle θ in the direction of the lens end can be limited to be smaller than the incident angle θ in the direction of the lens front.

[0140] Furthermore, the shape of the upper eaves 632 may be any shape capable of limiting the angle of incidence, such as a rectangle or an octagon, when viewed from above. Fig. 22(a) shows the case where the shape of the upper eaves 632 when viewed from above is rectangular, and Fig. 22(b) shows the case where the shape of the upper eaves 632 when viewed from above is octagonal. If it is rectangular, the upper eaves 632 is easy to process. If it is octagonal, the user can easily grasp the orientation of the lenses 602A and 602B, so when the user installs the meeting device 60, it is easy to adjust the orientation of the light from the ceiling lighting so that it is less likely to cause flare or ghosting.

[0141] Furthermore, as shown in Fig. 23, the upper eaves 632 may be provided with a light blocking member capable of adjusting the angle of incidence of light. Fig. 23 is a diagram explaining the function of light blocking members 643, 644 provided on the upper eaves 632. Fig. 23(a) shows the light blocking members 643, 644 in a stored state, and Fig. 23(b) shows the light blocking members 643, 644 in a pulled-out state.

[0142] The light blocking members 643, 644 are thin plates, and are usually stored inside the upper surface of the upper eaves 632. The user can pull out or push in the light blocking members 643, 644 with his / her fingers. When the user pulls out the light blocking members 643, 644, the angle of incidence of light can be reduced, and when the user pushes them in, the angle of incidence can be increased. Therefore, the user can pull out the light blocking members 643, 644 to an appropriate pull-out position so that a bright panoramic image without flare can be obtained while looking at the actually created panoramic image.

[0143] <<Side view of upper eaves>> Next, a side view of the upper eaves 632 will be described with reference to FIG. 24. FIG. 24 is a diagram illustrating the shape of the upper eaves 632 viewed from the side. The incident angle θ can also be adjusted by changing the linear distance between the upper eaves protruding portion 636 and the optical axis (upper eaves height d3). For example, as shown in FIG. 24(a), it is possible to reduce the incident angle θ by shortening the upper eaves height d3. Conversely, it is also possible to increase the incident angle θ by increasing the upper eaves height d3, as shown in FIG. 24(b).

[0144] The upper eaves height d3 can be freely set as long as the incident angle θ can be limited to a desired angle. An extension mechanism that can adjust the height of the upper eaves 632 may be provided. The shape from the upper part of the lenses 602A, 602B to the upper eaves protruding part 636 may be a smooth slope, a curve, or a step-like shape. FIG. 25 shows an example of the shape of the side of the upper eaves 632. FIG. 25(a) shows an example in which the shape of the side surfaces 641, 642 from the upper part of the lenses 602A, 602B to the upper eaves protruding part 636 is a curved shape that curves inward. FIG. 25(b) shows an example in which the shape of the side surfaces 643, 644 from the upper part of the lenses 602A, 602B to the upper eaves protruding part 636 is a step-like shape. The shapes of FIG. 25(a) and (b) have the effect that even if light incident on the lower side of the upper eaves 632 is reflected by these shapes, it is difficult for it to enter the lenses 602A, 602B.

[0145] <Example of a panoramic image captured with a meeting device> A display example of a panoramic image will be described with reference to Fig. 26. Fig. 26 shows a display example in which the terminal device 10 displays a panoramic image as a final output image. The panoramic image displayed by the terminal device 10 can be reconstructed and output as a final output image. Fig. 26(a) is a display example in which a panoramic image is displayed in its entirety at 360° in one stage. Fig. 26(b) is a display example in which a panoramic image is divided into two stages, each of which is displayed at 180°. Fig. 26(c) is a display example in which a panoramic image is divided into three stages, each of which is displayed at 120°.

[0146] In either display example, the participants in the Web conference can check the 360° panoramic image. In addition, the terminal device 10 divides the image into an appropriate number of stages, so that the image can be displayed with an appropriate aspect ratio and size when it is displayed on the display.

[0147] Fig. 26(d) is a display example in which a panoramic image is combined with a close-up image that is appropriately enlarged after trimming the areas of the participants and other objects shown in the panoramic image. Possible areas to be close-up include speakers, non-speakers, an electronic whiteboard, or any area specified by the user. In Fig. 26(d), the panoramic image conveys the 360° view in all directions, and also displays the details of a part of the conference room that is difficult to see in the panoramic image.

[0148] The final output image is not limited to these examples, and can be displayed in any layout based on the created panoramic image.

[0149] <Screen transition> Next, several screens displayed by the terminal device 10 during a remote conference will be described with reference to Fig. 27 to Fig. 36. Fig. 27 shows an initial screen 200 after login displayed by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs authentication information and succeeds in logging in, the initial screen 200 of Fig. 27 is displayed.

[0150] The initial screen 200 has a fixed display button 201, a change to front button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter, referred to as speaker images 204 when no distinction is made), and a start recording button 205. If the meeting device 60 has already started up and is capturing images of the surroundings at the time of login, the panoramic image 203 and speaker image 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether or not to start recording while viewing these. If the meeting device 60 is not started up (not capturing images), the panoramic image 203 and speaker image 204 are not displayed.

[0151] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or may display only the speaker images 204 of the N most recent speakers. Fig. 27 shows an example in which speaker images 204 of up to three people are displayed. Until a participant speaks, there may be no speaker image 204 (the speaker images are increased one by one according to the speech), or three speaker images 204 of participants in a predetermined direction may be displayed (the speaker images are replaced according to the speech).

[0152] When no one is speaking, such as immediately after the meeting device 60 is started, a predetermined direction (0°, 120°, 240°, etc.) within the horizontal 360° is created as the speaker image 204. When a fixed display, which will be described later, is set, the setting of the fixed display takes precedence.

[0153] The fixed display button 201 is a button for the user to perform an operation of fixing a certain area of ​​the panoramic image 203 in close-up as a speaker image 204 .

[0154] 28 is a diagram for explaining an operation method when the fixed display button 201 is on. For example, the user moves a rectangular window 206 on the panoramic image 203 using a pointing device such as a mouse or a touch panel. The user aligns the window 206 with a whiteboard, a podium, or the like included in the panoramic image 203. The user's operation is transmitted to the meeting device 60, which creates an image of the range specified by the window out of 360° in the horizontal direction, with the same size as the speaker image 204, and transmits it to the terminal device 10. In this way, the speaker image 204 can continue to display a whiteboard or the like other than the speaker.

[0155] Returning to FIG. 27, the front change button 202 is a button for the user to perform an operation to change the front of the panoramic image 203 (as a panoramic image is captured 360° horizontally, the right end and left end correspond in terms of direction). The user can slide the panoramic image 203 left and right with a pointing device to determine which participant will be shown in front. The user's operation is transmitted to the meeting device 60, which then changes the angle of the front out of the 360° horizontal direction, creates a panoramic image, and transmits it to the terminal device 10.

[0156] When the user presses the record start button 205, the information recording application 41 displays the recording setting screen 210 shown in FIG.

[0157] If the meeting device 60 is not connected or is not turned on when the information recording application 41 is started, an unrecognized device screen 250 in FIG. 29 is displayed.

[0158] 29 shows an example of an unrecognized device screen 250. The unrecognized device screen 250 displays a message 251 saying "The device cannot be recognized. Please turn on the device and connect it." The user can check the power and connection status of the meeting device 60 by looking at this message.

[0159] 30 is an example of a recording setting screen 210 displayed by the information recording application 41. In the recording setting screen 210, the user can set whether to record (whether to include in recording) the panoramic image and speaker image created by the meeting device 60, and the desktop screen or the screen of a running application of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or the screen of a running application, only the audio (audio output by the terminal device 10+audio collected by the meeting device 60) is recorded.

[0160] The camera toggle button 211 is a button for switching on and off the recording of the panoramic image and the speaker image created by the meeting device 60. The camera toggle button 211 may be set to record the panoramic image and the speaker image separately.

[0161] The PC screen toggle button 212 is a button for switching on and off recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10. When the PC screen toggle button 212 is in the on state, the desktop screen is recorded.

[0162] If the user wants to record the screen of an application, the user further selects the application in the application selection field 213. The application selection field 213 displays the names of applications currently being executed by the terminal device 10 in a pull-down format. The user can select the application to be recorded. The information recording application 41 acquires the application name from the OS. The information recording application 41 can display only those applications having a UI (screen) among the currently executing applications. The remote conference application 42 may be included in the selected applications. Therefore, the information recording application 41 can record materials displayed in the remote conference application 42 and participants at each location as videos. In addition, the applications displayed in the pull-down menu are various applications currently being executed on the terminal device, such as a presentation application, a word processing application, a spreadsheet application, an application for creating and editing materials such as documents, a cloud electronic whiteboard application, and a web browser application. Therefore, the user can flexibly select the screen of an application to be included in the recorded video.

[0163] Furthermore, when recording on an app basis, the user can select multiple apps, and the information recording app 41 can record the screens of all the selected apps.

[0164] When both the camera toggle button 211 and the PC screen toggle button 212 are off, the video recording confirmation window 214 displays "Only audio will be recorded." This audio is the audio output by the terminal device 10 (audio received by the remote conference application 42 from other locations 101) and audio collected by the meeting device 60. In other words, as long as a remote conference is being held, the audio of the remote conference application 42 and the audio of the meeting device 60 are saved regardless of the recording of images. However, the user may be able to selectively stop saving the audio of the remote conference application 42 and the audio of the meeting device 60 by user settings.

[0165] The recorded video is recorded as follows according to the on / off combination of the camera toggle button 211 and the PC screen toggle button 212. The recorded video is also displayed in real time in the recorded content confirmation window 214.

[0166] When the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image are displayed in the recorded content confirmation window 214 .

[0167] When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen is also selected), the recording content confirmation window 214 displays the desktop screen or the screen of the selected application.

[0168] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application are displayed side by side in the recording content confirmation window 214.

[0169] Therefore, there are cases where the panoramic image, the speaker image, and the app screen are not recorded, or cases where the panoramic image, the speaker image, and the app screen are not recorded at all, but in this embodiment, for convenience, the images created by the information recording application 41 are referred to as recorded videos.

[0170] Fig. 31 shows an example of the display of the recorded content confirmation window 214 when the camera toggle button 211 is on and the PC screen toggle button 212 is off. In Fig. 31, the panoramic image 203 and the speaker image 204 are displayed large.

[0171] Fig. 32 is a display example of the recorded content confirmation window 214 when the camera toggle button 211 is on and the PC screen toggle button 212 is on. In Fig. 32, the panoramic image 203 and the speaker image 204 are displayed on the left, and the application screen 217 is displayed on the right.

[0172] Therefore, the user can check, via the recording content confirmation window 214, what content will be recorded in the video (particularly images from the meeting device 60) according to the settings on the recording setting screen 210 before starting recording.

[0173] 32 shows an example of a recorded video display when only one app is selected, but when two or more apps are selected, the screens of the second and subsequent apps are linked to the right side in sequence. Alternatively, the screens of the second and subsequent apps may be arranged in two dimensions, vertically and horizontally.

[0174] Returning to FIG. 30, the explanation will be given. The recording setting screen 210 has a check box 215 together with a message saying "Automatically transcribe after uploading recording." The recording setting screen 210 also has a start recording now button 216. When the user checks the check box 215, text data into which comments made during the remote conference are converted is attached to the recorded video. In this case, after recording ends, the information recording application 41 uploads the audio to the information processing system 50 together with a request to convert it into text data. Furthermore, when the user presses the start recording now button 216, the recording in progress screen 220 in FIG. 33 is displayed.

[0175] Fig. 33 is an example of a recording screen 220 displayed by the information recording application 41 during recording. Note that the explanation of Fig. 33 will mainly focus on the differences from Fig. 30. The recording screen 220 displays a recorded video that is recorded under the conditions set by the user on the recording setting screen 210. The recording screen 220 in Fig. 33 is when the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays a panoramic image 203 and a speaker image 204 (both of which are videos) created by the meeting device 60. The recording screen 220 displays a recording icon 225, a pause button 226, and an end recording button 227.

[0176] When the user sets the PC screen toggle button 212 to ON, as shown in FIG. 32, the recording in progress screen 220 displays the panoramic image and speaker image, as well as the desktop screen and application screen next to each other.

[0177] The pause button 226 is a button for stopping recording, and after stopping, it also accepts resuming recording. The recording end button 227 is a button for ending recording. With the pause button 226, the recording ID does not change, while with the recording end button 227, the recording ID changes. When pausing and then resuming recording, the user can also reset the recording conditions set on the recording settings screen 210 again. In that case, the information recording application 41 may create a plurality of recording files each time recording is stopped (for example, when the recording end button 227 is pressed), or may concatenate the plurality of files so as to be continuous as one video (for example, when the pause button 226 is pressed). Also, when the information recording application 41 displays the recorded video, the plurality of recording files may be continuously displayed as one video.

[0178] Also, the recording screen 220 during recording has an information acquisition button 221 from the calendar, a meeting name field 222, a time field 223, and a location field 224. The information acquisition button 221 from the calendar is a button for the user to acquire meeting information from the meeting management system 9. When the information acquisition button 221 from the calendar is pressed, the information recording application 41 acquires and displays a list of meetings for which the user has viewing authority from the information processing system 50. The user selects a remote meeting to be held from the list of meetings. Also, the information recording application 41 can also acquire the meeting information on the application side. Thereby, the meeting information is reflected in the meeting name field 222, the time field 223, and the location field 224. The title of the meeting information is reflected in the meeting name field 222, the start time and end time are reflected in the time field 223, and the location is reflected in the location field 224. Also, the remote meeting in the meeting management system 9 and the recording information are associated with each other by the meeting ID.

[0179] When the remote meeting ends and the user ends the recording, a recorded video with audio is created.

[0180] 34 is an example of the conference list screen 230 displayed by the information recording application 41. The conference list screen 230 is a list of conferences, and can display a list of recorded information recorded in remote conferences. It also includes conferences that were held only in a certain conference room, regardless of whether they were remote conferences. The conference list screen 230 displays an integrated display of conference information that the logged-in user has permission to view in the conference information storage unit 3001, application-side conference information, and information stored in the video recording information storage unit 5002 that is associated with the remote conference. The video recording information stored in the information storage unit 1001 may be further integrated.

[0181] The conference list screen 230 is displayed when the user selects the conference list tab 231 on the initial screen 200 in Fig. 27. The conference list screen 230 displays a list 236 of recorded information that the user has viewing authority for. The conference creator (person who creates the minutes) can set viewing authority for participants. The conference list may be a list of stored recorded information or a list of conference schedules and conference data.

[0182] The conference list screen 230 has the following items: check box 232 , update date and time 233 , title 234 , and status 235 . The checkboxes 232 are used to accept the selection of recorded files. The checkboxes 232 are used when the user wishes to delete recorded files collectively. The update date and time 233 indicates the start and end times of recording the recorded video. If the video has been edited, the edit date and time may be used. The title 234 is the title of the meeting (topic, etc.). It may be transcribed from the meeting information or set by the user. The status 235 indicates whether or not the recorded video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, an upload button is displayed. If there is recorded video that has not been uploaded, it is preferable that the information recording application 41 automatically uploads it when the user logs in to the information processing system 50.

[0183] When the user selects an arbitrary title or the like from the list 236 of recorded moving pictures with a pointing device, the information recording application 41 displays a recorded moving picture display screen 240 shown in Fig. 35. On the recorded moving picture display screen 240, it is possible to display a recorded moving picture, etc.

[0184] 35 is an example of a recorded video display screen 240 after a recorded video is selected, displayed by the information recording application 41. The recorded video display screen 240 has an image display field 241, a transcription button 242, one or more text display fields 243, an auto scroll button 244, and a search button 245.

[0185] The image display field 241 has a display button 241a, a rewind button 241b, a fast forward button 241c, a time indicator 241d, a display speed button 241e, a volume button 241f, etc. The image display field 241 displays a recorded moving image. In FIG. 35, the recorded moving image in the image display field 241 has a panoramic image and a speaker image arranged on the left side, and the screen of the remote conference application 42 displayed on the right side. During the remote conference, the screen of the remote conference application 42 changes to an image of the location or an image of the document. Therefore, the user can view the screen of a desired scene by operating various buttons.

[0186] When the voice data of the recorded moving image being displayed in the image display field 241 has been converted into text data, the remarks are displayed as text in the text display field 243. The text display field 243 displays the text data into which the synthesized voice data has been converted by voice recognition.

[0187] The transcription button 242 is a button that allows the user to switch whether or not the text data displayed in the text display field 243 is displayed in accordance with the display time of the recorded video.

[0188] The automatic scroll button 244 is a button that allows the user to switch whether or not to automatically scroll the text data regardless of the display time.

[0189] The search button 245 is a button for the user to search for text data by specifying a keyword. Details will be described later.

[0190] Furthermore, the recorded video display screen 240 may allow the recorded video to be downloaded.

[0191] FIG. 36 is an example of an editing screen 260 for a recorded video. The editing screen 260 is transitioned to automatically from the recording screen 220 or when the user performs a predetermined operation on the recorded video display screen 240. The editing screen 260 has a first display field 261 and a second display field 262. The first display field 261 displays the recorded video at a certain moment during display, and the second display field 262 displays frames that make up the recorded video in chronological order. The user can select one or more frames and delete unnecessary frames. The user can also extract a portion of a frame and insert it after any frame. The editing processing unit 21 edits the recorded video in response to a user operation, and overwrites the edited recorded video or saves it separately.

[0192] <Operation or processing procedure> Next, the operation and processing performed by the recorded information display system 100 based on the above configuration will be described.

[0193] <<Saving recorded video>> The process of saving a recorded video will be described with reference to Fig. 37. Fig. 37 is an example of a sequence diagram showing a procedure in which the information recording application 41 records a panoramic image, a speaker image, and an application screen.

[0194] S1: The user starts the information recording application 41 on the terminal device 10 and connects the terminal device 10 (information recording application 41) to the information processing system 50. If the access token has expired, the display control unit 13 displays a login screen. The user inputs authentication information (e.g., a user ID and a password) for logging in to the tenant into the information recording application 41. The operation reception unit 12 of the information recording application 41 receives the input.

[0195] S2: The communication unit 11 of the information recording application 41 transmits a login request to the information processing system 50 by specifying authentication information.

[0196] S3: The communication unit 51 of the information processing system 50 receives the login request, and the authentication unit 52 authenticates the user based on the authentication information. Here, it is assumed that the authentication is successful. The communication unit 51 of the information processing system 50 transmits the access token 1 to the information recording application 41. Although not shown in the figure, thereafter, the communication unit 51 attaches the access token 1 to communications with the information processing system 50. The access token 1 is associated with the authority of the logged-in user.

[0197] S4: Similarly, the user also logs in to the storage service system 70. This is because the recorded information is stored in the storage service system 70. The user inputs authentication information (e.g., a user ID and a password) for logging in to the storage service system 70. The operation reception unit 12 of the information recording application 41 receives the input.

[0198] S5: The communication unit 11 of the information recording application 41 transmits a login request to the information processing system 50 by specifying authentication information.

[0199] S6: The communication unit 51 of the information processing system 50 receives the login request, and since it is a login request to the storage service system 70, transfers the login request to the storage service system 70.

[0200] S7: The storage service system 70 authenticates the user based on the authentication information. Here, it is assumed that the authentication is successful. The storage service system 70 transmits the access token 2 to the information processing system 50.

[0201] S8: The communication unit 51 of the information processing system 50 receives the access token 2 and transmits it to the information recording application 41. Although not shown in the figure, thereafter, the communication unit 51 attaches the access token 2 to communications with the storage service system 70. The access token 2 is associated with the authority of the logged-in user.

[0202] S21: Next, the user operates the remote conference application 42 to start a remote conference. Here, it is assumed that the remote conference application 42 of the own base 102 and the other base 101 start a remote conference. The remote conference application 42 of the own base 102 transmits an image captured by a camera of the terminal device 10 and an audio collected by a microphone to the remote conference application 42 of the other base 101. The remote conference application 42 of the other base 101 displays the received image on a display and outputs the received audio from a speaker. Similarly, the remote conference application 42 of the other base 101 transmits an image captured by a camera of the terminal device 10 and an audio collected by a microphone to the remote conference application 42 of the own base 102. The remote conference application 42 of the own base 102 displays the received image on a display and outputs the received audio from a speaker. Each remote conference application 42 repeats this to realize the remote conference.

[0203] S22: The user performs settings related to recording on the recording setting screen 210 of the information recording application 41 shown in Fig. 30. The operation acceptance unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that the camera toggle button 211 and the PC screen toggle button 212 are both on.

[0204] If the user has booked a remote conference in advance, the user can display a list of remote conferences by pressing the get information button 221 on the calendar in Fig. 33 and select a remote conference to which the recorded video is to be associated. Since the user has already logged in to the information processing system 50, the information processing system 50 identifies remote conferences that the logged-in user has viewing authority for. The information processing system 50 transmits a list of the identified remote conferences to the terminal device 10, so that the user can select a remote conference that is currently being held or will be held in the future. This determines information about the remote conference, such as the conference ID.

[0205] In addition, even if the user does not reserve a remote conference in advance, the user can create a conference when creating a recorded video. In the following, a case will be described in which the information recording application 41 creates a conference when creating a recorded video and obtains a conference ID from the information processing system 50.

[0206] S23: The user instructs the information recording application 41 to start recording (the start recording now button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0207] S24: Since the remote conference has not been selected (the conference ID has not been determined), the communication unit 11 of the information recording application 41 transmits a remote conference creation request to the information processing system 50.

[0208] S25: The communication unit 51 of the information processing system 50 receives the remote conference creation request, the conference information acquisition unit 54 acquires a unique conference ID assigned by the conference management system 9, and the communication unit 51 transmits the conference ID to the information recording application 41.

[0209] S26: The conference information acquisition unit 54 also transmits, via the communication unit 51, to the information recording application 41, the destination where the recorded video is to be saved (the URL of the storage service system 70).

[0210] S27: When the communication unit 11 of the information recording application 41 receives the conference ID and the destination to save the recorded file, the video storage unit 17 determines that recording is ready, and starts recording.

[0211] S28: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user from the application (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In FIG. 37, the application selected by the user is the remote conference application 42.

[0212] S29: The video storage unit 17 of the information recording application 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. When notifying, the video storage unit 17 may also notify that the camera toggle button 211 is on (request for a panoramic image and a speaker image). Regardless of the presence or absence of a request, the meeting device 60 transmits the panoramic image and the speaker image to the information recording application 41.

[0213] S30: When the terminal communication unit 61 of the meeting device 60 receives a recording start command, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or may be obtained from the information processing system 50.

[0214] S31: The voice acquisition unit 15 of the information recording application 41 acquires the voice data output by the terminal device 10 (voice data received by the remote conference application 42).

[0215] S32: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and a synthesis request to the meeting device 60.

[0216] S33: The terminal communication unit 61 of the meeting device 60 receives the voice data and the synthesis request, and the voice synthesis unit 65 synthesizes the surrounding voice data collected by the sound collection unit 64 with the received voice data. For example, the voice synthesis unit 65 adds the two pieces of voice data together. Since clear voices around the meeting device 60 are recorded, the accuracy of converting voices around the meeting device 60 (conference room side) into text is improved.

[0217] This voice synthesis is also possible in the terminal device 10. However, by distributing the recording function to the terminal device 10 and the voice processing to the meeting device 60, it is possible to reduce the load on the terminal device 10 and the meeting device 60. The recording function may be distributed to the meeting device 60 and the voice processing may be distributed to the terminal device 10.

[0218] S34: Furthermore, the panoramic image creating unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creating unit 63 creates a speaker image.

[0219] S35: The device communication unit 16 of the information recording application 41 repeatedly acquires the panoramic image and the speaker image from the meeting device 60. The device communication unit 16 also repeatedly requests and acquires the synthesized voice data from the meeting device 60. The acquisition of these may be performed by the device communication unit 16 making a request to the meeting device 60. Alternatively, the meeting device 60 that has received a notice that the camera toggle button 211 is on may automatically transmit the panoramic image and the speaker image. The meeting device 60 that has received a request to synthesize voice data may automatically transmit the synthesized voice data to the information recording application 41.

[0220] S36: The display control unit 13 of the information recording application 41 displays the application screen, the panoramic image, and the speaker image side by side on the recording screen 220. The video storage unit 17 of the information recording application 41 also stores the application screen, the panoramic image, and the speaker image acquired from the remote conference application 42 as separate recorded videos. That is, the video storage unit 17 creates each recorded video by specifying the repeatedly received application screen, the panoramic image, and the speaker image as frames that constitute the video. The video storage unit 17 also stores the audio data received from the meeting device 60.

[0221] The information recording application 41 repeats the above steps S31 to S36.

[0222] S37: When the remote conference ends and recording is no longer necessary, the user instructs the information recording application 41 to end recording (for example, by pressing the recording end button 227). The operation acceptance unit 12 of the information recording application 41 accepts the instruction.

[0223] S38: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 of the end of recording. The meeting device 60 continues creating the panoramic image and the speaker image and synthesizing the voice. However, the meeting device 60 may change the processing options, such as changing the resolution or fps, depending on whether recording is in progress.

[0224] S39: The video storage unit 17 of the information recording application 41 combines the audio data with the recorded video to create a combined image video with audio. If no recorded video is stored, the audio data may be independent. Also, there are three recorded videos: the application screen, the panoramic image, and the speaker image, and which recorded video is combined with which is determined in advance along with the priority order. Also, the recorded video and the audio data do not necessarily have to be combined.

[0225] S40: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the video recording setting screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. In detail, the audio data processing unit 18 specifies the URL of the save destination via the communication unit 11, and transmits a conversion request for the audio data combined with the recorded video to the information processing system 50 together with the conference ID and video recording ID.

[0226] S41: The communication unit 51 of the information processing system 50 receives a request to convert the voice data, and the text conversion unit 55 converts the voice data into text data using the voice recognition service system 80. The communication unit 51 saves the text data in the same storage destination (URL of the storage service system 70) as the storage destination of the recorded video. In the video recording information storage unit 5002, the text data is associated with the recorded video by the conference ID and the video recording ID. The text data may be managed by the conference information acquisition unit 54 of the information processing system 50 and stored in the storage unit 5000. In addition, the terminal device 10 may request voice recognition from the voice recognition service system 80, and the text data acquired from the voice recognition service system 80 may be saved in the storage destination. In addition, the voice recognition service system 80 returns the converted text data to the information processing system 50, but may also directly send it to the URL of the storage destination. The voice recognition service system 80 may select or switch from multiple services according to the setting information set by the user in the information processing system 50.

[0227] S42: Furthermore, the upload unit 20 of the information recording application 41 saves the recorded video in the save destination for the recorded video via the communication unit 11. In the video recording information storage unit 5002, the recorded video is associated with the conference ID and the video recording ID. The recorded video is recorded as having been uploaded.

[0228] The save destination is notified to the user, so the user can share the recorded video with participants by informing them of the save destination by email, etc. Even if the recorded video, audio data, and text data are created using different devices, they can be collected and stored in one storage location, making them easy for users to view later.

[0229] In the case of real-time voice recognition, the meeting device 60 or the terminal device 10 transmits voice data in real time to the information processing system 50. The terminal device 10 displays the text data transmitted from the meeting device 60 or returned from the information processing system 50 on the recording screen 220 and stores it.

[0230] The processing of steps S31 to S36 does not have to be performed in the order shown in FIG. 37, and the synthesis of the audio data and the saving of the recorded moving image may be performed in the reverse order.

[0231] <Major Effects> By providing the upper canopy 632, the meeting device 60 of this embodiment can reduce the effect of direct light from lighting installed on the ceiling in the office on the seams of the panoramic image. Since the processing load of reducing flare and ghosting is reduced by performing image processing using software, the terminal device 10 can display a natural panoramic image in real time, even in a video. By taking structural measures in the meeting device 60, the software processing load can be reduced, and the sense of incongruity at the seams can be reduced even in devices that require real-time performance, such as web cameras.

[0232] In addition, by adjusting the incidence angle θ to 30° or more, the angle of view necessary for capturing images of people is ensured, while by limiting the incidence angle to 67° or less, the effect of direct light from the lighting installed on the ceiling in the office on the seams of the panoramic image is reduced, making it possible to capture images of people.

[0233] In addition, the impact of direct light from lighting installed on the ceiling in the office near the end of the lens, where the seam is located, can be reduced more effectively than in the front direction of the lens, thereby making it possible to more effectively reduce the adverse effects on the seams of the panoramic image.

[0234] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0235] For example, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. The meeting device 60 may be an omnidirectional camera, a microphone, and a speaker connected by cables.

[0236] Furthermore, the meeting device 60 can distribute video to the terminal devices 10 at the same location. The meeting device 60 broadcasts video to the terminal devices 10 connected to the same SSID, for example.

[0237] Also, a meeting device 60 may be placed at another site 101. The other site 101 creates combined image video and text data using a separate meeting device 60. Also, a plurality of meeting devices 60 may be placed at one site. In this case, a plurality of pieces of record information are created for each meeting device 60.

[0238] The layout of the panoramic image 203, the speaker image 204, and the application screen in the recorded video used in this embodiment is merely an example. The panoramic image 203 may be on the bottom and the speaker image 204 on the top, or the user may change the layout, or may individually switch between displaying and not displaying the panoramic image 203 and the speaker image 204 when displaying the recorded video.

[0239] In addition, the configuration examples in FIG. 7 and the like are divided according to main functions in order to facilitate understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the terminal device 10, the meeting device 60, and the information processing system 50 can be divided into more processing units according to the processing contents. Also, the processing units can be divided so that they include more processes. For example, a system for creating recorded videos and a system for displaying recorded videos may be separate.

[0240] Additionally, the devices described in the examples are merely one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the information processing system 50 includes a plurality of computing devices, such as a server cluster. The computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processes disclosed herein.

[0241] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in FIG. 37, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Also, the information processing system 50 may be integrated into one server device, or may be divided into multiple devices.

[0242] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a device such as a conventional circuit module designed to execute each function described above.

[0243] <Additional Notes> [Appendix 1] An imaging device capable of capturing an image with a field of view of at least 360° in the horizontal direction, A plurality of lenses each having an optical axis in any horizontal direction; and an upper eaves that blocks light incident on the plurality of lenses from directly above, An imaging device characterized in that the upper eaves limits the angle of incidence of light between the direction in which light from above enters the lens at the maximum angle and the optical axis of the lens to between 30° and 67°. [Appendix 2] The imaging device described in Appendix 1, characterized in that the shape of the upper visor when viewed from above is such that the angle of incidence in the direction of the lens end is smaller than the angle of incidence in the optical axis direction. [Appendix 3] 3. The imaging device according to claim 2, wherein the upper eaves limits the angle of incidence toward the lens end to 30° or more and 45° or less. [Appendix 4] 3. The imaging device according to claim 2, wherein a ratio of an incident angle in a direction toward an end of the lens to the incident angle in the optical axis direction is 2 or less. [Appendix 5] The imaging device described in any one of appendix 1 to 4, wherein the shape of the side of the upper canopy from the upper canopy protrusion portion on the outer edge of the upper canopy to the multiple lenses is curved inward. [Appendix 6] The imaging device described in any one of appendix 1 to 4, wherein a shape of a side surface of the upper canopy from an upper canopy protrusion portion on an outer edge of the upper canopy to the plurality of lenses is a stepped shape. [Appendix 7] 7. The imaging device according to claim 1, wherein a shape of the upper canopy when viewed from above is a circle, an ellipse, a rectangle, or an octagon. [Appendix 8] A terminal device connected to an imaging device according to any one of claims 1 to 7, a display control unit that displays an image captured by the imaging device; a video storage unit for storing the image; A program that functions as a [Explanation of symbols]

[0244] 10 Terminal Equipment 50 Information Processing Systems 60 Meeting Devices 100 Recording Information Display System

Prior Art Documents

Patent Documents

[0245]

Patent Document 1

Claims

1. An imaging device capable of capturing an image with a field angle of at least about 360° in the horizontal direction, A plurality of lenses each having an optical axis in a horizontal direction; and an upper eaves that blocks light incident on the plurality of lenses from directly above, An imaging device characterized in that the upper eaves limits the angle of incidence of light between the direction in which light from above is incident on the lens at the maximum angle and the optical axis of the lens to between approximately 30° and approximately 67°.

2. 2 . The imaging device according to claim 1 , wherein the shape of the upper visor in a top view is such that the angle of incidence in a direction toward a lens end is smaller than the angle of incidence in a direction toward an optical axis.

3. 3. The imaging device according to claim 2, wherein the upper eaves limits the angle of incidence toward the lens end to about 30 degrees or more and about 45 degrees or less.

4. 3. The imaging device according to claim 2, wherein a ratio of an angle of incidence in a direction toward an end of the lens to the angle of incidence in the optical axis direction is 2 or less.

5. 5. The imaging device according to claim 1, wherein a shape of a side surface of the upper canopy from an upper canopy protrusion portion on an outer edge of the upper canopy to the plurality of lenses is curved inward.

6. 5. The imaging device according to claim 1, wherein a shape of a side surface of the upper canopy from an upper canopy protrusion portion on an outer edge of the upper canopy to the plurality of lenses is a stepped shape.

7. 2. The imaging device according to claim 1, wherein a shape of the upper canopy when viewed from above is a circle, an ellipse, a rectangle, or an octagon.

8. A terminal device connected to the imaging device according to claim 1, a display control unit that displays an image captured by the imaging device; a video storage unit for storing the image; A program that functions as a

9. A panoramic image creation method performed by an imaging device capable of capturing an image with a field angle of at least about 360° in the horizontal direction, comprising: The imaging device includes a plurality of lenses each having an optical axis in a horizontal direction; and an upper eaves that blocks light incident on the plurality of lenses from directly above, The upper eaves limit the angle of incidence of light between the direction in which light from above is incident on the lens at the maximum angle and the optical axis of the lens to about 30° or more and 67° or less; A process of generating an image having a horizontal direction of greater than 180° using a plurality of image pickup elements corresponding to the plurality of lenses; A process of determining a most similar joint portion by pattern matching an edge portion of an image generated by one imaging element with an edge portion of an image generated by the other imaging element; A process of combining a plurality of images in the combining unit to create a panoramic image of approximately 360° in the horizontal direction; A method for creating panoramic images.

Images