Display system, method, and program

Abstract

To stop display of a video image when a change in a video image is small.SOLUTION: A display system according to an embodiment of the present invention, is a display system containing a projecting device and a terminal. The projecting device comprises: a photographing part that photographs the video image; an image processing part that creates the video image for a display on the basis of the video image photographed and a video image layout designated from a terminal; a change ratio calculation part that calculates a change ratio of the video image of a specific region in the video image designated and photographed from the terminal; and a communication part that transmits the video image for the display and the change ratio to the terminal. The terminal comprises a video image layout having a fixing display of the video image of the specific region and a video image layout determination part that selects any one of the video image layout with no fixing display of the video image of the specific region on the basis of the change ratio received from the photographing device.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a display system, a method, and a program. [Background technology]

[0002] Conventionally, there is known a web conference in which video and audio are communicated two-way between terminals installed at multiple locations. Video captured at each location is displayed on the terminals (Patent Document 1). This allows the participants in the conference to share the atmosphere of the conference. Summary of the Invention [Problem to be solved by the invention]

[0003] However, when the changes in the image are small (ie the object being photographed is not changing), it is preferable to display another image (eg an image in which the object being photographed is changing).

[0004] Therefore, an object of the present invention is to not continue displaying an image when the change in the image is small. [Means for solving the problem]

[0005] A display system according to one embodiment of the present invention is a display system including a photographing device and a terminal, wherein the photographing device comprises a photographing unit that photographs video, an image processing unit that creates a video for display based on the photographed video and a video layout specified by the terminal, a change rate calculation unit that calculates a rate of change of a video of a specific area within the photographed video specified by the terminal, and a communication unit that transmits the video for display and the change rate to the terminal, and the terminal comprises a video layout determination unit that selects, based on the change rate received from the photographing device, either a video layout with a fixed display of the video of the specific area or a video layout without a fixed display of the video of the specific area. Effect of the Invention

[0006] According to the present invention, an image is not continually displayed when the changes in the image are small. [Brief description of the drawings]

[0007] [Figure 1] 1 is an overall configuration diagram according to an embodiment of the present invention; [Diagram 2] 1 is a diagram for explaining the imaging direction of an imaging device according to an embodiment of the present invention; [Diagram 3] 1 is a diagram for explaining conversion of a captured image by a photographing device according to an embodiment of the present invention. FIG. [Figure 4] 1 is a hardware configuration diagram of a photographing device according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a hardware configuration diagram of a terminal (personal computer) according to one embodiment of the present invention. [Figure 6] FIG. 2 is a hardware configuration diagram of a terminal (video conference terminal) according to an embodiment of the present invention. [Figure 7] 1 is a functional block diagram of a display system according to an embodiment of the present invention. [Figure 8] FIG. 2 is a diagram for explaining a video layout according to an embodiment of the present invention. [Figure 9] FIG. 11 is a diagram for explaining a rate of change according to an embodiment of the present invention. [Figure 10] 10 is a flowchart showing a process in which the photographing device according to the embodiment of the present invention notifies a rate of change of an image in a fixed display area. [Figure 11] 10 is a flowchart showing a process in which a terminal according to an embodiment of the present invention determines a video layout. [Figure 12] 10 is a flowchart showing a process in which a terminal according to an embodiment of the present invention determines a video layout. [Figure 13] 13 is an example of an operation screen displayed on a terminal according to an embodiment of the present invention. [Figure 14] 13 is an example of a fixed display area designation screen displayed on a terminal according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] <Overall composition> Fig. 1 is a diagram showing the overall configuration of an embodiment of the present invention. A web conference system (or a video conference system) 1 includes terminals installed at multiple locations (in the example of Fig. 1, a terminal 12 at location 1 and a terminal 20 at location 2) and a server 30.

[0010] [Base 1] In one embodiment of the present invention, a terminal 12 and a photographing device 11 are installed at at least one base of a web conference (hereinafter, a system including the terminal 12 and the photographing device 11 is also referred to as a display system). The display system 10 may include an electronic whiteboard 13.

[0011] [Device] The terminal 12 is a terminal used to participate in a web conference. The terminal 12 acquires video data and audio data from the photographing device 11 and transmits the video data and audio data to the server 30. The terminal 12 also receives from the server 30 the video data and audio data acquired by the server 30 from the terminal 20 at the site 2. For example, the terminal 12 is a computer such as a personal computer, a tablet, a smartphone, or a terminal for video conferences.

[0012] [Filming equipment] The photographing device 11 is a device equipped with a photographing function (camera). The photographing device 11 may also be equipped with a sound collection function (microphone) and a sound output function (speaker). The photographing device 11 can transmit and receive data to and from the terminal 12 (for example, the photographing device 11 and the terminal 12 are connected via a USB (Universal Serial Bus)).

[0013] Fig. 2 is a diagram for explaining the photographing direction of the photographing device 11 according to one embodiment of the present invention. As shown in Fig. 2, the photographing device 11 can photograph a 360-degree direction (around the photographing device 11) on the horizontal plane, and can photograph a direction at a predetermined angle (e.g., a=60 degrees, b=30 degrees) above and below the horizontal (0 degrees) on the vertical plane.

[0014] Fig. 3 is a diagram for explaining conversion of a video captured by the photographing device 11 according to an embodiment of the present invention. As shown in Fig. 3, the photographing device 11 can convert a video captured by the photographing device 11 (for example, a video captured in the shooting direction described in Fig. 2) into a panoramic video (hereinafter, also referred to as a "360-degree panoramic video"). Furthermore, the photographing device 11 can cut out a specific area (for example, an area in which a speaker is being photographed, an area specified by the terminal 12 (for example, an area in which a document displayed on the electronic whiteboard 13 is being photographed)) from the 360-degree panoramic video.

[0015] [Electronic whiteboard] The electronic whiteboard 13 is a display device that displays materials for web conferences and the like.

[0016] [Base 2] At a location other than the location where the display system 10 is installed (location 2 in the example of Figure 1), only a terminal with a photographing function may be installed as in Figure 1, or a terminal and photographing equipment with a photographing function may be installed.

[0017] The photographing device 11 and the terminal 20 may be devices equipped with a communication function. For example, the photographing device 11 and the terminal 20 may be an output device such as a PJ (Projector), an IWB (Interactive White Board: a white board having an electronic blackboard function capable of mutual communication), a digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collecting device, a medical device, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, or a desktop PC.

[0018] [server] The server 30 can transmit and receive data to and from terminals at each base via the communication network 40. The server 30 transfers video data and audio data acquired from a terminal at a base to a terminal at another base.

[0019] Specifically, the server 30 acquires video data and audio data (specifically, video data and audio data acquired by the terminal 12 from the photographing device 11) from the terminal 12 at the site 1, and transfers the video data and audio data to the terminal 20 at the site 2. Similarly, the server 30 acquires video data and audio data (specifically, video data photographed by the terminal 20 and audio data collected by the terminal 20) from the terminal 20 at the site 2, and transfers the video data and audio data to the terminal 12 at the site 1.

[0020] The devices described in the example are merely representative of one of several computing environments for implementing the embodiments disclosed herein.

[0021] In one embodiment, server 30 includes multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communications link, including a network, shared memory, etc., to perform the processes disclosed herein.

[0022] <Hardware configuration> With reference to Figure 4, the hardware configuration of the photographing equipment 11 will be described, with reference to Figure 5, the hardware configuration of the terminals (personal computers) 12, 20 and the server 30 will be described, and with reference to Figure 6, the hardware configuration of the terminals (video conferencing terminals) 12, 20 will be described.

[0023] [Filming equipment] FIG. 4 is a hardware configuration diagram of a photographing device (a meeting device capable of photographing 360-degree video) 11 according to an embodiment of the present invention.

[0024] In the following, the photographing device 11 is assumed to be a device that uses an imaging element to capture a 360-degree video around the device at a predetermined height, but the number of imaging elements may be one or more than one. Also, it does not necessarily have to be a dedicated device, and a personal computer, digital camera, smartphone, etc. may have substantially the same function by attaching a 360-degree video imaging unit to it after the fact.

[0025] As shown in FIG. 4, the photographing equipment 11 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 106, a sound processing unit 107, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an external device connection I / F 116, a communication unit 117, an antenna 117a, an audio sensor 118, and a speaker 119.

[0026] Of these, the imaging unit 101 includes a wide-angle lens (so-called fisheye lens) 102 having a 360-degree angle of view for forming a hemispherical image, and imaging elements 103 (image sensors) provided in correspondence with each wide-angle lens. The imaging elements 103 include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image captured by the fisheye lens 102 into image data of an electrical signal and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers in which various commands and parameters necessary for the operation of the imaging element are set.

[0027] The imaging elements 103 (image sensors) of the imaging unit 101 are connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103 of the imaging unit 101 are connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 107 are connected to a CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an external device connection I / F (Interface) 116, a communication unit 117, an audio sensor 118, a speaker 119, and the like.

[0028] The image processing unit 104 takes in the image data output from the image sensor 103 via a parallel I / F bus, performs a predetermined process on each image data, and creates data for a panoramic image and a speaker image from the fisheye image. It also synthesizes the panoramic image and the speaker image, etc., and outputs one video.

[0029] The imaging control unit 105 generally uses an I2C bus to set commands and the like in the registers of the imaging element 103, with the imaging control unit 105 acting as a master device and the imaging element 103 acting as a slave device. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 103 and send it to the CPU 111.

[0030] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the imaging start button on the operation unit 115 is pressed or when an imaging start instruction is received from a personal computer. Some imaging devices 11 have a preview display function or a function corresponding to video display on a display (for example, a display of a personal computer or a smartphone). In this case, the image data is output from the imaging element 103 continuously at a predetermined frame rate (frames / minute).

[0031] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging element 103 in cooperation with the CPU 111. In this embodiment, the photographing device 11 is not provided with a display, but a display unit may be provided.

[0032] The microphone 106 converts sound into sound (signal) data. The sound processing unit 107 takes in the sound data output from the microphone 106 via the I / F bus, and performs a predetermined process on the sound data.

[0033] The speaker 119 converts the input voice data into voice. The microphone 106 and the speaker 119 are an example of an input / output unit. The input / output unit does not need to be integrated into a single configuration. Instead, the input means and the output means may be configured separately.

[0034] The CPU 111 controls the overall operation of the photographing device 11 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104, and data of a processed equirectangular projection image.

[0035] The operation unit 115 is a general term for operation buttons such as a shooting start button 115a. By operating the operation unit 115, the user can start shooting or recording, as well as turn the power on / off, establish a communication connection, and input settings such as various shooting modes and shooting conditions.

[0036] The external device connection I / F 116 is an interface for connecting various external devices. In this case, the external device is, for example, a PC (Personal Computer) or the like. The video data and image data stored in the DRAM 114 are transmitted to an external terminal (device) via this network I / F 116, or recorded on an external medium.

[0037] The communication unit 117 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via the antenna 117a provided in the photographing device 11, and transmit the stored video data and image data to the cloud server. Also, the communication unit 117 may communicate with nearby devices using a short-range wireless communication technology such as BLE or NFC.

[0038] The sound sensor 118 is a sensor that acquires 360-degree sound information in order to identify from which direction a loud sound is input within 360 degrees around (horizontal plane) the photographing device 11. The sound processing unit 107 identifies the strongest direction based on the input 360-degree sound parameters and outputs the sound input direction within 360 degrees.

[0039] Note that direction, position, angle, acceleration, etc. may be calculated using other sensors (direction / acceleration sensors, GPS, etc.) and used for image correction and adding location information.

[0040] In addition, the image processing unit 104 performs the following processes. A panoramic image is generated in the following way. Raw data input from an image sensor that inputs spherical images is subjected to predetermined camera image processing such as Bayer conversion (RGB complementation processing) to create a fisheye image (curved image). Furthermore, DeWarp processing (distortion correction processing) is performed on the generated fisheye image (curved image) to generate a panoramic image (flat image) that shows 360 degrees around the imaging device 11. The speaker video is generated in the following way. The image capturing device 11 creates a speaker video by cutting out the speaker from a panoramic video (flat video) that captures 360 degrees of the surroundings. The voice input direction identified from the 360 ​​degrees output using the audio sensor 118 and the audio processing unit 107 is set as the speaker's direction, and the speaker video is cut out from the panoramic video. The method for cutting out a person's image from the voice input direction is to cut out a 30-degree area centered on the identified voice direction from the 360 ​​degrees, and perform face detection within that area to cut out the image. Furthermore, from the cut-out speaker videos, the video of a specific number of speakers (e.g., three people) who have most recently spoken is identified. By positioning this panoramic image above and below (or next to) one or more speaker images, a video is created as meeting record information based on the 360-degree footage.

[0041] [Terminal (Personal Computer)] 5 is a hardware configuration diagram of the terminal 12 (in the case of a personal computer) according to an embodiment of the present invention. The same applies to the terminal 20 and the server 30.

[0042] As shown in FIG. 5, the terminal 12 is constructed by a computer, and as shown in FIG. 5, it is equipped with a CPU 201, a ROM 202, a RAM 203, an HD 204, an HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 207, a network I / F 208, a data bus 209, a keyboard 210, a pointing device 211, a DVD-RW (Digital Versatile Disk Rewritable) drive 213, and a media I / F 215.

[0043] Among these, the CPU 201 controls the operation of the entire terminal 12. The ROM 202 stores programs such as an IPL used to drive the CPU 201. The RAM 203 is used as a work area for the CPU 201. The HD 204 stores various data such as programs. The HDD controller 205 controls reading and writing of various data from and to the HD 204 under the control of the CPU 201. The display 206 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 207 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 208 is an interface for data communication using the communication network 40. The bus line 209 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201 shown in FIG. 5.

[0044] The keyboard 210 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 211 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 213 controls reading and writing of various data from a DVD-RW 212 as an example of a removable recording medium. Note that the medium is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 215 controls reading and writing (storing) of data from a recording medium 214 such as a flash memory.

[0045] [Terminal (Videoconferencing terminal)] 6 is a hardware configuration diagram of the terminal 12 (in the case of a video conference terminal) according to an embodiment of the present invention. The same is true for the terminal 20.

[0046] 6, the video conference terminal 12 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a media I / F 307, an operation button 308, a power switch 309, a bus line 310, a network I / F 311, a camera 312, an image sensor I / F 313, a microphone 314, a speaker 315, an audio input / output I / F 316, a display I / F 317, an external device connection I / F (Interface) 318, a short-range communication circuit 319, and an antenna 319a of the short-range communication circuit 319. Of these, the CPU 301 controls the operation of the entire video conference terminal 12. The ROM 302 stores a program used to drive the CPU 301, such as an IPL. The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data, such as communication programs, image data, and sound data. The SSD 305 controls reading and writing of various data from and to the flash memory 304 under the control of the CPU 301. Note that a HDD may be used instead of the SSD. The media I / F 307 controls reading and writing (storing) of data from and to a recording medium 306 such as a flash memory. The operation button 308 is a button that is operated when selecting a destination of the videoconference terminal 12, etc. The power switch 309 is a switch for switching the power of the videoconference terminal 12 ON / OFF.

[0047] The network I / F 311 is an interface for data communication using a communication network 40 such as the Internet. The CMOS (Complementary Metal Oxide Semiconductor) sensor 312 is a type of built-in imaging means for capturing an image of a subject under the control of the CPU 301 to obtain image data. It should be noted that the imaging means may be a CCD (Charge Coupled Device) sensor or the like, instead of a CMOS sensor. The imaging element I / F 313 is a circuit for controlling the driving of the CMOS sensor 312. The microphone 314 is a built-in circuit for converting sound into an electric signal. The speaker 315 is a built-in circuit for converting an electric signal into a physical vibration to generate sound such as music or voice. The sound input / output I / F 316 is a circuit for processing input / output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I / F 317 is a circuit for transmitting image data to an external display under the control of the CPU 301. The external device connection I / F 318 is an interface for connecting various external devices. The short-range communication circuit 319 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark).

[0048] 6. Moreover, the bus line 310 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 301 shown in FIG.

[0049] The display is a type of display means configured with liquid crystal or organic EL (Electro Luminescence) for displaying an image of a subject, operation icons, etc. The display is connected to the display I / F 117 by a cable. This cable may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0050] The CMOS (Complementary Metal Oxide Semiconductor) sensor 312 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 301 to obtain image data. Instead of the CMOS sensor, an imaging means such as a CCD (Charge Coupled Device) sensor may be used. External devices such as an external camera, an external microphone, and an external speaker can be connected to the external device connection I / F 318 via a USB cable or the like. When an external camera is connected, the external camera is driven in preference to the built-in CMOS sensor 312 under the control of the CPU 301. Similarly, when an external microphone or an external speaker is connected, the external microphone or the external speaker is driven in preference to the built-in microphone 314 or the built-in speaker 315, respectively, under the control of the CPU 301.

[0051] Moreover, the recording medium 306 is configured to be detachable from the video conference terminal 12. Moreover, as long as it is a non-volatile memory that reads or writes data under the control of the CPU 301, it is not limited to the flash memory 304, and an EEPROM or the like may be used.

[0052] <Functional configuration> FIG. 7 is a functional block diagram of a display system 10 according to an embodiment of the present invention.

[0053] [Filming equipment] The photographing device 11 includes a photographing unit 1001, an image processing unit 1002, a change rate calculation unit 1003, and a communication unit 1004. The photographing device 11 functions as the photographing unit 1001, the image processing unit 1002, the change rate calculation unit 1003, and the communication unit 1004 by executing a program.

[0054] The image capturing unit 1001 captures an image. For example, the image capturing unit 1001 can capture an image in a 360-degree direction (around the image capturing device 11) on the horizontal plane, and can capture an image in a direction at a predetermined angle above and below the horizontal (0 degrees) on the vertical plane.

[0055] The image processing unit 1002 creates a display image based on the image captured by the image capturing unit 1001 and the image layout specified by the terminal 12. Specifically, the image processing unit 1002 arranges the image captured by the image capturing unit 1001 (for example, a panoramic image and an image cut out from the panoramic image) according to the image layout specified by the terminal 12. This display image is displayed on the terminal 12 and the terminal 20 that is holding a web conference with the terminal 12.

[0056] For example, the image processing unit 1002 can convert an image captured by the image capturing unit 1001 (for example, an image captured in the shooting direction described in FIG. 2) into a panoramic image (a 360-degree panoramic image). The image processing unit 1002 can also cut out a specific area (for example, an area in which a speaker is being shot, or an area specified by the terminal 12 (for example, an area in which a document displayed on the electronic whiteboard 13 is being shot)) from the panoramic image.

[0057] The change rate calculation unit 1003 calculates the change rate of an image of a specific area (also called a fixed display area) in the image captured by the imaging unit 1001, which is specified by the terminal 12.

[0058] The communication unit 1004 transmits the image for display created by the image processing unit 1002 and the change rate calculated by the change rate calculation unit 1003 to the terminal 12. The communication unit 1004 also receives from the terminal 12 a designation of a specific area (fixed display area) in the image captured by the imaging unit 1001 and a designation of the image layout.

[0059] [Device] The terminal 12 includes a video layout determination unit 2001, a display unit 2002, a communication unit 2003, and an information recording unit 2004. The terminal 12 functions as the video layout determination unit 2001, the display unit 2002, the communication unit 2003, and the information recording unit 2004 by executing a program.

[0060] The video layout determination unit 2001 selects either a video layout with fixed display of the video in a specific area (fixed display area) or a video layout without fixed display of the video in the specific area (fixed display area) based on the change rate received from the photographing equipment 11 (i.e., the change rate of the video in a specific area (fixed display area) in the video photographed by the photographing equipment 11).

[0061] For example, in the case of a video layout in which the rate of change is below a certain value for a certain period of time and there is fixed display of video in a specific area (fixed display area) (i.e., fixed display is being performed at that time), the video layout determination unit 2001 changes the video layout to one in which there is no fixed display of video in a specific area (fixed display area).

[0062] For example, if the rate of change over a certain period of time is equal to or greater than a certain value and the video layout does not have a fixed display of video in a specific area (fixed display area) (i.e., no fixed display is being performed at that time), the video layout determination unit 2001 changes the video layout to one that has a fixed display of video in a specific area (fixed display area).

[0063] The video layout determination unit 2001 can receive a designation of whether or not to change the video layout based on the rate of change, and can automatically change the video layout when the designation of change is received. The video layout determination unit 2001 can also receive and use a designation of a fixed time and a fixed value. The video layout determination unit 2001 can also receive a designation of a specific area (fixed display area).

[0064] The display unit 2002 displays the image for display created by the photographing device 11 .

[0065] The communication unit 2003 receives from the photographing equipment 11 the image for display created by the photographing equipment 11 and the rate of change calculated by the photographing equipment 11. In addition, the communication unit 2003 transmits to the photographing equipment 11 the designation of a specific area (fixed display area) in the image photographed by the photographing equipment 11 and the designation of the image layout (i.e., the image layout selected by the image layout determination unit 2001).

[0066] The information recording unit 2004 stores information on the rate of change, the time when the rate of change was received, and information on whether or not fixed display is being performed at that time.

[0067] [Video Layout] FIG. 8 is a diagram for explaining a video layout according to an embodiment of the present invention.

[0068] The video layout includes a video layout without fixed display of a video of a specific region (when the "Fixed Display Setting" in FIG. 8 is "Fixed Display OFF") and a video layout with fixed display of a video of a specific region (when the "Fixed Display Setting" in FIG. 8 is "Fixed Display ON"). For example, the video layout without fixed display of a video of a specific region (when the "Fixed Display Setting" in FIG. 8 is "Fixed Display OFF") may be set as the default value.

[0069] The video layout in which there is no fixed display of a specific area of ​​video (fixed display OFF) includes a “two-stage panorama” video layout and a “speaker focus” video layout. For example, the “speaker focus” video layout may be set as the default value.

[0070] In the "two-tier panoramic" video layout, a 360-degree panoramic image is displayed in two tiers (two 180-degree panoramic images are displayed).

[0071] The "speaker focus" video layout displays a 360-degree panoramic image and an image of the speaker (the speaker is detected and the image of the speaker is cut out from the panoramic image).

[0072] The video layout with fixed display of a specific area (fixed display ON) includes a video layout of "fixed display" and a video layout of "fixed display + speaker frame." For example, the video layout of "fixed display + speaker frame" may be set as the default.

[0073] In the "fixed display" video layout, a 360-degree panoramic image and a fixed image (image of a specific area (fixed display area)) are displayed.

[0074] The "fixed display + speaker frame" video layout displays a 360-degree panoramic image, a fixedly displayed image (image of a specific area (fixed display area)), and an image of the speaker (the speaker is detected and the speaker's image is cut out from the panoramic image).

[0075] [Rate of change] 9 is a diagram for explaining the rate of change according to an embodiment of the present invention. The rate of change is the ratio of the number of pixels that differ between a certain image (image before the change) and an image after a certain time (image after the change) to the total number of pixels.

[0076] For example, suppose that conference material A is photographed in the fixed display area. If conference material A is still photographed after a predetermined time (e.g., 1 second), the rate of change is 0 percent. If conference material B, which is different from conference material A, is photographed after a predetermined time (e.g., 1 second), the rate of change is calculated as the ratio (e.g., 6 percent) of the number of pixels that differ between the video of conference material A and the video of conference material B to the total number of pixels.

[0077] <Method> Each processing method will be described with reference to FIGS.

[0078] FIG. 10 is a flow chart showing a process in which the photographing device 11 notifies the user of the rate of change of the image in the fixed display area according to an embodiment of the present invention.

[0079] In step 11 (S11), the photographing unit 1001 of the photographing device 11 captures an image.

[0080] In step 12 (S12), the change rate calculation unit 1003 of the photographing device 11 calculates the change rate of the image of the fixed display area from among the images photographed in S11.

[0081] In step 13 (S13), the communication unit 1004 of the photographing device 11 transmits the rate of change calculated in S12 to the terminal 12.

[0082] While the imaging device 11 is imaging, steps S11 to S13 are continuously executed.

[0083] FIG. 11 is a flowchart showing a process in which the terminal 12 determines a video layout according to an embodiment of the present invention.

[0084] In step 21 (S21), the communication unit 2003 of the terminal 12 receives, from the photographing device 11, a rate of change in an image of a specific area (fixed display area) in an image photographed by the photographing device 11. The information recording unit 2004 stores the rate of change and information on the time when the rate of change was received.

[0085] In step 22 (S22), the video layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 to determine whether the fixed display is set (ON) or not (OFF). If the fixed display is set (ON), the process proceeds to step 23, and if the fixed display is not set (OFF), the process ends.

[0086] In step 23 (S23), the video layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 and judges whether or not all the change rates within the most recent fixed time B (e.g., 2 minutes) are equal to or less than a certain value C (e.g., 3 percent). If all the change rates within the most recent fixed time B are equal to or less than the certain value C, the process proceeds to step 24, and if the change rates within the most recent fixed time B are not equal to or less than the certain value C, the process ends.

[0087] In step 24 (S24), the video layout determination unit 2001 of the terminal 12 turns off the setting of fixed display (that is, changes to a video layout without fixed display of a specific area of ​​video). After that, the image capture device 11 is notified of the video layout.

[0088] The terminal 12 may display a message to the effect that the video layout has been automatically changed.

[0089] Fig. 12 is a flowchart showing a process in which the terminal 12 according to an embodiment of the present invention determines a video layout. In Fig. 12, the fixed display setting can be turned ON after being turned OFF.

[0090] In step 31 (S31), the communication unit 2003 of the terminal 12 receives, from the photographing device 11, a rate of change in an image of a specific area (fixed display area) in an image photographed by the photographing device 11. The information recording unit 2004 stores the rate of change and information on the time when the rate of change was received.

[0091] In step 32 (S32), the video layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 to determine whether the fixed display is set (ON) or not (OFF). If the fixed display is not set (OFF), the process proceeds to step 33, and if the fixed display is set (ON), the process proceeds to step 35.

[0092] In step 33 (S33), the video layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 and judges whether the rate of change is equal to or greater than a certain value C (e.g., 3 percent). If the rate of change is equal to or greater than the certain value C, the process proceeds to step 34, and if the rate of change is not equal to or greater than the certain value C, the process ends.

[0093] In step 34 (S34), the video layout determination unit 2001 of the terminal 12 turns on the setting of fixed display (i.e., changes to a video layout with fixed display of a specific area of ​​video (for example, the video layout before the change stored in the information recording unit 2004 in S36, or a default video layout)). After that, the image capture device 11 is notified of the video layout.

[0094] In step 35 (S35), the video layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 and judges whether or not all the change rates in the most recent fixed time B (e.g., 2 minutes) are equal to or less than a certain value C (e.g., 3 percent). If all the change rates in the most recent fixed time B are equal to or less than the certain value C, the process proceeds to step 36, and if the change rates in the most recent fixed time B are not equal to or less than the certain value C, the process ends.

[0095] In step 36 (S36), the video layout determination unit 2001 of the terminal 12 turns off the setting of fixed display (i.e., changes to a video layout without fixed display of a specific area of ​​video). Then, the video layout determination unit 2001 of the terminal 12 stores information of the video layout before the change in the information recording unit 2004. After that, the image capture device 11 is notified of the video layout.

[0096] The terminal 12 may display a message to the effect that the video layout has been automatically changed.

[0097] <User Interface> The screens displayed on the terminal 12 will be described with reference to FIGS.

[0098] Fig. 13 is an example of an operation screen displayed on the terminal 12 according to an embodiment of the present invention. The operator of the terminal 12 can set the video layout and the fixed display on the screen of Fig. 13.

[0099] (1) The application program name is displayed. Select (2) to maximize, minimize, or close the screen. By selecting (3), a screen for operating the photographing device 11 can be displayed. By selecting (4), the setting screen for the photographic device 11 can be displayed. (5) The name of the photographic device 11 is displayed. Select (6) to display help. (7) shows a display screen created by the photographing device 11. · (8) When you select it, you can change the video layout (for example, change to a desired video layout from the video layouts in FIG. 8). By selecting (9), the fixed display area specification screen shown in FIG. 14 can be displayed. Selecting (10) allows you to change the front of the image captured by the camera 11. Select (11) to enable or disable (mute) the microphone. Select (13) to increase or decrease the volume. Selecting (14) will display information about the application program. Selecting (15) will display license information. Select (16) to display the meeting list screen. Selecting (17) will display the application program settings screen. Selecting (18) allows you to check the name of the user logged in to the application program or to log out. Select (19) to record the web conference.

[0100] Fig. 14 is an example of a fixed display area designation screen displayed on the terminal 12 according to an embodiment of the present invention. On the screen of Fig. 14, the operator of the terminal 12 can designate a specific area of ​​the image captured by the photographing device 11 as the fixed display area.

[0101] (1) shows an explanation of the fixed display. In (2), you can choose whether or not to display the information fixedly. In (3), a fixed display area can be specified on the image captured by the image capturing device 11. In (4), the size of the fixed display area specified in (3) can be changed. If the "Cancel" button is selected in (5), the fixed display is not set and the screen returns to the operation screen of FIG. When the "OK" button is selected in (6), the fixed display is set and the screen returns to the operation screen of FIG.

[0102] <Effects> In one embodiment of the present invention, when there is little change in the image (for example, when the meeting materials displayed on the electronic whiteboard are fixed and the meeting materials do not change as the conversation gets lively or goes off topic), the image is not continuously displayed, and other images (for example, images that share the atmosphere of the meeting, such as the facial expressions of the speaker and other participants) are automatically displayed.

[0103] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above. [Explanation of symbols]

[0104] 1. Web conferencing system 10 Display System 11. Filming Equipment 12 Terminal 13. Electronic whiteboard 20 Terminals 30 Servers 40 Communication Network 1001 Photography Department 1002 Image processing unit 1003 Change rate calculation unit 1004 Communications Department 2001 Video Layout Decision Department 2002 Display section 2003 Communications Department 2004 Information Recording Department [Prior art documents] [Patent documents]

[0105] [Patent Document 1] Japanese Patent Application Publication No. 5-103324

Claims

1. A display system including a photographing device and a terminal, The photographing device is A filming unit that shoots video; an image processing unit that creates an image to be displayed based on the captured image and an image layout specified by the terminal; a change rate calculation unit that calculates a change rate of an image of a specific area in the captured image specified by the terminal; a communication unit that transmits the image to be displayed and the rate of change to the terminal; Equipped with The terminal includes: A display system comprising a video layout determination unit that selects, based on a rate of change received from the photographing device, either a video layout with fixed display of the image of the specific area or a video layout without fixed display of the image of the specific area.

2. The video layout determination unit of the terminal, The display system according to claim 1 , wherein, when the rate of change over a certain period of time is equal to or less than a certain value and the video layout includes a fixed display of the video in the specific area, the video layout is changed to one in which the video in the specific area is not fixedly displayed.

3. The video layout determination unit of the terminal, The display system according to claim 2 , wherein if the rate of change is equal to or greater than a certain value and the video layout does not have a fixed display of the video in the specific area, the video layout is changed to one in which the video in the specific area is fixedly displayed.

4. The display system according to claim 1 , wherein the terminal receives a specification as to whether or not to change the video layout based on the rate of change.

5. The display system according to claim 2 , wherein the terminal accepts designation of the certain period of time and the certain value.

6. The display system according to claim 1 , wherein the terminal accepts designation of the specific area.

7. The display system according to claim 1 , wherein the display image is displayed on the terminal and another terminal holding a web conference with the terminal.

8. The display system of claim 1 , wherein the captured image is a 360-degree panoramic image.

9. A step of photographing an image by a photographing device; A step of the photographing device creating an image for display based on the photographed image and an image layout specified by a terminal; A step in which the photographing device calculates a rate of change of an image of a specific area in the photographed image specified by the terminal; a step of the photographing device transmitting the image for display and the rate of change to the terminal; A method comprising: the step of the terminal selecting, based on the rate of change received from the photographing device, either a video layout with a fixed display of the image of the specific area or a video layout without a fixed display of the image of the specific area.

10. The device, A communication unit that transmits to the image capture device a designation of a specific area in an image captured by the image capture device and a designation of an image layout. A display unit that receives and displays the image captured by the photographing device and the image for display created based on the image layout specified by the terminal. a video layout determination unit that selects, based on the rate of change of the specific area received from the image capture device, either a video layout with a fixed display of the video of the specific area or a video layout without a fixed display of the video of the specific area; A program to function as a

Images