Display system, method, computer program product, and storage medium

Abstract

The invention provides a display system, a method, a computer program product and a storage medium, which can not continuously display the same image when the displayed image only changes slightly. The display system according to the present invention comprises an imaging device and a terminal, and is characterized in that the imaging device comprises: an imaging unit for imaging an image; an image processing unit for generating a display image on the basis of the captured image and an image layout specified by the terminal; a rate-of-change calculation unit for calculating a rate of change of an image in a specific region within the captured image specified by the terminal; and a communication unit for transmitting the display image and the rate of change to the terminal, the terminal having an image layout determination unit for determining the layout of the display image on the basis of the rate of change received from the imaging device. A video layout is selected from among a video layout in which a video having the specific region is fixedly displayed and a video layout in which a video not having the specific region is fixedly displayed.

Description

Technical Field

[0001] The present invention relates to a display system, a method, a computer program product and a storage medium. Background Art

[0002] As is well known, network conferences can perform two-way video and audio communication between terminals installed at multiple locations. The terminals display images taken at each location (see Patent Document 1: Japanese Patent Publication H5-103324). Therefore, the participants of the conference can share the atmosphere of the conference.

[0003] However, when there are only slight changes in the image, or when the subject does not change, it is best to display other images instead, such as an image of the subject that has changed. Summary of the invention

[0004] In view of the above problems, the present invention proposes a technical solution, which can stop displaying the same image when there is only a slight change in the displayed image.

[0005] A display system involved in one embodiment of the present invention includes a photographic device and a terminal, characterized in that the photographic device has: a photographic unit for photographing an image; an image processing unit for generating a display image based on the photographed image and an image layout specified by the terminal; a change rate calculation unit for calculating the change rate of an image of a specific area within the photographed image specified by the terminal; and a communication unit for sending the display image and the change rate to the terminal, the terminal having an image layout determination unit for selecting any one image layout from an image layout in which an image with the specific area is fixedly displayed and an image layout in which an image without the specific area is fixedly displayed, based on the change rate received from the photographic device.

[0006] The effect of the present invention is that the same image can be discontinued when there are only slight changes in the displayed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic diagram of the overall structure of a network conference system involved in one embodiment of the present invention.

[0008] Figure 2 It is a schematic diagram of the photographing direction of the photographing device involved in one embodiment of the present invention.

[0009] Figure 3 This is a schematic diagram of transformation of an image captured by a photographic device according to one embodiment of the present invention.

[0010] Figure 4 This is a block diagram of the hardware structure of a photographic device according to one embodiment of the present invention.

[0011] Figure 5 This is a block diagram of the hardware structure of a terminal (personal computer) according to one embodiment of the present invention.

[0012] Figure 6 This is a block diagram of the hardware structure of a terminal (video conferencing terminal) according to one embodiment of the present invention.

[0013] Figure 7 This is a functional block diagram of a display system according to one embodiment of the present invention.

[0014] Figure 8 It is a schematic diagram of an image layout involved in one embodiment of the present invention.

[0015] Fig. 9 It is a schematic diagram of the change rate involved in one embodiment of the present invention.

[0016] Fig.10 This is a flowchart of the image change rate notification process of a fixed display area of ​​a photographic device involved in one embodiment of the present invention.

[0017] Fig.11 This is a flowchart of a video layout determination process of a terminal according to one embodiment of the present invention.

[0018] Fig.12 This is a flowchart of a video layout determination process of a terminal according to one embodiment of the present invention.

[0019] Fig.13 This is a schematic diagram of an example of an operation screen displayed on a terminal involved in one embodiment of the present invention.

[0020] Fig.14 This is a schematic diagram of an example of a fixed display area designation screen displayed on a terminal according to an embodiment of the present invention.

[0021] Explanation of symbols

[0022] 1 Network conference system

[0023] 10 Display System

[0024] 11 Photographic Installation

[0025] 12.20 Terminal

[0026] 13 Electronic Blackboard

[0027] 30 Servers

[0028] 40 Communication Network

[0029] 1001 Photography Department

[0030] 1002 Image Processing Department

[0031] 1003 Change rate calculation unit

[0032] 1004 Department of Communications

[0033] 2001Image Layout Decision Department

[0034] 2002 Display Department

[0035] 2003 Ministry of Communications

[0036] 2004 Information Recording Department DETAILED DESCRIPTION

[0037] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0038] <Overall composition>

[0039] Figure 1 1 is a schematic diagram of the overall structure of a network conference system according to an embodiment of the present invention. The network conference system (also referred to as a video conference system) 1 includes terminals ( Figure 1 In the example of FIG. 1 , the terminal 12 at location 1 and the terminal 20 at location 2 ) and the server 30 .

[0040] [Location 1]

[0041] In one embodiment of the present invention, a terminal 12 and a photographing device 11 are provided in at least one location of the network conference system (hereinafter, the system including the terminal 12 and the photographing device 11 is also referred to as a display system). The display system 10 may further include an electronic blackboard 13 .

[0042] [terminal]

[0043] The terminal 12 is a terminal used to participate in a network conference. The terminal 12 obtains image data and audio data from the imaging device 11, and sends the image data and audio data to the server 30. The terminal 12 also receives from the server 30 the image data and audio data obtained by the server 30 from the terminal 20 at the location 2. For example, the terminal 12 is a computer such as a personal computer, a tablet, a smart phone, or a terminal for video conferencing.

[0044] [Photographic Installation]

[0045] The photographing device 11 is a device with a photographing function (camera). The photographing device 11 may also have a sound collecting function (microphone) and a sound output function (speaker). The photographing device 11 can send and receive information with the terminal 12, for example, the photographing device 11 and the terminal 12 are connected via a USB (Universal Serial Bus).

[0046] Figure 2 FIG. 1 is a schematic diagram of the photographing direction of the photographing device 11 according to one embodiment of the present invention. Figure 2 As shown in (A), the photographic device 11 can shoot in 360 degrees (around the photographic device 11) on the horizontal plane. Figure 2 As shown in (B), it is possible to capture images in directions at predetermined angles (for example, a=60 degrees, b=30 degrees) above and below the horizontal (0 degrees) on the vertical plane.

[0047] Figure 3 FIG. 1 is a schematic diagram of the transformation of an image captured by the imaging device 11 according to an embodiment of the present invention. Figure 3 As shown, the photographing device 11 can capture the image (e.g. Figure 2 The image captured in the photographic direction described in the figure is converted into a panoramic image (hereinafter also referred to as a "360-degree panoramic image"). In addition, the photographic device 11 can cut out a specific area (for example, an area where a speaker is photographed, an area specified by the terminal 12 (for example, an area where data displayed on the electronic blackboard 13 is photographed)) from the 360-degree panoramic image.

[0048] [Electronic Blackboard]

[0049] The electronic blackboard 13 is a display device that displays data of a network conference and the like.

[0050] [Location 2]

[0051] At other locations other than the location where the display system 10 is located, such as Figure 1 Location 2 in the example can be Figure 1 In that case, only a terminal with a photographic function may be provided, or a terminal and a photographic device with a photographic function may be provided.

[0052] The photographing device 11 and the terminal 20 can be any device as long as they have a communication function. The photographing device 11 and the terminal 20 can be, for example, output devices such as PJ (Projector), IWB (Interactive White Board), digital signage, HUD (Head Up Display) equipment, industrial machinery, camera equipment, sound collection equipment, medical equipment, network appliances, automobiles (Connected Car), notebook PCs (Personal Computers), mobile phones, smart phones, tablet terminals, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs or desktop PCs, etc.

[0053] [server]

[0054] The server 30 can transmit and receive data with terminals at various locations via the communication network 40. The server 30 also transmits image data and audio data obtained from a terminal at a certain location to terminals at other locations.

[0055] Specifically, the server 30 obtains image data and sound data from the terminal 12 at the location 1, such as the image data and sound data obtained by the terminal 12 from the photographing device 11, and transmits the image data and the sound data to the terminal 20 at the location 2. Similarly, the server 30 obtains image data and sound data from the terminal 20 at the location 2, such as the image data taken by the terminal 20 and the sound data collected by the terminal 20, and transmits the image data and the sound data to the terminal 12 at the location 1.

[0056] The device described in the embodiments represents only one of many computing environments for implementing the embodiments disclosed herein.

[0057] In one embodiment, the server 30 includes multiple computing devices such as a server cluster. The multiple computing devices can communicate with each other through any type of communication link (including network, shared memory, etc.) to perform various processes disclosed herein.

[0058] <Hardware Configuration>

[0059] The following references Figure 4 , describing the hardware structure of the photographic device 11, referring to Figure 5 , describing the hardware configuration of the terminals (personal computers) 12, 20, and the server 30, referring to Figure 6 , describing the hardware structure of the terminals (video conferencing terminals) 12 and 20.

[0060] [Photographic Installation]

[0061] Figure 4 This is a block diagram of the hardware structure of a photographing device 11 (a conference device capable of photographing 360-degree animation) according to one embodiment of the present invention.

[0062] In the following description, the photographing device 11 is a device that uses a camera element to shoot a 360-degree animation around the device at a predetermined height, but the camera element may be one or more than two. It does not necessarily need to be a dedicated device, and a 360-degree animation camera element installed later on a personal computer, a digital camera, a smart phone, etc. may have substantially the same function.

[0063] like Figure 4As shown, the photographic device 11 has a camera unit 101, an image processing unit 104, a camera control unit 105, a microphone 106, a sound processing unit 107, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operating unit 115, an external device connection I / F 116, a communication unit 117, an antenna 117a, a sound sensor 118, and a speaker 119.

[0064] The imaging unit 101 includes a wide-angle lens (so-called fisheye lens) 102 with a 360-degree viewing angle for forming a hemispherical image, and an imaging element 103 (image sensor) provided corresponding to each wide-angle lens. The imaging element 103 includes an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image based on the fisheye lens 102 into an electrical signal and outputs an image signal, a timing generation circuit that generates a horizontal or vertical synchronization signal or a pixel clock of the image sensor, and a register group that sets various instructions or parameters required for the operation of the imaging element.

[0065] The imaging element 103 (image sensor) of the imaging unit 101 is connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging element 103 of the imaging unit 101 is connected to the imaging control unit 105 via a serial I / F bus (I2C bus, etc.). The image processing unit 104, the imaging control unit 105, and the sound processing unit 107 are connected to the CPU 111 via the bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the peripheral connection interface (I / F) 116, the communication unit 117, the sound sensor 118, the speaker 119, etc. are also connected to the bus 110.

[0066] The image processing unit 104 reads the image data output by the imaging element 103 through the parallel I / F bus, performs predetermined processing on each image data, generates panoramic image data or speaker image data based on the fisheye image, and further synthesizes the panoramic image and speaker image to output an animation.

[0067] The camera control unit 105 generally uses the camera control unit 105 as a master device and the camera element 103 as a slave device, and uses the I2C bus to set instructions and the like in the register group of the camera element 103. The necessary instructions and the like are received from the CPU 111. The camera control unit 105 also uses the same I2C bus to read the status data and the like of the register group of the camera element 103, and then sends it to the CPU 111.

[0068] When the imaging start button of the operation unit 115 is pressed or an imaging start instruction is received from a personal computer, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data. Some imaging devices 11 also have a function corresponding to a preview display function or an animation display on a display (for example, a display of a personal computer or a smartphone). In this case, the imaging element 103 continuously outputs image data at a predetermined frame rate (frames / minute).

[0069] The image capture control unit 105 also has a synchronization control function for cooperating with the CPU 111 to synchronize the output timing of the image data of the image capture element 103, which will be described later. In this embodiment, the image capture device 11 is not provided with a display, but a display unit may be provided.

[0070] The microphone 106 converts the sound into audio (signal) data. The sound processing unit 107 receives the audio data output from the microphone 106 through the I / F bus and performs predetermined processing on the sound data.

[0071] The speaker 119 converts the input audio data into sound. The microphone 106 and the speaker 119 are examples of input and output devices. The input and output devices do not need to be of the same structure, and the input device and the output device may be formed separately.

[0072] The CPU 111 controls the overall operation of the imaging device 11 and performs necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are working memories that store programs executed by the CPU 111 and data in process. In particular, the DRAM 114 stores image data in process of the image processing unit 104 and data of equirectangular projection images that have been processed.

[0073] The operation unit 115 is a general term for operation buttons such as the photography start button 115a. By operating the operation unit 115, the user can start photography or video recording and input power ON / OFF, communication connection, various photography modes, photography conditions, and other settings.

[0074] The peripheral connection I / F 116 is an interface for connecting various external devices. The external device at this time is, for example, a PC (Personal Computer), etc. The animation data or image data stored in the DRAM 114 is sent to an external terminal (device) through the network I / F 116, or recorded in an external medium.

[0075] The communication unit 117 may also communicate with a cloud server via the Internet through wireless communication technology such as Wi-Fi via an antenna 117a provided on the photographing device 11, and send the stored animation data or image data to the cloud server. It may also communicate with nearby devices using short-range wireless communication technology such as BLE or NFC.

[0076] The sound sensor 118 is a sensor for obtaining 360-degree sound information and is used to determine which direction of the input sound is louder in the 360-degree surrounding (horizontal plane) of the camera 11. The sound processing unit 107 determines the strongest direction based on the input 360-degree sound parameters and outputs the 360-degree sound input direction.

[0077] Other sensors (such as orientation and acceleration sensors or GPS) may also be used to calculate orientation, position, angle, acceleration, etc. for image compensation or additional position information.

[0078] The image processing unit 104 performs the following processing.

[0079] A panoramic image is generated by the following method. The RAW data input from the image sensor for inputting a spherical image is subjected to a predetermined camera image processing such as Bayer transformation (RGB compensation processing) to produce a fisheye image (curved surface image). Furthermore, the generated fisheye image (curved surface image) is subjected to a DeWarp processing (distortion compensation processing) to generate a 360-degree panoramic image (flat image) around the camera device 11.

[0080] ·Generate a speaker image according to the following method. The photographic device 11 creates a speaker image that cuts out the speaker based on the surrounding 360-degree panoramic image (plane image). The sound input direction found from 360 degrees output by the sound sensor 118 and the sound processing unit 107 is used as the direction of the speaker, and the speaker image is cut out from the above panoramic image. At this time, the method of cutting out the image of a person from the sound input direction is to cut out 30 degrees with the sound direction found from 360 degrees as the center, perform face detection in it, and then cut it. Furthermore, the image of the specific number of speakers who just spoke, such as 3 people, is found in the cut speaker image. By arranging the panoramic image and one or more speaker images up and down (or adjacent to each other), an animation of the meeting record information based on the 360-degree photographic image is created.

[0081] [Terminal (Personal Computer)]

[0082] Figure 5 This is a block diagram of the hardware configuration of the terminal 12 (in the case of a personal computer) according to one embodiment of the present invention. The same applies to the terminal 20 and the server 30.

[0083] like Figure 5 As shown, the terminal 12 is composed of a computer, such as Figure 5 As shown, it includes CPU 201, ROM 202, RAM 203, HD 204, hard disk drive (HDD) controller 205, display 206, peripheral connection I / F 207, network I / F 208, data cable 209, keyboard 210, pointing device 211, DCD-RW (Digital Versati le Dis sk Rewritable) drive 213, and media I / F 215.

[0084] Among them, CPU201 controls the overall operation of terminal 12. ROM202 stores programs such as IPL for driving CPU201. RAM203 is used as a work area for CPU201. HD204 stores various data such as programs. HDD controller 205 controls the reading or writing of various data in HD204 according to the control of CPU201. Display 206 displays various information such as cursor, menu, window, text or image. Peripheral connection I / F207 is an interface for connecting various external devices. In this case, the peripheral is, for example, a USB (Universal Serial Bus) memory or a printer. Network I / F208 is an interface for data communication using communication network 40. Bus 209 is an interface for electrical connection. Figure 5 The address bus and data bus of each component such as the CPU201 shown.

[0085] The keyboard 210 is an input device having a plurality of keys for inputting characters, numerical values, various instructions, etc. The pointing device 211 is a type of input device for selecting and executing various instructions, selecting a processing object, moving a cursor, etc. The DVD-RW drive 213 controls reading or writing of various data to a DVD-RW 212 as an example of a removable recording medium. In addition to DVD-RW, a DVD-R, etc. may also be used. The medium I / F 215 controls reading or writing (saving) of data in a recording medium 214 such as a flash memory.

[0086] [Terminal (video conferencing terminal)]

[0087] Figure 6 This is a diagram showing the hardware configuration of the terminal 12 (in the case of a video conference terminal) according to one embodiment of the present invention. The terminal 20 is also the same.

[0088] like Figure 6 As shown, the video conference terminal 12 includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, an SSD 305, a medium I / F 307, an operation button 308, a power switch 309, a bus 310, a network I / F 311, a CMOS 312, an image sensor I / F 313, a microphone 314, a speaker 315, a sound input and output I / F 316, a display I / F 317, an external device connection I / F 318, a short-distance communication circuit 319, and an antenna 319a of the short-distance communication circuit 319. Among them, the CPU 301 controls the overall operation of the video conference terminal 12. The ROM 302 stores programs such as IPL for driving the CPU 301. The RAM 303 is used as a work area for the CPU 301. The flash memory 304 stores various data such as communication processes, image data, and sound data. The SSD 305 controls the reading or writing of various data in the flash memory 304 according to the control of the CPU 301. The HDD can also be used instead of the SSD. The media I / F 307 controls reading or writing (saving) of data in the recording medium 306 such as a flash memory. The operation button 308 is a button operated when selecting a destination of the video conference terminal 12. The power switch 309 is a switch for switching the power of the video conference terminal 12 on and off.

[0089] The network I / F 311 is an interface for data communication using a communication network 40 such as the Internet. The CMOS (Complementary Metal Oxide Semiconductor) sensor 312 is an example of a built-in camera device that captures a subject to obtain image data under the control of the CPU 301. Instead of a CMOS sensor, it may be a camera device such as a CCD (Charge Coupled Device) sensor. The camera element I / F 313 is a circuit that controls the drive of the CMOS sensor 312. The microphone 314 is a built-in circuit that converts sound into an electrical signal. The speaker 315 is a built-in circuit that converts an electrical signal into physical vibration to generate sound such as music or voice. The sound input and output I / F 316 is a circuit that processes the input and output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display I / F 317 is a circuit that sends image data to an external display under the control of the CPU 301. The peripheral connection I / F 318 is an interface for connecting various external devices. The short-distance communication circuit 319 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark).

[0090] Bus 310 is used to electrically connect Figure 6 The address bus and data bus of each component such as the CPU301 shown.

[0091] The display is a display device composed of liquid crystal and organic EL (Electro Luminescence), etc., and is used to display images of subjects and operation icons, etc. The display is connected to the display I / F 117 via a cable. The cable can be a cable for simulated RGB (VGA) signals, a cable for component video, or a cable for HDMI (High-Definition Multimedia Interface) (registered trademark) or DVI (Digital Video Interactive) signals.

[0092] The CMOS sensor 312 is a built-in camera device that obtains image data by photographing a subject under the control of the CPU 301. Instead of a CMOS sensor, it may be a camera device such as a CCD sensor. The peripheral connection I / F 318 can be connected to external devices such as an external camera, an external microphone, and an external speaker through a USB cable or the like. When an external camera is connected, the external camera is driven first under the control of the CPU 301, and then the built-in CMOS sensor 312 is driven. Similarly, when an external microphone is connected or an external speaker is connected, the external microphone and the external speaker are driven first under the control of the CPU 301, and then the built-in microphone 314 and the built-in speaker 315 are driven.

[0093] The recording medium 306 can be freely attached to and detached from the video conference terminal 12. As long as it is a nonvolatile memory that reads or writes data under the control of the CPU 301, it is not limited to the flash memory 304, and an EEPROM or the like may be used.

[0094] <Functional Configuration>

[0095] Figure 7 It is a functional block diagram of a display system 10 according to one embodiment of the present invention.

[0096] [Photographic Installation]

[0097] The photographing device 11 includes a photographing unit 1001, an image processing unit 1002, a change rate calculation unit 1003, and a communication unit 1004. The photographing device 11 has the functions of the photographing unit 1001, the image processing unit 1002, the change rate calculation unit 1003, and the communication unit 1004 by executing a program.

[0098] The photographing unit 1001 photographs an image. For example, the photographing unit 1001 can photograph a direction of 360 degrees (around the imaging device 11) on the horizontal plane, and can photograph a direction of a predetermined angle upward or downward from the horizontal (0 degrees) on the vertical plane.

[0099] The image processing unit 1002 generates an image for display based on the image captured by the photographing unit 1001 and the image layout specified by the terminal 12. Specifically, the image processing unit 1002 arranges the image captured by the photographing unit 1001 (for example, a panoramic image and an image cut from the panoramic image) according to the image layout specified by the terminal 12. The image for display is displayed on the terminal 12 and the terminal 20 that conducts a network conference with the terminal 12.

[0100] For example, the image processing unit 1002 may process the image (e.g. Figure 2 The image processing unit 1002 can cut out a specific area (e.g., an area where a speaker is photographed, an area specified by the terminal 12 (e.g., an area where data displayed on the electronic blackboard 13 is photographed) from the panoramic image.

[0101] The change rate calculation unit 1003 calculates the change rate of the image of a specific area (also referred to as a fixed display area) in the image captured by the imaging unit 1001 and specified by the terminal 12 .

[0102] The communication unit 1004 transmits the display image created by the image processing unit 1002 and the change rate calculated by the change rate calculation unit 1003 to the terminal 12. The communication unit 1004 also receives from the terminal 12 the designation of a specific area (fixed display area) in the image captured by the imaging unit 1001 and the designation of the image layout.

[0103] [terminal]

[0104] The terminal 12 includes a video layout determination unit 2001, a display unit 2002, a communication unit 2003, and an information recording unit 2004. The terminal 12 has the functions of the video layout determination unit 2001, the display unit 2002, the communication unit 2003, and the information recording unit 2004 by executing a program.

[0105] The image layout determination unit 2001 selects, based on the rate of change received from the photographic device 11 (i.e., the rate of change of the image of the specific area (fixed display area) within the image captured by the photographic device 11), either an image layout having a fixed display of the image of the specific area (fixed display area) or an image layout having a fixed display of the image without the specific area (fixed display area).

[0106] For example, when the change rate is below a certain value within a certain time, and the image layout has a fixed display of the image with a specific area (fixed display area) (i.e., fixed display is performed at this time), the image layout determination unit 2001 changes the image layout to a fixed display of the image without a specific area (fixed display area).

[0107] For example, when the rate of change over a certain period of time is above a certain value, and the image layout is a fixed display of the image without a specific area (fixed display area) (i.e., no fixed display is performed at this time), the image layout determination unit 2001 will change the affected layout to a fixed display of the image with a specific area (fixed display area).

[0108] The image layout determination unit 2001 receives a designation of whether to implement a change in the image layout based on the rate of change, and when the designation of implementation is received, the image layout change can be automatically implemented. The image layout determination unit 2001 can also receive a designation of a certain time and a certain value and use them. Furthermore, the image layout determination unit 2001 can also receive a designation of a specific area (fixed display area).

[0109] The display unit 2002 displays the display image created by the imaging device 11 .

[0110] The communication unit 2003 receives from the photographing device 11 the display image produced by the photographing device 11 and the rate of change calculated by the photographing device 11. The communication unit 2003 sends to the photographing device 11 the designation of a specific area (fixed display area) in the image captured by the photographing device 11 and the designation of the image layout (i.e., the image layout selected by the image layout determination unit 2001).

[0111] The information recording unit 2004 stores information on the rate of change and the time when the rate of change was received, and information on whether fixed display was performed at that time.

[0112] [Image Layout]

[0113] Figure 8 It is a schematic diagram of an image layout involved in one embodiment of the present invention.

[0114] Image layouts include image layouts where images without specific areas are fixedly displayed ( Figure 8 ), and image layouts with a specific area of ​​image fixed display ( Figure 8 (For example, you can set an image layout in which you set "Fixed display setting" of the image to "Fixed display ON"). You can also set an image layout in which, for example, an image without a specific area is fixedly displayed as a default value.

[0115] The image layout without fixed display of the image of a specific area (fixed display OFF) includes the image layout of "panoramic two-stage" and the image layout of "focus on the speaker". For example, the image layout of "focus on the speaker" can be set as a default value.

[0116] In the "panoramic two-segment" image layout, the 360-degree panoramic image is displayed in two segments (displaying two 180-degree panoramic images).

[0117] In the "Focus on the speaker" image layout, a 360-degree panoramic image and a speaker image are displayed (the speaker is detected and the speaker image is cut out from the panoramic image).

[0118] The image layout (fixed display ON) of the image fixed display having a specific area includes the image layout of "fixed display" and the image layout of "fixed display + speaker frame". For example, the image layout of "fixed display + speaker frame" may be set as a default value.

[0119] The "Fixed display" image layout displays a 360-degree panoramic image and fixed display images (images of a specific area (fixed display area)).

[0120] The image layout of "Fixed display + speaker frame" displays a 360-degree panoramic image, a fixed display image (an image of a specific area (fixed display area)), and an image of the speaker (the speaker is detected and the speaker's image is cut out from the panoramic image).

[0121] [Rate of change]

[0122] Fig. 9 Schematic diagram of the change rate according to one embodiment of the present invention. The change rate is the ratio of the number of pixels that are different between a certain image (image before change) and the image after a predetermined time (image after change) to the total number of pixels.

[0123] For example, suppose that conference data A is captured in the fixed display area. If conference data A is captured again after a specified time (e.g., 1 second), the change rate is 0%. If conference data B different from conference data A is captured after a specified time (e.g., 1 second), the ratio of the number of different pixels between the image capturing conference data A and the image capturing conference data B to the total number of pixels (e.g., 6%) is calculated as the change rate.

[0124] <Method>

[0125] refer to Figure 10 to Figure 12 , explaining each treatment method.

[0126] Fig.10 This is a flowchart of a notification process of a fixed display area image change rate implemented by the photographing device 11 according to one embodiment of the present invention.

[0127] In step 11 (S11), the photographing unit 1001 of the photographing device 11 photographs an image.

[0128] In step 12 ( S12 ), the change rate calculation unit 1003 of the imaging device 11 calculates the image change rate of the fixed display area in the image captured in S11 .

[0129] In step 13 ( S13 ), the communication unit 1004 of the photographing device 11 transmits the change rate obtained in S12 to the terminal 12 .

[0130] During the photography by the photography device 11, steps S11 to S13 are continuously executed.

[0131] Fig.11 This is a flowchart of the image layout determination process performed by the terminal 12 according to one embodiment of the present invention.

[0132] In step 21 (S21), the communication unit 2003 of the terminal 12 receives the image change rate of a specific area (fixed display area) in the image captured by the image capturing device 11 from the image capturing device 11. The information recording unit 2004 stores the change rate and the time when the change rate was received.

[0133] In step 22 (S22), the image layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 to determine whether the fixed display is set (ON) or not set (OFF). If the fixed display is set (ON), the process proceeds to step 23, and if the fixed display is not set (OFF), the process ends.

[0134] In step 23 (S23), the image layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 to determine whether all the change rates within the previous certain time B (for example, 2 minutes) are less than a certain value C (for example, 3%). If all the change rates within the previous certain time B are less than the certain value C, the process proceeds to step 24, and if the change rates within the previous certain time B are not less than the certain value C, the process ends.

[0135] In step 24 ( S24 ), the image layout determination unit 2001 of the terminal 12 sets the fixed display to OFF (ie, changes to an image layout without fixed display of the specific area image), and then notifies the image layout to the photographing device 11 .

[0136] The terminal 12 may also display that the image layout has automatically changed.

[0137] Fig.12 This is a flowchart of a video layout determination process performed by the terminal 12 according to one embodiment of the present invention. Fig.12 After setting the fixed display to OFF in , you can set the fixed display to ON.

[0138] In step 31 (S31), the communication unit 2003 of the terminal 12 receives the image change rate of a specific area (fixed display area) in the image captured by the image capturing device 11 from the image capturing device 11. The information recording unit 2004 stores information on the change rate and the time when the change rate was received.

[0139] In step 32 (S32), the image layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 to determine whether the fixed display is set (ON) or not set (OFF). If the fixed display is not set (OFF), the process proceeds to step 33, and if the fixed display is set (ON), the process proceeds to step 35.

[0140] In step 33 (S33), the image layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 to determine whether the change rate is greater than a certain value C (e.g., 3%). If the change rate is greater than the certain value C, the process proceeds to step 34, and if not, the process ends.

[0141] In step 34 (S34), the image layout determination unit 2001 of the terminal 12 sets the fixed display to ON (i.e., changes the image layout to a fixed display with a specific area image (e.g., the image layout before the change or the image layout of the preset value stored in the information recording unit 2004 in S36)). Then, the image layout is notified to the photographing device 11.

[0142] In step 35 (S35), the image layout determination unit 2001 of the terminal 12 refers to the information recording unit 2004 to determine whether the change rate before and after the specified time B (for example, 2 minutes) is less than a certain value C (for example, 3%). When the change rate before and after the specified time B is less than the certain value C, the process proceeds to step 36, and if the change rate before and after the specified time B is not less than the certain value C, the process ends.

[0143] In step 36 (S36), the image layout determination unit 2001 of the terminal 12 turns the fixed display off (i.e., changes to an image layout without a fixed display of a specific area image). Then, the image layout determination unit 2001 of the terminal 12 saves the information of the image layout before the change in the information recording unit 2004. Then, the image layout is notified to the photographing device 11.

[0144] Furthermore, the terminal 12 may display content indicating that the image layout is automatically changed.

[0145] <User Interface>

[0146] refer to Fig.13 and Fig.14 , illustrating the screen displayed on the terminal 12.

[0147] Fig.13 is an example of an operation screen displayed on the terminal 12 according to one embodiment of the present invention. The operator of the terminal 12 can Fig.13 On the screen, you can set the image layout and fixed display.

[0148] ·(1) displays the application name.

[0149] Select (2) to maximize, minimize, or close the screen.

[0150] Select (3) to display the operation screen of the imaging device 11.

[0151] ·Select (4) to display the setting screen of the imaging device 11.

[0152] (5) The name of the imaging device 11 is displayed.

[0153] Select (6) to display help.

[0154] In (7), a display screen generated by the photographing device 11 is displayed.

[0155] Select (8) to change the image layout (for example, to Figure 8 the image layout you want in the image layout you want).

[0156] Select (9) to display Fig.14 The specified screen of the fixed display area.

[0157] Select (10) to change the front side of the image of the photographic device 11.

[0158] Select (11) to enable or disable (mute) the microphone.

[0159] Select (13) to increase or decrease the volume.

[0160] Select (14) to display application information.

[0161] Select (15) to display the protocol information.

[0162] Select (16) to display the meeting list screen.

[0163] ·Select (17) to display the application settings screen.

[0164] · Select (18) to confirm the user name logged in to the application or log out.

[0165] Select (19) to record the web conference.

[0166] Fig.14 is an example of a designation screen of a fixed display area displayed on the terminal 12 according to one embodiment of the present invention. The operator of the terminal 12 can Fig.14 On the screen, a specific area of ​​the image captured by the photographic device 11 is designated as a fixed display area.

[0167] ·(1) shows the instructions for fixed display.

[0168] In (2), you can choose whether to display the image fixedly.

[0169] In (3), a fixed display area can be designated on the image captured by the image capturing device 11 .

[0170] In (4), the size of the fixed display area specified in (3) can be changed.

[0171] In (5), if you select the "Cancel" button, the display will not be fixed and the Fig.13 operation screen.

[0172] In (6), if you select the "OK" button, the setting is fixed and the display returns to Fig.13 operation screen.

[0173] <Effect>

[0174] One embodiment of the present invention may stop displaying an image when there are small changes within the image (for example, when the meeting data displayed on the electronic blackboard is fixedly displayed, when the discussion at the meeting is heated or the topic deviates, and the data of the meeting remains unchanged), and instead automatically display other images (for example, images that can share the atmosphere of the meeting, such as the expressions of the speaker or participants other than the speaker).

[0175] Each function of the above-described embodiments can be implemented by one or more processing circuits. Here, the "processing circuit" herein includes a processor that is programmed by software to perform each function, such as a processor installed by a circuit, and an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), an existing circuit module and other devices designed to perform each function described above.

Claims

1. A display system, comprising a photographing device and a terminal, characterized in that: The photographic device comprises: The photography department, for taking images; An image processing unit, configured to generate an image for display based on the captured image and the image layout specified by the terminal; a change rate calculation unit, configured to calculate a change rate of an image of a specific area within the captured image specified by the terminal; and a communication unit, configured to send the display image and the change rate to the terminal, The terminal includes an image layout determination unit for selecting, based on the change rate received from the photographic device, an image layout in which an image having the specific area is fixedly displayed and an image layout in which an image not having the specific area is fixedly displayed.

2. The display system according to claim 1, wherein: When the rate of change of the image before and after a predetermined time is equal to or less than a certain value and the image layout has a fixed image display of the specific area, the image layout determination unit of the terminal changes the image layout to a fixed image display without the specific area.

3. The display system according to claim 2, wherein: When the change rate is equal to or greater than a certain value and the image layout does not include the fixed image display of the specific area, the image layout determination unit of the terminal changes the image layout to include the fixed image display of the specific area.

4. The display system according to claim 1, wherein: The terminal accepts a designation of whether to change an image layout based on the change rate.

5. The display system according to claim 2 or 3, wherein: The terminal accepts designation of the certain time and the certain value.

6. The display system according to claim 1, wherein: The terminal receives designation of the specific area.

7. The display system according to claim 1, wherein: The display image is displayed on the terminal and other terminals that are in a network conference with the terminal.

8. The display system according to claim 1, wherein: The captured image is a 360-degree panoramic image.

9. A method, characterized in that The following steps are involved: capture images using photographic devices; The photographing device produces an image for display based on the captured image and the image layout specified by the terminal; Calculating, by the photographing device, an image change rate of a specific area within the captured image specified by the terminal; The photographing device sends the display image and the change rate to the terminal; The terminal selects any one of the image layouts from the image layout in which the image having the specific area is fixedly displayed and the image layout in which the image not having the specific area is fixedly displayed, according to the change rate received from the photographing device.

10. A computer program product, characterized in that The terminal can have the following functions: a communication unit for sending the designation of a specific area in an image captured by the photographing device and the designation of an image layout to the photographing device; A display unit, configured to receive and display a display image generated by the photographing device according to the image taken by the photographing device and the image layout specified by the terminal; and An image layout determination unit is used to select any one of an image layout of a fixed display of an image having the specific area and an image layout of a fixed display of an image not having the specific area, based on a change rate of the specific area received from the photographic device.

11. A computer-readable storage medium, characterized in that: The computer stores a program that can realize the following functions: a communication unit for sending the designation of a specific area in an image captured by the photographing device and the designation of an image layout to the photographing device; A display unit, configured to receive and display a display image generated by the photographing device according to the image taken by the photographing device and the image layout specified by the terminal; and An image layout determination unit is used to select any one of an image layout of a fixed display of an image having the specific area and an image layout of a fixed display of an image not having the specific area, based on a change rate of the specific area received from the photographic device.

Images