Image processing device, image correction processing method, conference system, and program

Abstract

When combining multiple images for output, the system delivers the desired image while reducing the user's burden. [Solution] An image processing apparatus comprising: an image acquisition means for acquiring a first image captured by an imaging means and a second image different from the first image; and a control means for controlling a display means to display a predetermined image, wherein the control means controls the display means to display a corrected image obtained by correcting at least a part of the first image and the second image when both the first image and the second image have been acquired, and controls the display means to display the acquired image when either the first image or the second image has been acquired.

Description

Technical Field

[0004] , , , , , , ,

[0006] , , , , , , , , , ,

[0005] ,

[0007] ,

[0001] The present invention relates to an image processing apparatus, an image correction processing method, a conference system, and a program.

Background Art

[0002] In recent years, when holding a meeting or lecture for people participating in a physical venue, it has become common to simultaneously distribute an image of the venue to online participants. For example, there is a technology that uses an imaging device to capture the speaker and the state of the venue for distribution and processes a part of the captured image for distribution.

[0003] In Patent Document 1, a technology for extracting people, objects, and backgrounds in a venue, synthesizing them for distribution, and outputting them, and a technology for manually switching the presence or absence of image synthesis and background correction for an image selected by a user are disclosed.

[0004] In addition, when distributing a captured image, when displaying a captured image such as the state of the venue or people and presentation materials such as a PC screen on the same screen for distribution, it is necessary to process the distribution image so that viewers can easily focus on the presentation materials.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the conventional technologies disclosed in the above-mentioned patent documents, it is necessary to manually switch the presence or absence of synthesizing different images with an image and the presence or absence of background correction for the image. As a result, there may be a time when information unintended by the user is included in the distribution screen until the switching.

[0007] In view of the above technology, the object of the present invention is to provide a technology that enables the delivery of desired images while reducing the burden on the user when outputting a combination of multiple images. [Means for solving the problem]

[0008] An image processing apparatus comprising: an image acquisition means for acquiring a first image captured by an imaging means and a second image different from the first image; and a control means for controlling a display means to display a predetermined image, wherein the control means controls the display means to display a corrected image obtained by correcting at least a part of the first image and the second image when both the first image and the second image have been acquired, and controls the display means to display the acquired image when either the first image or the second image has been acquired. [Effects of the Invention]

[0009] According to the present invention, when outputting a combination of multiple images, it is possible to deliver the desired image while reducing the burden on the user. [Brief explanation of the drawing]

[0010] [Figure 1] This is a system configuration diagram in the first embodiment. [Figure 2] This figure shows an example of the hardware configuration of the image processing device and imaging device in the first embodiment. [Figure 3] This figure shows an example of a functional block diagram of an image processing device in the first embodiment. [Figure 4] This is an image diagram of the first embodiment. [Figure 5] This is a flowchart for performing the correction process in the first embodiment. [Figure 6] This figure shows an example of a UI screen in the first embodiment. [Figure 7] This is a system configuration diagram in the second embodiment. [Figure 8] This is an image diagram of the second embodiment. [Figure 9] This is a flowchart for performing the correction process in the second embodiment. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] <First Embodiment> Figure 1 is a diagram showing the configuration of an imaging system according to the first embodiment. The imaging system in this embodiment includes an imaging device 100a, an imaging device 100b, an image processing device 200, a monitor 300, a communication server 400, and a network 500. The method of connecting each device is not limited to a specific method. For example, they may be connected by wired cables.

[0013] The imaging devices 100a and 100b transmit image data based on the imaging results to the image processing device 200 via the network 500. For the sake of explanation, we have used two imaging devices, 100a and 100b, but there is no limit to the number; one or more devices are sufficient.

[0014] The image processing device 200 is connected to the imaging device 100a, imaging device 100b, and communication server 400 via the network 500. There are no restrictions on the type of connection used by the network 500.

[0015] The image processing device 200 performs image processing (details will be described later) on the images received from the imaging devices 100a and 100b, and outputs them to the server 400 via the network 500. Here, although the image processing device 200 is set to output to the server 400 for distribution, it may not be necessary to output to the server 400, such as when creating a video for recording.

[0016] The signal input to the image processing device 200 is not limited to the case where the imaging device 100 is directly connected in the previous stage, and it is also possible to input image data via other processing devices such as switches and effectors, and systems with layout change functions such as video conferencing systems.

[0017] Also, the output image data obtained by performing the image processing described later in the image processing device 200 can be output to the monitor 300.

[0018] Also, the image processing device 200 will be described as being integrated with hardware having an operation unit that enables operations on a camera or output image, such as a camera controller, but it may be separate and independent.

[0019] When it is independent, the control device of the camera controller is used to perform operations such as setting the position of the drive unit such as the PTZ (pan-tilt-zoom) of the imaging device that is operably connected, and other image quality settings, and registering them as preset positions. Also, operations such as calling and deleting the preset function and detailed setting operations of the automatic loop function are possible. In this case, the controller is connected to the imaging devices 100a and 100b via the network 500 or using a serial connection or the like.

[0020] The monitor 300 is a terminal having a display unit such as a display. In this embodiment, the image processing device 200 and the monitor 300 are described as separate entities, but the image processing device 200 may be a general client terminal having a display unit and an operation unit, such as a PC or a tablet.

[0021] Furthermore, the image (video) processed by the image processing device 200 is output to the communication server 400 via the network 500, either through a network connection or a wired connection.

[0022] The communication server 400 is a dedicated server for real-time video distribution via network or wired connections, and can also provide edited video for distribution. The above connection relationship is just one example of use, and the number of servers and connection methods are not limited to this.

[0023] Network 500 is a network connecting imaging device 100a, imaging device 100b, image processing device 200, and communication server 400. Network 500 is implemented using multiple routers, switches, cables, etc., that comply with communication standards such as ETHERNET®. Network 300 may be implemented using the Internet, wired LAN (LOCAL AREA NETWORK), wireless LAN, WAN (WIDE AREA NETWORK), etc.

[0024] <Hardware Configuration> Next, an example of the hardware configuration of imaging devices 100a and 100b will be described in detail using Figure 2. In this embodiment, imaging devices 100a and 100b will be described as having the same hardware configuration.

[0025] The imaging device 100 (imaging devices 100a, 100b) includes an image sensor 110, an image processing engine 120, RAM (Random access memory) 130, and ROM (Read Only Memory) 140. Furthermore, it is composed of a network interface 160, a CPU (central processing unit) 170, and an image output interface 180, and each of the above components is connected to communicate with each other via an internal bus 150.

[0026] The imaging optical system (not shown) is a lens that focuses light from the subject onto the imaging surface of the image sensor, and is composed of, for example, a zoom lens, a focus lens, and an image stabilization lens.

[0027] The image sensor 110 captures an image of a subject incident on it via the imaging optical system and converts the image into an electrical signal. The image sensor 110 can be implemented, for example, by a CCD sensor or a CMOS sensor.

[0028] The image processing engine 120 is connected to the image sensor 110. It converts the video signal output from the image sensor 110 into a digital format, performs image processing and compression as needed, and transfers the digital signal to the RAM 130. The image processing engine 120 also drives the zoom lens (not shown) and focus lens (not shown) of the imaging optical system based on instructions from the CPU 170. By driving the focus lens (not shown), the zoom and focus values ​​can be changed. Furthermore, the exposure can be adjusted by changing the shutter speed and gain of the image sensor 110.

[0029] RAM130 is the main memory of the CPU170, which will be described later, and is used as a temporary storage area for the work area and for deploying various programs. RAM130 may also be used as a storage area for temporarily holding various data, such as image data corresponding to the imaging results from the image sensor 110.

[0030] ROM140 is a non-volatile storage device, such as flash memory, HDD, SSD (Solid State Drive), or SD card. ROM140 is used as a persistent storage area to store the OS, various programs, and various data, including programs used by the CPU170 (described later) to control the controller. It is also used as a short-term storage area to store various setting parameters.

[0031] The network interface 160 is an interface for connecting the aforementioned network 500 and the aforementioned components, and is responsible for communication with external devices such as the image processing device 200 via a communication medium such as Ethernet®. Here, the control of the imaging device 100 may be performed via the network interface 160, or via another interface such as a serial communication interface (not shown).

[0032] The CPU 170 is a central processing unit that provides overall control over the imaging device 100.

[0033] The Image Output I / F180 is an interface for outputting captured images to an external source. For example, it can be configured using SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark). While this example uses the Image Output I / F180 for illustrative purposes, the image could also be output from the Network I / F160.

[0034] Next, an example of the hardware configuration of the image processing device 200 will be described in detail using Figure 2.

[0035] The image processing device 200 has an image input interface 210, RAM 220, user interface 230, ROM 240, CPU 250, network interface 260, and image output interface 270, and each component is connected to communicate with the others via an internal bus 280.

[0036] The image input interface 210 is an interface for receiving captured images from the imaging device 100, and consists of SDI and HDMI.

[0037] RAM220 is a storage device such as DRAM (Dynamic Random Access Memory) that temporarily stores computer programs executed by the CPU. RAM220 also provides a work area where the OS, various programs, and various data are loaded, and which the CPU250 (described later) uses to execute processing. It is also used as a workspace for the OS and various programs.

[0038] The user interface 230 enables the exchange of information between the user and the image processing device 200. Examples of the user interface 230 include output devices such as displays and input devices such as keyboards, mice, and touchpads. The user interface 230 receives input information from the user and transmits it to the CPU 250.

[0039] ROM240 is a non-volatile storage device, such as flash memory, HDD, SSD (Solid State Drive), and SD card. ROM240 is used as persistent storage for the OS, various programs, and data, including programs used by the CPU250 to control the controller. It is also used as short-term storage for various setting parameters.

[0040] The CPU 250 is a central processing unit that provides overall control to the image processing unit 200. The CPU 250 analyzes control-related information (control information) obtained from the user interface 230 and performs necessary data processing. This processing includes reading instructions from the ROM 240 and reading and writing data to the RAM 220.

[0041] The network interface 260 is an interface for connecting to the aforementioned network 500 and is responsible for communication with external devices such as the imaging device 100 via a communication medium such as Ethernet®. In this embodiment, the transmission of an image specified by the user or an acquired image to the distribution server 400 via the network 500 by the network interface 260 is referred to as distribution.

[0042] The image output interface 270 is an interface for outputting data from the image processing device 200 to an external source. For example, it can be configured as an SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark). In the example shown in the diagram, the image output interface 270 is connected to a monitor.

[0043] <Functional block diagram of image processing device> Next, an example of the functional configuration of the image processing device 200 will be explained using Figure 3. The image processing device 20 consists of an image acquisition unit 301, an image registration unit 302, a preset registration unit 303, a storage unit 304, a control unit 305, a region determination unit 306, an image correction unit 307, a synthesis unit 308, an output unit 309, and a user I / F 310. Each function (system) shown in Figure 3 is realized by the CPU 250 reading programs stored in the ROM 240 into the RAM 220 and executing them. In other words, the CPU 250 uses these programs to perform image processing and data calculations. The CPU 250 also executes programs stored in the RAM 220, manages tasks such as reading, processing, and outputting image data, and realizes the operation of the entire system.

[0044] The image acquisition unit 301 acquires images captured by the imaging device 100. It then outputs the acquired images to the storage unit 304. In this embodiment, the image acquisition unit 301 acquires images captured by each imaging device, from the imaging device 100a and the imaging device 100b, and outputs them to the storage unit 304.

[0045] The image registration unit 302 registers an image different from the captured image based on control information obtained from the user interface 310. Furthermore, it outputs the registered image to the storage unit 304. The user can read and rewrite data such as images stored in the storage unit 304 by specifying it via the user interface. In this embodiment, the image registered in the image registration unit 302 is described as an image different from the captured image, but it is not limited to this. For example, it may be the captured image. In this case, the user selects the imaging device to output the image to be registered in the image registration unit 302. For example, when imaging devices 100a and 100b are connected via the network 500, and the user selects imaging device 100a, the image registration unit 302 registers the captured image of imaging device 100a acquired by the image acquisition unit 301. Furthermore, it outputs the image registered in the image registration unit 302 to the storage unit 304.

[0046] The Preset Registration Unit 303 stores the name of the preset entered from the User I / F 310 and layout information when multiple images are combined. Furthermore, it stores information regarding the processing of images acquired from the Image Registration Unit 302 and the Image Acquisition Unit 301, as well as information regarding corrections and compositing processes performed on the images, in association with the preset number.

[0047] In this embodiment, Preset registration refers to registering various control information in association with a preset number. For example, this includes layout information for each image when executing a composite mode (first display mode) that controls multiple images to be displayed on the same screen in association with a predetermined preset number, and control information regarding image quality and correction methods when correcting captured images. Furthermore, if the imaging device 100 is a PTZ (pan-tilt-zoom) capable imaging device, it is also possible to register predetermined PTZ drive unit position information in the Preset registration unit 303. The user can execute control based on the control information associated with the preset number by specifying the preset number. The specific registration process will be described later.

[0048] The storage unit 304 stores various information, including images acquired from the image registration unit 302 and the image acquisition unit 301, as well as layout information associated with the preset numbers registered in the preset registration unit 303.

[0049] The control unit 305 controls each component based on control information acquired via the user interface 310. Specifically, it outputs control commands to the image registration unit 302 and the image acquisition unit 301 to acquire and store images. It also outputs control commands to the region determination unit 306, the image correction unit 307, and the synthesis unit 308 to perform image processing based on layout information corresponding to the preset number input from the user interface 310.

[0050] Furthermore, the control unit 305 can switch display modes by executing control based on a preset number obtained from the user. Specifically, there is a mode (first display mode) that controls the display to show the captured image (first image) obtained from the image acquisition unit 301 and the image registered in the image registration unit 302 (second image) on the same screen. In addition, there is a mode (second display mode) that controls the display to show the image obtained from the image acquisition unit 301 or the image registered in the image registration unit 302. Note that the switching between this first and second display mode is performed based on a preset number input by the user via the user I / F 310, but is not limited to this. For example, the switching may be performed without any operation from the user to instruct the switching based on information based on the order of the preset numbers registered by the user and control information corresponding to the preset number.

[0051] In this embodiment, during the first display mode, the image acquired by the image acquisition unit 301 and the image registered in the image registration unit 302 are combined in the later-described combination unit 308 so that they are displayed on the same screen, but this is not limited to this. For example, the images may be sent individually to the communication server 400 and processed to be displayed on the same screen at the distribution destination.

[0052] The region determination unit 306 performs predetermined processing based on the layout information associated with the preset number. Specifically, if the layout information for the specified preset number is set to the first display mode described above, the region determination unit 306 makes a determination on the image acquired by the image acquisition unit 301. Using a predetermined determination method, the acquired image is classified into a corrected region and an uncorrected region.

[0053] In this context, the uncorrected area refers to the parts of the image acquired by the image acquisition unit 301 that contain information of high importance, while the corrected area refers to the parts of the image acquired by the image acquisition unit 301 that contain information of low importance to the viewer. For example, the uncorrected area is the subject detected in the captured image, and the corrected area is the background area in the captured image.

[0054] Furthermore, the determination method referred to here is the method for determining the area to be corrected from the image acquired from the imaging device. Methods for determining the area to be corrected include manually specifying the corrected and uncorrected areas in advance, or determining the corrected and uncorrected areas based on the depth information of the subject. In addition, the area may be determined using the background subtraction method.

[0055] Furthermore, the determination method may be based on objects or regions set in advance by the user. For example, by having the user pre-set people or specific objects as high-priority objects, the region determination unit 306 will determine the people or objects set by the user as non-corrected regions.

[0056] The image correction unit 307 performs correction on the correction area specified by the area determination unit 306 using a method that has been set in advance. The correction method performed by the image correction unit 307 will be described later.

[0057] The compositing unit 308 creates an image to be output to the monitor 300 based on the information from the Preset registration unit 303, which is associated with the specified preset number. It then outputs the composited image to the output unit 309.

[0058] The output unit 309 converts the image processed by the synthesis unit 308 into a format that can be displayed on the monitor 300 for the user to view as a preview of the distributed image, and outputs it. Furthermore, if the output unit 309 obtains information regarding distribution from the user and receives control information to distribute the image, it outputs the image being displayed on the monitor 300 to the communication server 400. At this time, the image from the synthesis unit 308 is output to the monitor 300 or the communication server 400, but this is not limited to this. For example, if the layout information associated with the specified preset number is in synthesis mode (first display mode), the image obtained from the synthesis unit 308 is output to the monitor 300. Similarly, if the layout information associated with the specified preset number is not in synthesis mode (second display mode), either the image obtained by the image acquisition unit 301 or the image registered in the image registration unit 302 is obtained. This image may then be output to the monitor 300.

[0059] The user interface 310 receives control instructions from the user for the image processing device 200, the imaging device 100a, or the imaging device 100b, and outputs them as control information to the control unit 305.

[0060] <Preset registration process> The preset registration process in the Preset Registration Unit 303 in this embodiment will be explained with reference to Figure 4. The Preset Registration Unit 303 is capable of registering and updating settings related to preset functions. In other words, the user can execute control processes stored in association with a predetermined preset number by specifying the preset number via the User I / F 310. Specifically, the user registers control information in association with a preset number (preset registration). After preset registration, when the user specifies a preset number via the User I / F 310, the control unit 305 executes various controls (preset execution) based on the control information corresponding to the preset number registered in the Preset Registration Unit 303.

[0061] In this embodiment, the user can register settings related to the image to be output to the monitor 300, associated with a preset number.

[0062] The image processing device 200 of this embodiment can register preset settings by associating the preset number with the layout information, the image acquired by the image acquisition unit 301, and the image registered by the image registration unit 302.

[0063] In this embodiment, the image acquisition unit 301 is connected to the imaging device 100a and the imaging device 100b in a controllable manner, and the image acquisition unit 301 acquires the image 410 shown in Figure 4(a) from the imaging device 100a and the image 420 shown in Figure 4(b) from the imaging device 100b.

[0064] Similarly, the image registration unit 302 has the image 430 shown in Figure 5(c) registered in advance.

[0065] At this point, the user selects a preset number they wish to register, and then can specify whether to use the first display mode, which displays multiple images combined, or the second display mode, which displays only a single image, by inputting layout information.

[0066] For example, if a layout information for displaying multiple images on the same screen, which is the first display mode, is specified for preset number 1, the user specifies which images to combine and display on the same screen. In this case, we will explain by registering a preset for preset number 1 to display images 410 and 430 combined on the same screen, as shown in Figure 4(e).

[0067] In this case, the user may display two or more images on the same screen, and any number of images can be selected from multiple images acquired from the image acquisition unit 301 and multiple images registered in the image registration unit 302, as shown in Figure 4(f).

[0068] Furthermore, when the user inputs control information for saving an image to be displayed on the same screen and layout information associated with a preset number, information for creating the image shown in Figure 4(e) is registered in preset number 1. At this time, information regarding the correction process shown in Figure 4(d) performed by the image correction unit 307 may also be registered simultaneously, or it may be determined automatically. The correction process will be described later.

[0069] Next, we will explain the case where layout information for displaying a single image, which is the second display mode, is specified for preset number 2. When layout information for the second display mode is specified, the user specifies the image to be displayed on the screen. In this case, we will explain by registering a preset that displays image 420 as preset number 2.

[0070] At this time, the image selected by the user may be either an image acquired from the image acquisition unit 301 or an image registered in the image registration unit 302.

[0071] Similar to Preset 1, when the user enters control information to associate layout information and image information with a preset number and save it, the preset registration is executed.

[0072] In this way, by performing a preset registration in advance, when the user inputs a user operation to execute a desired preset number after registration, an image associated with that preset number is created. For example, when the user executes preset number 1, the image correction unit 307 performs a correction process on image 410 in a manner described later, according to the layout information, and creates image 431. Furthermore, image 440, which is a composite of image 431 and image 430, is created by the compositing unit 308 and output to the output unit 309 (preset execution). The specific processing method will be described later.

[0073] In this embodiment, a method for registering layout information as a preset has been described, but preset registration is not limited to this. For example, if a PTZ camera is connected as an imaging device, the positions of the pan drive unit, tilt drive unit, and lens drive unit of the PTZ camera can be associated with a preset number and registered as a preset position (setting). When the user inputs a predetermined preset number from the user I / F 310 with preset linkage turned on, the control unit 305 executes the control corresponding to that preset number. Here, "preset" refers to a function for recalling settings and operation history stored in the image processing device. For example, if a control instruction for the imaging device is associated with a preset number, a preset recall instruction will be output to the imaging device. When the imaging device receives a control command instructing the registration of a preset including a preset number, it records the current position and setting information on the terminal side. After registration, when the image processing device 200 transmits control information including the preset number to the imaging device, the imaging device recalls the stored position and setting and reflects the movement and settings to the preset position. Similarly, if the control information including the preset number does not include the imaging device, the control unit 305 can execute control in accordance with the control information based on the control history stored in the storage unit 304.

[0074] By registering presets in this way, it becomes possible to create images for distribution based on information previously set by the user when the preset execution command is issued.

[0075] <Explanation and effects of the correction> The correction shown in this embodiment refers to a processing operation performed on a correction area other than the image registered in the image registration unit 302 or the non-correction area, which is specified by the user and contains, for example, information of high importance.

[0076] By performing this correction, even when using images that combine multiple images, the correction process can draw the viewer's attention to the image they are interested in. Examples of processing include blurring, brightness adjustment, overlaying other images, grayscale conversion (grayscale processing), saturation adjustment, and brightness adjustment.

[0077] This correction is achieved by having the area determination unit 306 determine the correction area from the image acquired by the image acquisition unit 301. The correction process is performed on the area determined to be the correction area by the area determination unit 306. For example, let's consider the case where the background area is determined to be the correction area for image 410 in Figure 4(a). The correction area determination unit determines the correction area 411 and the non-correction area 412 for image 410. Furthermore, the image correction unit 307 performs a correction process on the correction area 411 to create image 431. By performing this correction process, it becomes possible to create a corrected image (corrected image) like the image 431 shown in Figure 4(d). The type of correction may be set by the user, or a pre-set process may be executed.

[0078] <Blur effect> One example of this correction process is blurring, which smooths out a portion of an image. This process uses an averaging filter, a weighted averaging filter, and a Gaussian filter. These filters apply a weight to each pixel of the image to create a blurring effect. By applying blurring in this way, even when using images that combine multiple images, the corrected area will appear visually smoother in terms of contrast than the uncorrected area, making it easier for viewers to focus on the uncorrected area.

[0079] <Explanation of Brightness Adjustment> One example of this correction process is brightness adjustment, an image processing technique that changes the brightness of an image. In the case of RGB data, brightness is adjusted by adding or subtracting a fixed value to the value of each pixel. In the case of YCC data, brightness is adjusted by adding or subtracting an arbitrary value to the Y value (brightness value) of each pixel. In addition, the aforementioned Gaussian filter may be used to equalize the brightness. By performing brightness adjustment in this way, even when using images that combine multiple images, the corrected area will appear visually darker or brighter than the uncorrected image. Therefore, it is possible to reduce the visibility of information contained in the corrected area, making it easier for viewers to focus on the uncorrected area.

[0080] <Explanation of grayscale conversion> One example of this correction process is grayscale conversion (grayscale processing), a technique that represents a color image using only shades of black and white, employing the luminance method, average method, and maximum value method. By performing grayscale conversion in this way, even when using images that combine multiple images, the corrected area is converted to a grayscale image. Therefore, the amount of color information in the corrected area is reduced compared to the uncorrected area, making it possible to create an image where the uncorrected area is easier to focus on.

[0081] <Explanation of saturation adjustment> Saturation adjustment can be done by adding or subtracting the color components of each pixel equally, or by adding or subtracting the values ​​of the color difference components. By adjusting the saturation, even when using images that combine multiple images, the corrected areas will appear visually duller, making it possible to create captured images where the uncorrected areas are easier to focus on.

[0082] <Explanation of brightness adjustment> Brightness adjustment affects the brightness of an image. Brightness is calculated from the average of the maximum and minimum color components of a pixel. By adjusting the calculated brightness values, the correction area will appear visually darker or brighter than the uncorrected image, even when using images that combine multiple images. As a result, it is possible to reduce the visibility of information contained in the correction area, making it possible to create captured images where the uncorrected area is easier to focus on.

[0083] <Explanation of filling in> The fill function fills the correction area with a specified color. By using the fill function, even when combining multiple images, the correction area is filled with the predetermined color, making it possible to create images where the uncorrected areas are easily noticeable. The color used can be anything and can be changed by the user.

[0084] <Explanation of image superposition> Even when using an image that combines multiple images, the image superposition will be performed using an image that is not registered in the image registration unit 302 and has not been acquired from the image acquisition unit 301 for the correction region. This process makes the correction region invisible when the image is superimposed, thus enabling the creation of an image that makes it easier to focus on the uncorrected region.

[0085] <Image correction processing> Next, the image correction method in this embodiment will be explained using Figures 4 and 5. Figure 4 is an illustrative diagram of the image correction process, and Figure 5 is a flowchart related to this control.

[0086] This flowchart is implemented when the OS, various programs, and various data are loaded into RAM220, which temporarily stores the computer program executed by CPU250, and then executed by CPU250. Furthermore, this flowchart starts by acquiring control information for preset execution and ends by acquiring control information from the user to stop the acquisition of captured images.

[0087] In step S501, based on the control information for executing the preset obtained from the user, the control unit 305 obtains the preset information registered in the Preset registration unit 303. After obtaining the information, the process proceeds to step S502. If the control information for executing the preset cannot be obtained, this process may be repeated until the control information is obtained.

[0088] In step S502, the control unit 305 obtains layout information related to the specified preset number from the Preset registration unit 303. If the layout information corresponds to the first display mode, the process proceeds to step S503; otherwise, the process proceeds to step S506.

[0089] In step S503, the control unit 305 retrieves the image acquired by the image acquisition unit 301 and the image registered in the image registration unit 302 from the storage unit 304 based on the preset number information. At this time, it is explained that there are images registered in the image registration unit 302, but if there are no registered images, the process may proceed to step S501. The user may also be notified. The notification method at this time may be controlled by the control unit 250 to display on the monitor 300, or controlled by the control unit 250 to notify by voice, and is not limited to these.

[0090] In step S504, the region determination unit 306 determines the corrected region 411 and the uncorrected region 412 of the image 410 acquired from the image acquisition unit 301 in step S501. The method for determining each region can be any of the following: segmentation method, edge detection, background subtraction, deep learning, clustering, graph-based method, optical flow method, or use of depth information. Furthermore, for the segmentation method, color space division method, region growth method, superpixel method, edge detection, or image depth information may be used. After the determination, the process proceeds to step S505.

[0091] In step S505, the image correction unit 307 performs the corrections specified in advance by the user on the correction area 411 of the image 410 determined in step S504, and creates image 431. After correcting image 410, the process proceeds to step S506. Regarding the uncorrected area 412, correction is not required as shown in the example; instead, the contours may be emphasized to make the person's features clearer, or the sharpness settings may be changed.

[0092] In step S506, the synthesis unit 308 creates an image based on the layout information set in the Preset registration unit 303. For example, if the first display mode is set, the corrected image 431 (corrected image) created in step S505 and the image 430 registered in the image registration unit 302 are retrieved from the storage unit 304, and a composite image is created in step S506. Furthermore, the composite image 440 is output to the communication server 400 via the network 500, and the process proceeds to step S507.

[0093] In step S507, it is determined whether a termination command to stop acquiring captured images has been received from the user via the user interface 310. If a termination command has been received, this flow is terminated; otherwise, the process proceeds to step S501.

[0094] By performing this correction process when controlling the display of important materials and captured images on the same screen, it is possible to deliver the desired images while reducing the burden on the user during distribution.

[0095] <Description of the image editing UI for distribution using user-registered presets> Next, the UI of the image processing apparatus in the first embodiment of the present invention will be described with reference to Figure 6.

[0096] Figure 6(a) shows an example of the GUI displayed on monitor 300, specifically an example of the GUI screen when PresetA is selected. Here, PresetA is the name that the user has assigned to preset number 1.

[0097] The image area 600 is an area for displaying information obtained by the image registration unit 302 and the image acquisition unit 301.

[0098] The output image area 610 is the area that displays the image output to the output unit 309 after executing preset number 1 based on the information in the preset registration unit 303 and executing the flow shown in Figure 5. In the output image area 610, you can check the image output after a synthesis process has been performed in which the image acquired from the image acquisition unit 301 based on the information registered in Preset A (preset number 1) is corrected and the image registered in the image registration unit 302 is combined.

[0099] The control panel area 620 is a button that displays the name of the preset name 621 selected by the user based on the information from the preset registration unit 303. It is also a button that displays the name of the acquired image within an area divided based on the placement information of the image acquired from the preset registration unit, that is, the image acquired from the image registration unit 302 and the image acquisition unit 301. In Figure 6, the area that displays the image 612 acquired by the image registration unit 302 is designated as Area A, and the area that displays the image 613, which is the image acquired by the image acquisition unit 301 and corrected, is designated as Area B.

[0100] When a user selects button 622 or 623 in each area, they can edit the layout to any desired configuration from the layout information registered in the Preset Registration section.

[0101] 601 is an image acquired by the image acquisition unit 301, and in this embodiment, it is an image captured by the imaging device 100a.

[0102] 602 displays the name of the imaging device 100a that was registered in the Preset registration unit 303 in association with the image acquired by the image acquisition unit 301. 603 is an image acquired by the image acquisition unit 301, and in this embodiment, it is an image captured by the imaging device 100b.

[0103] 604 displays the name of the imaging device 100b that was registered in the Preset registration unit 303 in association with the image acquired by the image acquisition unit 301. 605 indicates an image that has been previously registered in the image registration unit 302.

[0104] 606 displays the file name of an image previously registered in the image registration unit 302.

[0105] 611 displays an output image that is a composite of image 605 from the image registration unit 302 and image 603 acquired from the image acquisition unit 301, and image 613 which has been corrected by the image correction unit 307.

[0106] The Previous button 624 is used to execute the preset that was registered immediately before PresetA when multiple presets are registered. In this case, the user may set the order of the presets.

[0107] The NEXT button 625 is used to execute the preset that is registered immediately after PresetA when multiple presets are registered.

[0108] Figure 6(b) shows the GUI after the Previous button 624 is pressed, transitioning from PresetA (preset number 1) to PresetB (preset number 3) from the state shown in Figure 6(a). Preset B (preset number 3) shows an example where a third area is added, and one GUI button 626 is added for selecting the image to be captured. In this way, it is possible to control the display of images from multiple imaging devices and registered images on the same screen, and it is also possible to change the imaging device being used by pressing buttons 623 or 626 during execution.

[0109] Figure 6(c) shows an example of the GUI screen after the user selects the NEXT button 625 in Figure 6(a) and transitions from presetA to presetC (preset number 2).

[0110] In this way, users can easily switch between images being delivered by specifying a preset number or by pressing the NEXT button 625 or Previous button 624. Furthermore, when controlling the display of multiple images on the same screen (first display mode), the correction area is automatically determined, and the corrected image (corrected image) and the registered image are displayed on the same screen without the user having to perform the correction process.

[0111] <Second Embodiment> Next, the image processing control according to the second embodiment will be described using Figures 7, 8, and 9. The second embodiment is an embodiment in which the image processing described in the first embodiment is applied to a video conferencing system. Explanations of redundant content will be omitted.

[0112] Figure 7 is an illustrative diagram showing an example of the system configuration in this embodiment, and Figure 9 is an illustrative diagram of the output image. Figure 9 is also a flowchart of the image correction process.

[0113] The hardware configuration and functional block diagram of the image processing device 200 are the same as in Embodiment 1, so their explanation will be omitted.

[0114] Figure 7 shows that imaging devices 100a to 100c, which capture images of each participant, and external devices 600a to 600c, which are information processing devices that allow each participant to view the distributed images, are connected to each imaging device 100 so that they can communicate with each other. In this embodiment, external devices 600a to 600c are general client terminals such as PCs and tablets, which have a display unit and an operation unit. In this embodiment, a PC with an integrated monitor and input device (user interface) is shown as an example, but separate devices may also be used.

[0115] For example, an input terminal such as a joystick may be connected to an external device 600 via USB or Bluetooth® and used as an input for remotely controlling the imaging device 100.

[0116] In this embodiment, participants output images 810a to 810c (first images) captured by imaging device 100c from each imaging device 100a to the image processing device 200 via the network 500. Participants can also output images other than captured images (second images), such as materials, to the image processing device 200 via the network 500. In this embodiment, it is assumed that participants output captured images, but this is not limited to this. For example, the output of captured images can be controlled, and it is possible for a participant who does not output captured images to output only second images, or to acquire only images distributed by the image processing device 200.

[0117] In this embodiment, the image registration unit 302 can acquire images other than the captured image 100, such as images displayed on the screen of the external device 600 or documents, from the participant. After acquisition, it outputs to the storage unit 304.

[0118] Figure 8(a) shows an image 810a (first image) acquired from the imaging device 100a that captures each participant, and is acquired by the image acquisition unit 301.

[0119] Figure 8(b) shows an example of an image 820 (second image), which is different from the captured image and is used as explanatory material shared by presenters among the participants. This image is acquired by the image registration unit 302.

[0120] In step S901, the image acquisition unit 301 determines whether it has acquired an image from the imaging device 100. If it has acquired one or more images from images 810a to 810c acquired by imaging device 100c from each imaging device 100a, it proceeds to step S902. If it has not acquired any images, it proceeds to step S907.

[0121] In step S902, it is determined whether there are any images registered in the image registration unit 302. If there are any images registered in the image registration unit 302, the process proceeds to step S903; otherwise, it proceeds to step S907. In other words, if at least one participant outputs an image different from the captured image to the image processing device 200 via the network 500, the process proceeds to step S903. Specifically, when a participant set as a presenter outputs a specified image to the image processing device 200 via the network 500 in order to control the distribution of materials on their PC or the screen of an external device 600 to other participants in real time, the image registration unit 302 acquires it. At this time, it is determined in step S902 that there are images registered in the image registration unit 302, and the process proceeds to step S903. If there are no registered images, the process proceeds to step S908.

[0122] In step S903, the control unit 305 determines whether the participant's display settings are set to a mode that controls the display of each image 810a to 810c acquired by the image acquisition unit 301 and the image 820 acquired by the image registration unit 302 on the same screen (first display mode). The display settings (display mode) can be changed by operating a setting UI (not shown) from the user I / F 310. If the mode that controls the display to be on the same screen (first display mode) is set, the process transitions to S904; otherwise, the process transitions to S908. In this embodiment, this process is described as being performed based on the participant's display settings, but it is not limited to this. For example, if there are images acquired from both the image acquisition unit 301 and the image registration unit 302, it may be automatically determined to be the first display mode. In this case, if either the image acquisition unit 301 or the image registration unit 302 has acquired an image, it will be determined to be the second display mode.

[0123] In step S904, the region determination unit 306 determines that the images 810a to 810c acquired by the image acquisition unit 301 are divided into a corrected region 811 and an uncorrected region 812. In Figure 8(a), the region determination unit 306 determines that the corrected region (shaded area) 811a and the uncorrected region 812a of image 810a are divided into a corrected region (shaded area) 811a and an uncorrected region 812a. After the determination, the process proceeds to step S905.

[0124] In step S905, the image correction unit 308 performs a pre-specified correction on the correction region 811a determined in step S904. For example, it creates an image 431 (corrected image) by applying the correction process to image 410 (image 810a) and outputs it to the storage unit 304. After the correction process on the correction region, the process proceeds to step S906.

[0125] In step S906, the image 531 (corrected image) corrected in step S905 and the explanatory materials 820 acquired by the image registration unit 302 are output to the external device 600 via the network 500. At this time, a composite image 830 as shown in Figure 8(c) may be created by the compositing unit 308, or the images may be output from the external device 600a to the external device 600c without compositing them. After output, the process proceeds to step S909.

[0126] In step S907, it is determined whether there are any images registered in the image registration unit 302. The specific method is the same as in step S902, so the explanation is omitted. If there are any images registered in (or acquired by) the image registration unit 302, the process proceeds to step S908; otherwise, the process proceeds to step S901.

[0127] In step S908, the image acquired by the image acquisition unit 301 is not corrected, and either the image acquired by the image acquisition unit 301 or the image acquired by the image registration unit 302 is output from the external device 600a to the external device 600c for display as is. If explanatory materials are registered in the image registration unit 302, an image of the explanatory materials alone is output as shown in Figure 8(d). If the explanatory materials are not shared (No in step S902), an image 850 of a single participant (e.g., speaker) captured by the imaging device is output as shown in Figure 8(e). Alternatively, if images are acquired from multiple imaging devices, images 860 of multiple participants are output to the external device 600 so that they are displayed as shown in Figure 8(f). After output, the process proceeds to step S909.

[0128] In step S909, it is determined whether a termination command to stop acquiring captured images has been received from the user via the user interface 310. If a termination command has been received, this flow is terminated; otherwise, the process proceeds to step S901.

[0129] In this embodiment, the uncorrected region 812a does not need to be corrected as shown in the figure example, or its outline may be emphasized or the sharpness setting changed so that the person's appearance is easily conveyed to the broadcaster.

[0130] By performing this correction process when controlling the display of important materials and captured images on the same screen, it is possible to deliver the desired images while reducing the burden on the user during distribution.

[0131] In this embodiment, the presenter distributes the image from the imaging device and the explanatory materials as separate images, and the case where correction is applied or not is switched according to each user's display mode setting has been described. However, this is not the only case; even if the distribution side pre-combines the image from the imaging device and the explanatory materials into a single image and distributes it, the audience can split and judge the images and perform similar correction processing on the image from the imaging device.

[0132] In summary, by performing this correction, even when using images that combine multiple images, the correction process can draw the participant's attention to the image they are interested in. [Explanation of Symbols]

[0133] 100 Imaging device 200 Image Processing Devices 300 monitors

Claims

1. The first image captured by the imaging means, Image acquisition means for acquiring a second image different from the first image, A control means for controlling the display means to display a predetermined image, It has, The control means If the first image and the second image have been acquired, the display means is controlled to display the corrected image, which is obtained by correcting at least a portion of the first image, and the second image. An image processing apparatus characterized by controlling the display means to display the acquired image if either the first image or the second image has been acquired.

2. The control means A first display mode controls the display means to display the corrected image obtained by correcting the first image and the second image on the same screen, The image processing apparatus according to claim 1, characterized in that it has a second display mode for controlling the display means to display either the first image or the second image.

3. The image processing apparatus according to claim 2, characterized in that the second display mode does not correct the first image.

4. A determination means for determining the correction region in the first image, The determination means includes a correction means for correcting the region determined to be the correction region, The image processing apparatus according to claim 1, characterized by having

5. The correction by the correction means is at least one of the following: blurring, brightness adjustment, grayscale processing, saturation adjustment, lightness adjustment, fill processing, and image superposition. The image processing apparatus according to claim 1, characterized in that the image used for superimposing the aforementioned image is neither the first image nor the second image.

6. The image processing apparatus according to claim 4, characterized in that the determination means determines the background portion of the captured image as a correction region.

7. The image processing apparatus according to claim 4, characterized in that the determination means determines a correction region based on an object set by the user in the captured image.

8. A conference system comprising one or more information processing devices and image processing devices having imaging means, The aforementioned information processing device is Output means for outputting a first image captured by the imaging means and a second image different from the first image to the image processing device, Control means for controlling the display of an image acquired from the image processing device, It has, The aforementioned image processing device is The first image obtained from one or more of the information processing devices, An acquisition means for acquiring a second image different from the first image, It has, If the first image and the second image are acquired from at least one of the information processing devices, the corrected image obtained by correcting at least a portion of the first image and the second image are output to the information processing device. A conference system characterized in that, if either the first image or the second image is obtained from at least one of the information processing devices, the acquired image is output to the information processing device.

9. The aforementioned image processing apparatus Correction means for correcting at least a portion of the first image, The system includes a determination means for determining a correction region in at least one of the first images, The correction means corrects the region determined to be the correction region by the determination means. The image processing apparatus according to claim 7, characterized in that...

10. The correction by the correction means is at least one of the following: blurring, brightness adjustment, grayscale processing, saturation adjustment, lightness adjustment, fill processing, and image superposition. The image processing apparatus according to claim 7, characterized in that the image used for superimposing the aforementioned image is neither the captured image acquired from the information processing apparatus nor an image different from the captured image.

11. The first image captured by the imaging device, An image acquisition step to acquire a second image different from the first image, A control step that controls the display device to display a predetermined image, It has, In the image acquisition process described above, If the first image and the second image have been acquired, in the control step The system is controlled to display the corrected image, which is obtained by correcting at least a portion of the first image, and the second image on the display device. In the image acquisition process described above, An image processing method that, if either the first image or the second image has been acquired, controls the device to display the acquired image on the display device in the control step.

12. A program for causing a computer to function as an image processing device according to any one of claims 1 to 7 and 9 to 10.

Images