Image processing device, conferencing system, image processing method, and program

Abstract

When combining multiple images for output, the system delivers the desired image while reducing the user's burden. [Solution] An image processing apparatus comprising: an image acquisition means for acquiring a first image captured by an imaging means and a second image different from the first image; a control means for controlling a display means to display a predetermined image; a calculation means for calculating an evaluation value for the first image; and a determination means for determining whether or not to correct at least a part of the first image based on the evaluation value, wherein when the image acquisition means has acquired the first image and the second image, and the determination means determines that at least a part of the first image should be corrected, the control means controls the display means to display a corrected image in which at least a part of the first image has been corrected and the second image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a conference system, an image processing method, and a program.

Background Art

[0002] In recent years, when holding a conference or lecture for people participating in a physical venue, it has become common to simultaneously distribute images of the venue to online participants. For example, there is a technology that uses an imaging device to capture the speaker and the venue for distribution and processes a part of the captured image for distribution.

[0003] Patent Document 1 discloses a technology for extracting people, objects, and backgrounds in a venue, synthesizing them for distribution, and outputting them, and a technology that allows a user to manually switch the presence or absence of image synthesis and background correction for a selected image.

[0004] Also, when distributing a captured image, when displaying the captured image of the venue scene or people and the presentation materials such as the PC screen on the same screen for distribution, it is necessary to process the distribution image so that the viewer can easily focus on the presentation materials.

[0005] For example, when the presentation materials and the venue scene are displayed on the same screen, the visibility of the presentation materials can be enhanced by performing correction processing on images of the venue scene with low importance.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, in the prior art disclosed in the aforementioned patent documents, when combining an image with a different image, manual compositing is required. Similarly, when correcting the background of an image, the user must manually select whether or not to perform the correction. As a result, there may be a period of time during the operation when unintended information is included in the delivery screen.

[0008] In view of the above technology, the object of the present invention is to provide a technology that makes it possible to switch whether or not to perform correction processing based on the captured image when outputting a combination of multiple images including an captured image. [Means for solving the problem]

[0009] An image processing apparatus comprising: an image acquisition means for acquiring a first image captured by an imaging means and a second image different from the first image; a control means for controlling a display means to display a predetermined image; a calculation means for calculating an evaluation value for the first image; and a determination means for determining whether or not to correct at least a part of the first image based on the evaluation value, wherein when the image acquisition means has acquired the first image and the second image, and the determination means determines that at least a part of the first image should be corrected, the control means controls the display means to display a corrected image with at least a part of the first image corrected and the second image; and when either the first image or the second image has been acquired, the control means controls the display means to display the acquired image. [Effects of the Invention]

[0010] According to the present invention, when outputting a combination of multiple images including an captured image, it is possible to switch whether or not to perform correction processing based on the captured image. [Brief explanation of the drawing]

[0011] [Figure 1] This is a system configuration diagram in the first embodiment. [Figure 2] This figure shows an example of the hardware configuration of the image processing device and imaging device in the first embodiment. [Figure 3] This figure shows an example of a functional block diagram of an image processing device in the first embodiment. [Figure 4] This is an image diagram of the first embodiment. [Figure 5] This is a flowchart for performing the correction process in the first embodiment. [Figure 6] This figure shows an example of a UI screen in the first embodiment. [Figure 7] This is a system configuration diagram in the second embodiment. [Figure 8] This is an image diagram of the second embodiment. [Figure 9] This is a flowchart for performing the correction process in the second embodiment. [Modes for carrying out the invention]

[0012] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0013] <First Embodiment> Figure 1 is a diagram showing the configuration of an imaging system according to the first embodiment. The imaging system in this embodiment includes an imaging device 100a, an imaging device 100b, an image processing device 200, a monitor 300, a communication server 400, and a network 500. The method of connecting each device is not limited to a specific method. For example, they may be connected by wired cables.

[0014] The imaging devices 100a and 100b transmit image data based on the imaging results to the image processing device 200 via the network 500. Here, for the sake of explanation, there are two imaging devices, 100a and 100b, but the number is not limited, and there may be one or more.

[0015] The image processing device 200 is connected to the imaging devices 100a, 100b and the communication server 400 via the network 500. Also, there is no limitation on the connection type of the network 500.

[0016] The image processing device 200 performs image processing (details will be described later) on the images received from the imaging devices 100a and 100b and outputs them to the server 400 via the network 500. Here, the image processing device 200 is set to output to the server 400 for distribution, but it may not be necessary to output to the server 400, such as when creating a video for recording.

[0017] The signal input to the image processing device 200 is not limited to the case where the imaging device 100 is directly connected in the previous stage, and it is also possible to input image data via other processing devices such as a switcher and an effector, and a system having a layout change function such as a video conferencing system.

[0018] Also, the output image data obtained by performing the image processing described later in the image processing device 200 can be output to the monitor 300.

[0019] Also, the image processing device 200 will be described as being integrated with hardware having an operation unit that enables operations on a camera or an output image, such as a camera controller, but it may be independent as a separate entity.

[0020] If the system is independent, the camera controller's control unit is used to set the position of the PTZ (pan, tilt, zoom) and other drive units of the connected imaging device, as well as other image quality settings, and register them as preset positions. It is also possible to recall and delete preset functions, and to perform detailed settings operations for the automatic loop function. In this case, the controller is connected to imaging devices 100a and 100b via network 500 or via serial connection, etc.

[0021] The monitor 300 is a terminal having a display unit such as a screen. In this embodiment, the image processing device 200 and the monitor 300 are described as separate components, but the image processing device 200 may be a general client terminal having a display unit such as a PC or tablet and an operation unit.

[0022] Furthermore, the image (video) processed by the image processing device 200 is output to the communication server 400 via the network 500, either through a network connection or a wired connection.

[0023] The communication server 400 is a dedicated server for real-time video distribution via network or wired connections, and can also provide edited video for distribution. The above connection relationship is just one example of use, and the number of servers and connection methods are not limited to this.

[0024] Network 500 is a network connecting imaging device 100a, imaging device 100b, image processing device 200, and communication server 400. Network 500 is implemented using multiple routers, switches, cables, etc., that comply with communication standards such as ETHERNET®. Network 300 may be implemented using the Internet, wired LAN (LOCAL AREA NETWORK), wireless LAN, WAN (WIDE AREA NETWORK), etc.

[0025] <Hardware Configuration> Next, an example of the hardware configuration of imaging devices 100a and 100b will be described in detail using Figure 2. In this embodiment, imaging devices 100a and 100b will be described as having the same hardware configuration.

[0026] The imaging device 100 (imaging devices 100a, 100b) includes an image sensor 110, an image processing engine 120, RAM (Random access memory) 130, and ROM (Read Only Memory) 140. Furthermore, it is composed of a network interface 160, a CPU (central processing unit) 170, and an image output interface 180, and each of the above components is connected to communicate with each other via an internal bus 150.

[0027] The imaging optical system (not shown) is a lens that focuses light from the subject onto the imaging surface of the image sensor, and is composed of, for example, a zoom lens, a focus lens, and an image stabilization lens.

[0028] The image sensor 110 captures an image of a subject incident on it via the imaging optical system and converts the image into an electrical signal. The image sensor 110 can be implemented, for example, by a CCD sensor or a CMOS sensor.

[0029] The image processing engine 120 is connected to the image sensor 110. It converts the video signal output from the image sensor 110 into a digital format, performs image processing and compression as needed, and transfers the digital signal to the RAM 130. The image processing engine 120 also drives the zoom lens (not shown) and focus lens (not shown) of the imaging optical system based on instructions from the CPU 170. By driving the focus lens (not shown), the zoom and focus values ​​can be changed. Furthermore, the exposure can be adjusted by changing the shutter speed and gain of the image sensor 110.

[0030] RAM130 is the main memory of the CPU170, which will be described later, and is used as a temporary storage area for the work area and for deploying various programs. RAM130 may also be used as a storage area for temporarily holding various data, such as image data corresponding to the imaging results from the image sensor 110.

[0031] ROM140 is a non-volatile storage device, such as flash memory, HDD, SSD (Solid State Drive), or SD card. ROM140 is used as a persistent storage area to store the OS, various programs, and various data, including programs used by the CPU170 (described later) to control the controller. It is also used as a short-term storage area to store various setting parameters.

[0032] The network interface 160 is an interface for connecting the aforementioned network 500 and the aforementioned components, and is responsible for communication with external devices such as the image processing device 200 via a communication medium such as Ethernet®. Here, the control of the imaging device 100 may be performed via the network interface 160, or via another interface such as a serial communication interface (not shown).

[0033] The CPU 170 is a central processing unit that provides overall control over the imaging device 100.

[0034] The Image Output I / F180 is an interface for outputting captured images to an external source. For example, it can be configured using SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark). While this example uses the Image Output I / F180 for illustrative purposes, the image could also be output from the Network I / F160.

[0035] Next, an example of the hardware configuration of the image processing device 200 will be described in detail using Figure 2.

[0036] The image processing device 200 has an image input interface 210, RAM 220, user interface 230, ROM 240, CPU 250, network interface 260, and image output interface 270, and each component is connected to communicate with the others via an internal bus 280.

[0037] The image input interface 210 is an interface for receiving captured images from the imaging device 100, and consists of SDI and HDMI.

[0038] RAM220 is a storage device such as DRAM (Dynamic Random Access Memory) that temporarily stores computer programs executed by the CPU. RAM220 also provides a work area where the OS, various programs, and various data are loaded, and which the CPU250 (described later) uses to execute processing. It is also used as a workspace for the OS and various programs.

[0039] The user interface 230 enables the exchange of information between the user and the image processing device 200. Examples of the user interface 230 include output devices such as displays and input devices such as keyboards, mice, and touchpads. The user interface 230 receives input information from the user and transmits it to the CPU 250.

[0040] ROM240 is a non-volatile storage device, such as flash memory, HDD, SSD (Solid State Drive), and SD card. ROM240 is used as persistent storage for the OS, various programs, and data, including programs used by the CPU250 to control the controller. It is also used as short-term storage for various setting parameters.

[0041] The CPU 250 is a central processing unit that provides overall control to the image processing unit 200. The CPU 250 analyzes control-related information (control information) obtained from the user interface 230 and performs necessary data processing. This processing includes reading instructions from the ROM 240 and reading and writing data to the RAM 220.

[0042] The network interface 260 is an interface for connecting to the aforementioned network 500 and is responsible for communication with external devices such as the imaging device 100 via a communication medium such as Ethernet®. In this embodiment, the transmission of an image specified by the user or an acquired image to the distribution server 400 via the network 500 by the network interface 260 is referred to as distribution.

[0043] The image output interface 270 is an interface for outputting data from the image processing device 200 to an external source. For example, it can be configured as an SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark). In the example shown in the diagram, the image output interface 270 is connected to a monitor.

[0044] <Functional block diagram of image processing device> Next, an example of the functional configuration of the image processing device 200 will be explained using Figure 3. The image processing device 20 consists of an image acquisition unit 301, an image registration unit 302, a preset registration unit 303, a storage unit 304, a control unit 305, a region determination unit 306, an image correction unit 307, a synthesis unit 308, an output unit 309, a user I / F 310, and a calculation unit 311. Each function (system) shown in Figure 3 is realized by the CPU 250 reading programs stored in the ROM 240 into the RAM 220 and executing them. In other words, the CPU 250 uses these programs to perform image processing and data calculations. The CPU 250 also executes programs stored in the RAM 220, manages tasks such as reading, processing, and outputting image data, and realizes the operation of the entire system.

[0045] The image acquisition unit 301 acquires images captured by the imaging device 100. It then outputs the acquired images to the storage unit 304. In this embodiment, the image acquisition unit 301 acquires images captured by each imaging device, from the imaging device 100a and the imaging device 100b, and outputs them to the storage unit 304.

[0046] The image registration unit 302 registers an image different from the captured image based on control information obtained from the user interface 310. Furthermore, it outputs the registered image to the storage unit 304. The user can read and rewrite data such as images stored in the storage unit 304 by specifying it via the user interface. In this embodiment, the image registered in the image registration unit 302 is described as being different from the captured image, but it is not limited to this. For example, it may be the captured image. In this case, the user selects the imaging device to output the image to be registered in the image registration unit 302. For example, when imaging devices 100a and 100b are connected via the network 500, and the user selects imaging device 100a, the image registration unit 302 registers the captured image of imaging device 100a acquired by the image acquisition unit 301. Furthermore, it outputs the image to the storage unit 304.

[0047] The Preset Registration Unit 303 stores the name of the preset entered from the User I / F 310 and layout information when multiple images are combined. Furthermore, it stores information regarding the processing of images acquired from the Image Registration Unit 302 and the Image Acquisition Unit 301, as well as information regarding corrections and compositing processes performed on the images, in association with the preset number.

[0048] In this embodiment, Preset registration refers to registering various control information in association with a preset number. For example, this includes layout information for each image when executing a composite mode (first display mode) that controls multiple images to be displayed on the same screen in association with a predetermined preset number, and control information regarding image quality and correction methods when correcting captured images. Furthermore, if the imaging device 100 is a PTZ (pan-tilt-zoom) capable imaging device, it is also possible to register predetermined PTZ drive unit position information in the Preset registration unit 303. The user can execute control based on the control information associated with the preset number by specifying the preset number. The specific registration process will be described later.

[0049] The storage unit 304 stores various information, including images acquired from the image registration unit 302 and the image acquisition unit 301, as well as layout information associated with the preset numbers registered in the preset registration unit 303.

[0050] The control unit 305 controls each component based on control information acquired via the user interface 310. Specifically, it outputs control commands to the image registration unit 302 and the image acquisition unit 301 to acquire and store images. It also outputs control commands to the region determination unit 306, the image correction unit 307, and the synthesis unit 308 to perform image processing based on layout information corresponding to the preset number input from the user interface 310.

[0051] Furthermore, the control unit 305 can switch display modes by executing control based on a preset number obtained from the user. Specifically, there is a mode (first display mode) that controls the display to show the captured image (first image) obtained from the image acquisition unit 301 and the image registered in the image registration unit 302 (second image) on the same screen. In addition, there is a mode (second display mode) that controls the display to show the image obtained from the image acquisition unit 301 or the image registered in the image registration unit 302. Note that the switching between the first and second display modes is performed based on a preset number input by the user via the user interface, but is not limited to this. For example, the switching may be performed without any user instruction operation based on information based on the order of preset numbers registered by the user and control information corresponding to the preset number.

[0052] In this embodiment, during the first display mode, the image acquired by the image acquisition unit 301 and the image registered in the image registration unit 302 are combined in the later-described combination unit 308 so that they are displayed on the same screen, but this is not limited to this. For example, the images may be sent individually to the communication server 400 and processed to be displayed on the same screen at the distribution destination.

[0053] The region determination unit 306 performs predetermined processing based on the layout information associated with the preset number. Specifically, if the layout information for the specified preset number is set to the first display mode described above, the region determination unit 306 makes a determination on the image acquired by the image acquisition unit 301. Using a predetermined determination method, the acquired image is classified into a corrected region and an uncorrected region.

[0054] In this context, the uncorrected area refers to the parts of the image acquired by the image acquisition unit 301 that contain information of high importance, while the corrected area refers to the parts of the image acquired by the image acquisition unit 301 that contain information of low importance to the viewer. For example, the uncorrected area is the subject detected in the captured image, and the corrected area is the background area in the captured image.

[0055] Furthermore, the determination method referred to here is the method for determining the area to be corrected from the image acquired from the imaging device. Methods for determining the area to be corrected include manually specifying the corrected and uncorrected areas in advance, or determining the corrected and uncorrected areas based on the depth information of the subject. In addition, the area may be determined using the background subtraction method.

[0056] Furthermore, the determination method may be based on objects or regions set in advance by the user. For example, by having the user pre-set people or specific objects as high-priority objects, the region determination unit 306 will determine the people or objects set by the user as non-corrected regions.

[0057] The image correction unit 307 performs correction on the correction area specified by the area determination unit 306 using a method that has been set in advance. The correction method performed by the image correction unit 307 will be described later.

[0058] The compositing unit 308 creates an image to be output to the monitor 300 based on the information from the Preset registration unit 303, which is associated with the specified preset number. It then outputs the composited image to the output unit 309.

[0059] The output unit 309 converts the image processed by the synthesis unit 308 into a format that can be displayed on the monitor 300 for the user to view as a preview of the distributed image, and outputs it. Furthermore, if the output unit 309 obtains information regarding distribution from the user and receives control information to distribute the image, it outputs the image being displayed on the monitor 300 to the communication server 400. At this time, the image from the synthesis unit 308 is output to the monitor 300 or the communication server 400, but this is not limited to this. For example, if the layout information associated with the specified preset number is in synthesis mode (first display mode), the image obtained from the synthesis unit 308 is output to the monitor 300. Similarly, if the layout information associated with the specified preset number is not in synthesis mode (second display mode), either the image obtained by the image acquisition unit 301 or the image registered in the image registration unit 302 is obtained. This image may then be output to the monitor 300.

[0060] The user interface 310 receives control instructions from the user for the image processing device 200, the imaging device 100a, or the imaging device 100b, and outputs them as control information to the control unit 305.

[0061] The calculation unit 311 reads the images acquired by the image acquisition unit 301 and the images registered in the image registration unit 302 from the storage unit 304 and calculates at least one of the similarity or complexity as a score (evaluation value). The specific method for calculating the score will be described later. The calculated score is output to the control unit 305.

[0062] <Preset registration process> In this embodiment, the preset registration process in the Preset Registration Unit 303 will be explained with reference to Figure 4. The Preset Registration Unit 303 is capable of registering and updating settings related to preset functions. In other words, the user can execute control processes stored in association with a predetermined preset number by specifying the preset number via the User I / F 310. Specifically, the user registers control information in association with a preset number (preset registration). After preset registration, when the user specifies a preset number via the User I / F 310, the control unit 305 executes various controls (preset execution) based on the control information corresponding to the preset number registered in the Preset Registration Unit 303.

[0063] In this embodiment, the user can register settings related to the image to be output to the monitor 300, associated with a preset number.

[0064] The image processing device 200 of this embodiment can register preset settings by associating the preset number with the layout information, the image acquired by the image acquisition unit 301, and the image registered by the image registration unit 302.

[0065] In this embodiment, the image processing device 200 is connected to the imaging device 100a and the imaging device 100b in a controllable manner. The image acquisition unit 301 acquires the image 410 shown in Figure 4(a) from the imaging device 100a and the image 420 shown in Figure 4(b) from the imaging device 100b.

[0066] Similarly, the image registration unit 302 has the image 430 shown in Figure 5(c) registered in advance.

[0067] At this point, the user selects a preset number they wish to register, and then can specify whether to use the first display mode, which displays multiple images combined, or the second display mode, which displays only a single image, by inputting layout information.

[0068] For example, if a layout information for displaying multiple images on the same screen, which is the first display mode, is specified for preset number 1, the user specifies which images to combine and display on the same screen. In this case, we will explain by registering a preset for preset number 1 to display images 410 and 430 combined on the same screen, as shown in Figure 4(e).

[0069] In this case, the user may display two or more images on the same screen, and any number of images can be selected from multiple images acquired from the image acquisition unit 301 and multiple images registered in the image registration unit 302, as shown in Figure 4(g).

[0070] Furthermore, when the user inputs control information for saving an image to be displayed on the same screen and layout information associated with a preset number, information for creating the image shown in Figure 4(e) is registered in preset number 1. At this time, information regarding the correction process shown in Figure 4(d) performed by the image correction unit 307 may also be registered simultaneously, or it may be determined automatically. The correction process will be described later.

[0071] Next, we will explain the case where layout information for displaying a single image, which is the second display mode, is specified for preset number 2. When layout information for the second display mode is specified, the user specifies the image to be displayed on the screen. In this case, we will explain by registering a preset that displays image 420 as preset number 2.

[0072] At this time, the image selected by the user may be either an image acquired from the image acquisition unit 301 or an image registered in the image registration unit 302.

[0073] Similar to Preset 1, when the user enters control information to associate layout information and image information with a preset number and save it, the preset registration is executed.

[0074] In this way, by performing a preset registration in advance, when the user inputs a user operation to execute the desired preset number after registration, an image of the layout associated with that preset number is created. For example, when the user executes preset number 1, the image to be output is determined according to the layout information. If there are multiple images to be output, the control unit 305 decides whether or not to correct the output image using the image correction unit 307 based on the similarity or complexity score (evaluation value) of the images calculated by the calculation unit 311. In this case, if correction processing is performed, image 410 is corrected by the image correction unit 307 in a manner described later, and image 431 is created. Furthermore, image 440, which is a composite of image 431 and image 430, is created by the synthesis unit 308 and output to the output unit 309 (preset execution). If no correction is performed, image 450, which is a composite of image 410 and image 430, is created by the synthesis unit 308 and output to the output unit 309 (preset execution). The specific processing method will be described later.

[0075] In this embodiment, a method for registering layout information as a preset has been described, but preset registration is not limited to this. For example, if a PTZ camera is connected as an imaging device, the positions of the pan drive unit, tilt drive unit, and lens drive unit of the PTZ camera can be associated with a preset number and registered as a preset position (setting). When the user inputs a predetermined preset number from the user I / F 310 with preset linkage turned on, the control unit 305 executes the control corresponding to that preset number. Here, "preset" refers to a function for recalling settings and operation history stored in the image processing device. For example, if a control instruction for the imaging device is associated with a preset number, a preset recall instruction will be output to the imaging device. When the imaging device receives a control command instructing the registration of a preset including a preset number, it records the current position and setting information on the terminal side. After registration, when the image processing device 200 transmits control information including the preset number to the imaging device, the imaging device recalls the stored position and setting and reflects the movement and settings to the preset position. Similarly, if the control information including the preset number does not include the imaging device, the control unit 305 can execute control in accordance with the control information based on the control history stored in the storage unit 304.

[0076] By registering presets in this way, it becomes possible to create images for distribution based on information previously set by the user when the preset execution command is issued.

[0077] <How the score (evaluation value) is calculated> In this embodiment, the calculation unit 311 evaluates the correction region 411 in the image 410 acquired from the image acquisition unit 301 according to a preset calculation method and calculates it as a score (evaluation value). Based on this score, the control unit 305 decides to execute the correction process for the correction region 411. The following are some examples of methods for calculating the score (evaluation value).

[0078] <Explanation and effects of similarity determination> In this embodiment, the calculation unit 311 evaluates the similarity between the correction region 411 of the image 410 acquired from the image acquisition unit 301 and the image 420 (reference image) acquired from the image registration unit 302. When the similarity is evaluated as high, the calculated score will be high. The calculated score is then output to the control unit 305. The control unit 305 controls the correction to be applied to the correction region 411 of the image acquired by the image acquisition unit 301 when the score acquired from the calculation unit 311 exceeds a preset threshold or a threshold registered by the user in the preset registration unit 303.

[0079] One method for evaluating similarity is to compare the histograms of the correction region 411 and image 420 (source image). In other words, this method involves representing the color distribution of the images as histograms and comparing them.

[0080] Alternatively, a method may be used to evaluate the similarity by comparing the pixel values ​​of the correction region 411 and the image 420 (reference image).

[0081] <Explanation and effects of complexity assessment> In this embodiment, the calculation unit 311 may evaluate the complexity of the correction region 411 and output it as a score to the control unit 305. The control unit 305 controls the system to perform correction when the score obtained from the calculation unit 311 exceeds a preset threshold or a threshold registered by the user in the preset registration unit 303.

[0082] To determine complexity, one method is to evaluate the number and complexity of edges in the correction region 411. Alternatively, one can analyze the texture features of the correction region 411 to evaluate its complexity.

[0083] If the calculated complexity score exceeds the threshold, the correction is applied to correction area 411.

[0084] <Explanation and effect of determining the presence or absence of characters> Furthermore, for images in correction region 411, a higher score may be calculated if text is included.

[0085] At this time, the calculation unit 311 calculates a weighted score indicating the likelihood that characters are likely to be included in the image of the correction region 411, and outputs it to the control unit 305. The control unit 305 controls the correction region 411 to perform correction if the acquired score exceeds a preset threshold or a threshold registered by the user in the preset registration unit 303.

[0086] When the calculation unit 311 calculates the score, edge detection is performed in the correction region 411. The control unit 305 may decide whether or not to perform correction based on whether the number of edges exceeds a preset threshold or a threshold registered by the user in the preset registration unit 303. Furthermore, for the image in the correction region 411, the image may be divided into an arbitrary number of blocks, and the result of calculating the variance value of the brightness of each block may be used as a score for evaluation. Alternatively, the evaluation may be based on the number of blocks that exceed the threshold value of the variance. The control unit 305 may make a decision based on whether the score obtained from the calculation unit 311 exceeds a preset threshold or a threshold registered by the user in the preset registration unit 303. In addition, if the variance value of a block is high, it is determined that there is a high possibility that characters are contained within the block.

[0087] <Explanation and effects of text content detection> The calculation unit 311 may weight the score output to the control unit 305 based on whether the correction area 411 contains characters and whether the same string exists in both the correction area 411 and the image 420 (source image). In other words, it determines the similarity between the characters detected in the correction area 411 and the characters detected in the image 420, and outputs the score to the control unit 305. At this time, image processing such as noise reduction and gamma correction is performed as preprocessing, and then the text area is identified. In this case, edge detection or template matching is used. The identification of the text area may be performed by the area determination unit 306. Furthermore, character segmentation may be performed to divide the characters within the text area into individual characters, which may be used as material for determination. At this time, the segmented characters are recognized and extracted as text. If the similarity score of the text string output by the calculation unit 311 exceeds a set threshold or a threshold registered by the user in the preset registration unit 303, correction is performed on the correction area 411.

[0088] <Preset registration of correction status and content> If a user has previously determined that at least a portion of the image 410 (camera image) requires visual correction, they may register the intention to perform the correction and the details of the correction in the Preset registration unit 303 and then execute the correction.

[0089] By applying this correction to the correction region 411, the overlapping information with image 420 in the correction region 411 and the complexity of the correction region 411 are reduced, thereby improving the visibility of image 420.

[0090] <Explanation and effects of the correction> The correction shown in this embodiment refers to a processing operation performed on a correction area other than the image registered in the image registration unit 302 or the non-correction area, which is specified by the user and contains, for example, information of high importance.

[0091] By performing this correction, even when using images that combine multiple images, the correction process can draw the viewer's attention to the image they are interested in. Examples of processing include blurring, brightness adjustment, overlaying other images, grayscale conversion (grayscale processing), saturation adjustment, and brightness adjustment.

[0092] This correction is achieved by having the area determination unit 306 determine the correction area from the image acquired by the image acquisition unit 301. The correction process is performed on the area determined to be the correction area by the area determination unit 306. For example, let's consider the case where the background area is determined to be the correction area for image 410 in Figure 4(a). The correction area determination unit determines the correction area 411 and the non-correction area 412 for image 410. Furthermore, the image correction unit 307 performs the correction process on the correction area 411 to create image 431. By performing this correction process, it becomes possible to create a corrected image (corrected image) like the image 431 shown in Figure 4(d). The type of correction may be set by the user, or a pre-set process may be executed.

[0093] <Blur effect> One example of this correction process is blurring, which smooths out a portion of an image. This process uses an averaging filter, a weighted averaging filter, and a Gaussian filter. These filters apply a weight to each pixel of the image to create a blurring effect. By applying blurring in this way, even when using images that combine multiple images, the corrected area will appear visually smoother in terms of contrast than the uncorrected area, making it easier for viewers to focus on the uncorrected area.

[0094] <Explanation of Brightness Adjustment> One example of this correction process is brightness adjustment, an image processing technique that changes the brightness of an image. In the case of RGB data, brightness is adjusted by adding or subtracting a fixed value to the value of each pixel. In the case of YCC data, brightness is adjusted by adding or subtracting an arbitrary value to the Y value (brightness value) of each pixel. In addition, the aforementioned Gaussian filter may be used to equalize the brightness. By performing brightness adjustment in this way, even when using images that combine multiple images, the corrected area will appear visually darker or brighter than the uncorrected image. Therefore, it is possible to reduce the visibility of information contained in the corrected area, making it easier for viewers to focus on the uncorrected area.

[0095] <Explanation of Grayscale Conversion> (Corresponding to Claim 5) One example of this correction process is grayscale conversion (grayscale processing), a technique that represents a color image using only shades of black and white, employing the luminance method, average method, and maximum value method. By performing grayscale conversion in this way, even when using images that combine multiple images, the corrected area is converted to a grayscale image. Therefore, the amount of color information in the corrected area is reduced compared to the uncorrected area, making it possible to create an image where the uncorrected area is easier to focus on.

[0096] <Explanation of saturation adjustment> Saturation adjustment can be done by adding or subtracting the color components of each pixel equally, or by adding or subtracting the values ​​of the color difference components. By adjusting the saturation, even when using images that combine multiple images, the corrected areas will appear visually duller, making it possible to create captured images where the uncorrected areas are easier to focus on.

[0097] <Explanation of brightness adjustment> Brightness adjustment affects the brightness of an image. Brightness is calculated from the average of the maximum and minimum color components of a pixel. By adjusting the calculated brightness values, the correction area will appear visually darker or brighter than the uncorrected image, even when using images that combine multiple images. As a result, it is possible to reduce the visibility of information contained in the correction area, making it possible to create captured images where the uncorrected area is easier to focus on.

[0098] <Explanation of filling in> The fill function fills the correction area with a specified color. By using the fill function, even when combining multiple images, the correction area is filled with the predetermined color, making it possible to create images where the uncorrected areas are easily noticeable. The color used can be anything and can be changed by the user.

[0099] <Explanation of image superposition> Even when using an image that combines multiple images, the image superposition will be performed using an image that is not registered in the image registration unit 302 and has not been acquired from the image acquisition unit 301 for the correction region. This process makes the correction region invisible when the image is superimposed, thus enabling the creation of an image that makes it easier to focus on the uncorrected region.

[0100] <Image correction processing> Next, the image correction method in this embodiment will be explained using Figures 4 and 5. Figure 4 is an illustrative diagram of the image correction process, and Figure 5 is a flowchart related to this control.

[0101] This flowchart is implemented when the OS, various programs, and various data are loaded into RAM220, which temporarily stores the computer program executed by CPU250, and then executed by CPU250. Furthermore, this flowchart starts by acquiring control information for preset execution and ends by acquiring control information from the user to stop the acquisition of captured images.

[0102] In step S501, based on the control information for executing the preset obtained from the user, the control unit 305 obtains the preset information registered in the Preset registration unit 303. After obtaining the information, the process proceeds to step S502. If the control information for executing the preset cannot be obtained, this process may be repeated until the control information is obtained.

[0103] In step S502, the control unit 305 obtains layout information related to the specified preset number from the Preset registration unit 303. If the layout information corresponds to the first display mode, the process proceeds to step S503; otherwise, the process proceeds to step S506.

[0104] In step S503, the control unit 305 retrieves the image acquired by the image acquisition unit 301 and the image registered in the image registration unit 302 from the storage unit 304 based on the preset number information. At this time, it is explained that there are images registered in the image registration unit 302, but if there are no registered images, the process may proceed to step S501. The user may also be notified. The notification method at this time may be controlled by the control unit 250 to display on the monitor 400, or controlled by the control unit 250 to notify by voice, and is not limited to these.

[0105] In step S504, the region determination unit 306 determines the corrected region 411 and the uncorrected region 412 of the image 410 acquired from the image acquisition unit 301 in step S501. The method for determining each region can be any of the following: segmentation method, edge detection, background subtraction, deep learning, clustering, graph-based method, optical flow method, or use of depth information. Furthermore, for the segmentation method, color space division method, region growth method, superpixel method, edge detection, or image depth information may be used. After the determination, the process proceeds to step S505.

[0106] In step S505, the calculation unit 311 calculates a score based on the correction region 411 determined in step S504. Furthermore, the calculation unit 311 outputs the score to the control unit 305 and proceeds to step S506. Methods for calculating the score include calculating it from the similarity between the correction region 411 and the image 420 registered in the image registration unit 302, or calculating the score from the complexity of the correction region 411.

[0107] In step S506, the control unit 305 determines whether the score calculated in step S505 is equal to or greater than a preset threshold. If the score is equal to or greater than the threshold, the process proceeds to step S507. If the score is less than the threshold, the process proceeds to step S508.

[0108] In step S507, the image correction unit 307 performs the correction specified in advance by the user on the correction area 411 of the image 410 determined in step S506, and creates image 431. After correcting image 410, the process proceeds to step S508. Furthermore, for the uncorrected area 412, correction is not required as shown in the example; instead, the contours may be emphasized to make the person's features clearer, or the sharpness settings may be changed.

[0109] In step S508, the synthesis unit 308 creates an image based on the layout information set in the Preset registration unit 303. For example, if the first display mode is set, the corrected image 431 (corrected image) created in step S505 and the image 520 registered in the image registration unit 302 are retrieved from the storage unit 304, and a composite image is created in step S406. Furthermore, the composite image 530 is output to the communication server 400 via the network 500, and the process proceeds to step S509.

[0110] In step S509, it is determined whether a termination command to stop acquiring captured images has been received from the user via the user interface 310. If a termination command has been received, this flow is terminated; otherwise, the process proceeds to step S501.

[0111] By performing this correction process when controlling the display of important materials and captured images on the same screen, it is possible to deliver the desired images while reducing the burden on the user during distribution.

[0112] <Description of the image editing UI for distribution using user-registered presets> Next, the UI of the image processing apparatus in the first embodiment of the present invention will be described with reference to Figure 6.

[0113] Figure 6(a) shows an example of the GUI displayed on monitor 300, specifically an example of the GUI screen when PresetA is selected. Here, PresetA is the name that the user has assigned to preset number 1.

[0114] The image area 600 is an area for displaying information obtained by the image registration unit 302 and the image acquisition unit 301.

[0115] The output image area 610 is the area that displays the image output to the output unit 309 after executing preset number 1 based on the information in the preset registration unit 303 and executing the flow shown in Figure 5. In the output image area 610, you can check the image output after a synthesis process has been performed in which the image acquired from the image acquisition unit 301 based on the information registered in Preset A (preset number 1) is corrected and the image registered in the image registration unit 302 is combined.

[0116] The control panel area 620 is a button that displays the name of the preset name 621 selected by the user based on the information from the preset registration unit 303. It is also a button that displays the name of the acquired image within an area divided based on the placement information of the image acquired from the preset registration unit, that is, the image acquired from the image registration unit 302 and the image acquisition unit 301. In Figure 6, the area that displays the image 612 acquired by the image registration unit 302 is designated as Area A, and the area that displays the image 531, which is the image acquired by the image registration unit 303 and corrected, is designated as Area B.

[0117] When a user selects button 622 or 623 in each area, they can edit the layout to any desired configuration from the layout information registered in the Preset Registration section.

[0118] 601 is an image acquired by the image acquisition unit 301, and in this embodiment, it is an image captured by the imaging device 100b.

[0119] 602 displays the name of the imaging device 100b that was registered in the Preset registration unit 303 in association with the image acquired by the image acquisition unit 301. 603 is an image acquired by the image acquisition unit 301, and in this embodiment, it is an image captured by the imaging device 100a.

[0120] 604 displays the name of the imaging device 100a that was registered in the Preset registration unit 303 in association with the image acquired by the image acquisition unit 301. 605 indicates an image that has been previously registered in the image registration unit 302.

[0121] 606 displays the file name of an image previously registered in the image registration unit 302.

[0122] 611 displays an output image that is a composite of image 605 from the image registration unit 302 and image 603 acquired from the image acquisition unit 301, and image 613 which has been corrected by the image correction unit 307.

[0123] The Previous button 624 is used to execute the preset that was registered immediately before PresetA when multiple presets are registered. In this case, the user may set the order of the presets.

[0124] The NEXT button 625 is used to execute the preset that is registered immediately after PresetA when multiple presets are registered.

[0125] Figure 6(b) shows the GUI after the Previous button 624 is pressed, transitioning from PresetA (preset number 1) to PresetB (preset number 3) from the state shown in Figure 6(a). Preset B (preset number 3) shows an example where a third area is added, and one GUI button 626 is added for selecting the image to be captured. In this way, it is possible to control the display of images from multiple imaging devices and registered images on the same screen, and it is also possible to change the imaging device being used by pressing buttons 623 or 626 during execution.

[0126] Figure 6(c) shows an example of the GUI screen after the user selects the NEXT button 625 in Figure 6(a) and transitions from presetA to presetC (preset number 2).

[0127] In this way, users can easily switch between images being delivered by specifying a preset number or by pressing the NEXT button 625 or Previous button 624. Furthermore, when controlling the display of multiple images on the same screen (first display mode), the correction area is automatically determined, and the corrected image (corrected image) and the registered image are displayed on the same screen without the user having to perform the correction process.

[0128] <Second Embodiment> Next, the image processing control according to the second embodiment will be described using Figures 7, 8, and 9. The second embodiment is an embodiment in which the image processing described in the first embodiment is applied to a video conferencing system. Explanations of redundant content will be omitted.

[0129] Figure 7 is an illustrative diagram showing an example of the system configuration in this embodiment, and Figure 8 is a flowchart of the image correction process. Figure 9 is an illustrative diagram of the output image.

[0130] The hardware configuration and functional block diagram of the image processing device 200 are the same as in Embodiment 1, so their explanation will be omitted.

[0131] Figure 7 shows that imaging devices 100a to 100c, which capture images of each participant, and external devices 600a to 600c, which are information processing devices that allow each participant to view the distributed images, are connected to each imaging device 100 so that they can communicate with each other. In this embodiment, external devices 600a to 600c are general client terminals such as PCs and tablets, which have a display unit and an operation unit. In this embodiment, a PC with an integrated monitor and input device (user interface) is shown as an example, but separate devices may also be used.

[0132] For example, an input terminal such as a joystick may be connected to an external device 600 via USB or Bluetooth® and used as an input for remotely controlling the imaging device 100.

[0133] In this embodiment, participants output images 810a to 810c (first images) captured by imaging device 100c from each imaging device 100a to the image processing device 200 via the network 500. Participants can also output images other than captured images (second images), such as materials, to the image processing device 200 via the network 500. In this embodiment, it is assumed that participants output captured images, but this is not limited to this. For example, the output of captured images can be controlled, and it is possible for a participant who does not output captured images to output only second images, or to acquire only images distributed by the image processing device 200.

[0134] In this embodiment, the image registration unit 302 can acquire images other than the captured image 100, such as images displayed on the screen of the external device 600 or documents, from the participant. After acquisition, it outputs to the storage unit 304.

[0135] Figure 8(a) shows an image 810a (first image) acquired from the imaging device 100a that captures each participant, and is acquired by the image acquisition unit 301.

[0136] Figure 8(b) shows an example of an image 820 (second image), which is different from the captured image and is used as explanatory material shared by presenters among the participants. This image is acquired by the image registration unit 302.

[0137] In step S901, the image acquisition unit 301 determines whether it has acquired an image from the imaging device 100. If it has acquired one or more images from images 810a to 810c acquired by imaging device 100c from each imaging device 100a, it proceeds to step S902. If it has not acquired any images, it proceeds to step S907.

[0138] In step S902, it is determined whether there are any images registered in the image registration unit 302. If there are any images registered in the image registration unit 302, the process proceeds to step S903; otherwise, it proceeds to step S909. In other words, the process proceeds to step S903 if at least one participant outputs an image different from the captured image to the image processing device 200 via the network 500. Specifically, when a participant set as a presenter outputs a specified image to the image processing device 200 via the network 500 in order to control the distribution of materials on their PC or the screen of an external device 600 to other participants in real time, the image registration unit 302 acquires it. At this time, it is determined in step S902 that there are images registered in the image registration unit 302, and the process proceeds to step S903. If there are no registered images, the process proceeds to step S910.

[0139] In step S903, the control unit 305 determines whether the participant's display settings are set to a mode that controls the display of each image 810a to 811a acquired by the image acquisition unit 301 and the image 820 acquired by the image registration unit 302 on the same screen (first display mode). The display settings (display mode) can be changed by operating a setting UI (not shown) from the user I / F 310. If the mode that controls the display on the same screen (first display mode) is set, the process transitions to S904; otherwise, the process transitions to S910. In this embodiment, this process is described as being performed based on the participant's display settings, but it is not limited to this. For example, if there are images acquired from both the image acquisition unit 301 and the image registration unit 302, it may be automatically determined to be the first display mode. In this case, if either the image acquisition unit 301 or the image registration unit 302 has acquired an image, it will be determined to be the second display mode.

[0140] In step S904, the region determination unit 306 determines that the images 810a to 811a acquired by the image acquisition unit 301 are divided into a corrected region 811a and an uncorrected region 812a. In Figure 8(a), the region determination unit 306 determines that the corrected region (shaded area) 811a and the uncorrected region 812a are in image 810a. After the determination, the process proceeds to step S905.

[0141] In step S905, the calculation unit 311 calculates a score based on the correction region 411 determined in step S904. The specific calculation method is the same as in step S505, so the explanation is omitted. After the score is calculated, the process proceeds to step S906.

[0142] In step S906, the control unit 305 determines whether the score calculated in step S505 is equal to or greater than a preset threshold. If the score is equal to or greater than the threshold, the process proceeds to step S907. If the score is less than the threshold, the process proceeds to step S908.

[0143] In step S907, the image correction unit 308 performs a pre-specified correction on the correction region 811a determined in step S906. For example, it creates an image 531 (corrected image) by applying a correction process to image 510 (image 810) and outputs it to the storage unit 304. After the correction process on the correction region, the process proceeds to step S908.

[0144] In step S908, the acquired image is output to the external device 600 via the network 500. That is, if in step S905 the score is determined to be less than the threshold, then in step S908 the image 810 and explanatory material 820 are output to the external device 600. Similarly, if in step S905 the score is determined to be greater than or equal to the threshold, then the image 431 (corrected image) and the explanatory material 820 acquired by the image registration unit 302 are output to the external device 600 via the network 500. At this time, a composite image 830 as shown in Figure 8(c) may be created by the compositing unit 308, or the images may be output from the external device 600a to the external device 600c without compositing each image. After output, the process proceeds to step S911.

[0145] In step S909, it is determined whether there are any images registered in the image registration unit 302. The specific method is the same as in step S902, so the explanation is omitted. If there are any images registered in (or acquired by) the image registration unit 302, the process proceeds to step S910; otherwise, the process proceeds to step S901.

[0146] In step S910, the image acquired by the image acquisition unit 301 is not corrected, and either the image acquired by the image acquisition unit 301 or the image acquired by the image registration unit 302 is output from the external device 600a to the external device 600c for display as is. If explanatory materials are registered in the image registration unit 302, an image of the explanatory materials alone is output as shown in Figure 8(d). If the explanatory materials are not shared (the answer is No in step S902), an image 850 of a single participant (e.g., speaker) captured by the imaging device is output as shown in Figure 8(e). Alternatively, if images are acquired from multiple imaging devices, images 860 of multiple participants are output to the external device 600 so that they are displayed as shown in Figure 8(f). After output, the process proceeds to step S911.

[0147] In step S911, it is determined whether a termination command to stop acquiring captured images has been received from the user via the user interface 310. If a termination command has been received, this flow is terminated; otherwise, the process proceeds to step S901.

[0148] In this embodiment, the uncorrected region 812 does not need to be corrected as shown in the figure example, or the outline may be emphasized or the sharpness setting changed so that the person's appearance is easily conveyed to the broadcaster.

[0149] By performing this correction process when controlling the display of important materials and captured images on the same screen, it is possible to deliver the desired images while reducing the burden on the user during distribution.

[0150] In this embodiment, the presenter distributes the image from the imaging device and the explanatory materials as separate images, and the case where correction is applied or not is switched according to each user's display mode setting has been described. However, this is not the only case; even if the distribution side pre-combines the image from the imaging device and the explanatory materials into a single image and distributes it, the audience can split and judge the images and perform similar correction processing on the image from the imaging device.

[0151] In summary, by performing this correction, even when using images that combine multiple images, the correction process can draw the participant's attention to the image they are interested in.

[0152] The disclosures herein include the following image processing devices, conferencing systems, image processing methods, and programs.

[0153] (Item 1) The first image captured by the imaging means, Image acquisition means for acquiring a second image different from the first image, A control means for controlling the display means to display a predetermined image, A calculation means for calculating an evaluation value for the first image, A determination means for determining whether or not to correct at least a portion of the first image based on the evaluation value, It has, The aforementioned image creation means When the first image and the second image have been obtained, If the determination means determines that at least a portion of the first image should be corrected, the control means controls the display means to display the corrected image, which is the corrected version of the first image, and the second image. An image processing apparatus characterized in that, when either the first image or the second image has been acquired, the control means controls the display means to display the acquired image.

[0154] (Item 2) The control means A first display mode controls the display means to display the corrected image obtained by correcting the first image and the second image on the same screen, The image processing apparatus according to item 1, characterized in that it has a second display mode for controlling the display means to display either the first image or the second image.

[0155] (Item 3) The image processing apparatus according to item 2, characterized in that the second display mode does not correct the first image.

[0156] (Item 4) A region determination means for determining the correction region in the first image, The determination means includes a correction means for correcting the region determined to be the correction region, The image processing apparatus described in item 1, characterized by having the following features.

[0157] (Item 5) The correction by the correction means is at least one of the following: blurring, brightness adjustment, grayscale processing, saturation adjustment, lightness adjustment, fill processing, and image superposition. The image processing apparatus according to item 1, characterized in that the image used for superimposing the aforementioned image is neither the first image nor the second image.

[0158] (Item 6) The image processing apparatus according to item 4, characterized in that the region determination means determines the background portion of the captured image as a correction region.

[0159] (Item 7) The image processing apparatus according to item 4, wherein the region determination means determines a correction region based on an object set by the user in the captured image.

[0160] (Item 8) The image processing apparatus according to item 4, characterized in that the calculation means calculates an evaluation value based on the correction region determined by the region determination means.

[0161] (Item 9) The image processing apparatus according to item 4, characterized in that the determination means determines that the correction region should be corrected if the evaluation value calculated by the calculation means is equal to or greater than a preset threshold.

[0162] (Item 10) The image processing apparatus according to item 4, characterized in that the calculation means calculates the evaluation value based on the similarity between the correction region and the second image.

[0163] (Item 11) The image processing apparatus according to item 4, characterized in that the calculation means calculates the evaluation value based on the characters in the correction region.

[0164] (Item 12) The image processing according to item 4, characterized in that the calculation means detects whether complex information is included in the correction region and calculates the evaluation value based on the detection result.

[0165] (Item 13) A conference system comprising one or more information processing devices and image processing devices having imaging means, The aforementioned information processing device is Output means for outputting a first image captured by the imaging means and a second image different from the first image to the image processing device, Control means for controlling the display of an image acquired from the image processing device, It has, The aforementioned image processing device is The first image obtained from one or more of the information processing devices, An acquisition means for acquiring a second image different from the first image, A calculation means for calculating an evaluation value for the first image, A determination means for determining whether or not to correct at least a portion of the first image based on the evaluation value, It has, If the information processing device has acquired at least one or more first images and a second image, and the determination means determines that at least a part of the first image should be corrected, the control means outputs the corrected image, in which at least a part of the first image has been corrected, and the second image to the information processing device. A conference system characterized by outputting an acquired image to an information processing device if at least one of the first image or the second image has been acquired from the information processing device.

[0166] (Item 14) The first image captured by the imaging device, An image acquisition step to acquire a second image different from the first image, A control step that controls the display device to display a predetermined image, A calculation step for calculating an evaluation value for the first image, A determination step of determining whether or not to correct at least a portion of the first image based on the evaluation value, It has, In the image acquisition step, if the first image and the second image are acquired, and in the determination step, it is determined that at least a part of the first image should be corrected, In the control process described above The system is controlled to display the corrected image, which is obtained by correcting at least a portion of the first image, and the second image on the display device. In the image acquisition process described above, An image processing method that, if either the first image or the second image has been acquired, controls the device to display the acquired image on the display device during the control step.

[0167] (Item 15) A program to cause a computer to function as an image processing device as described in one of items 1 through 12.

[0168] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0169] 100 Imaging device 200 Image Processing Devices 303 Preset Registration Section 410 Images acquired from the imaging device Images registered in the 420 Preset registration section 431 Corrected image

Claims

1. The first image captured by the imaging means, Image acquisition means for acquiring a second image different from the first image, A control means for controlling the display means to display a predetermined image, A calculation means for calculating an evaluation value for the first image, A determination means for determining whether or not to correct at least a portion of the first image based on the evaluation value, It has, The aforementioned image creation means When the first image and the second image have been acquired, If the determination means determines that at least a portion of the first image should be corrected, the control means controls the display means to display the corrected image, which is the corrected version of the first image, and the second image. An image processing apparatus characterized in that, when either the first image or the second image has been acquired, the control means controls the display means to display the acquired image.

2. The control means A first display mode controls the display means to display the corrected image obtained by correcting the first image and the second image on the same screen, The image processing apparatus according to claim 1, characterized in that it has a second display mode for controlling the display means to display either the first image or the second image.

3. The image processing apparatus according to claim 2, characterized in that the second display mode does not correct the first image.

4. A region determination means for determining the correction region in the first image, The determination means includes a correction means for correcting the region determined to be the correction region, The image processing apparatus according to claim 1, characterized by having

5. The correction by the correction means is at least one of the following: blurring, brightness adjustment, grayscale processing, saturation adjustment, lightness adjustment, fill processing, and image superposition. The image processing apparatus according to claim 1, characterized in that the image used for superimposing the aforementioned image is neither the first image nor the second image.

6. The image processing apparatus according to claim 4, characterized in that the region determination means determines the background portion of the captured image as a correction region.

7. The image processing apparatus according to claim 4, characterized in that the region determination means determines a correction region based on an object set by the user in the captured image.

8. The image processing apparatus according to claim 4, characterized in that the calculation means calculates an evaluation value based on the correction region determined by the region determination means.

9. The image processing apparatus according to claim 4, characterized in that the determination means determines that the correction region should be corrected if the evaluation value calculated by the calculation means is equal to or greater than a preset threshold.

10. The image processing apparatus according to claim 4, characterized in that the calculation means calculates the evaluation value based on the similarity between the correction region and the second image.

11. The image processing apparatus according to claim 4, characterized in that the calculation means calculates the evaluation value based on the characters in the correction region.

12. The image processing according to claim 4, characterized in that the calculation means detects whether complex information is included in the correction region and calculates the evaluation value based on the detected result.

13. A conference system comprising one or more information processing devices and image processing devices having imaging means, The aforementioned information processing device is Output means for outputting a first image captured by the imaging means and a second image different from the first image to the image processing device, Control means for controlling the display of an image acquired from the image processing device, It has, The aforementioned image processing device is The first image obtained from one or more of the information processing devices, An acquisition means for acquiring a second image different from the first image, A calculation means for calculating an evaluation value for the first image, A determination means for determining whether or not to correct at least a portion of the first image based on the evaluation value, It has, When the information processing device has acquired at least one of the first images and the second image, and the determination means determines that at least a part of the first image should be corrected, the control means outputs the corrected image, in which at least a part of the first image has been corrected, and the second image to the information processing device. A conference system characterized in that, if either the first image or the second image is obtained from at least one of the information processing devices, the acquired image is output to the information processing device.

14. The first image captured by the imaging device, An image acquisition step to acquire a second image different from the first image, A control step that controls the display device to display a predetermined image, A calculation step for calculating an evaluation value for the first image, A determination step of determining whether or not to correct at least a part of the first image based on the evaluation value, It has, In the image acquisition step, if the first image and the second image are acquired, and in the determination step, it is determined that at least a part of the first image should be corrected, In the control process described above The system is controlled to display the corrected image, which is obtained by correcting at least a portion of the first image, and the second image on the display device. In the image acquisition process described above, An image processing method that, if either the first image or the second image has been acquired, controls the device to display the acquired image on the display device in the control step.

15. A program for causing a computer to function as an image processing device according to any one of claims 1 to 12.

Images