Information processing apparatus and imaging apparatus

The information processing device addresses image quality unevenness in virtual production by determining subject and background areas and applying brightness corrections based on the angle or distance between the imaging device and the background display, ensuring consistent image quality across varied capture angles and distances.

JP2024168917APending Publication Date: 2024-12-05CANON KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023085973
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing imaging technologies fail to maintain consistent image quality when capturing images from different angles with respect to LED displays used in virtual production, leading to uneven brightness in background video regions.

Method used

An information processing device that acquires input images, determines subject and background areas, and performs image processing based on the angle or distance between the imaging device and the background display, applying brightness corrections to reduce unevenness.

Benefits of technology

Ensures reduced image quality unevenness in background video regions regardless of the arrangement of the background display and imaging device, even when multiple devices capture images from different angles or distances.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168917000001_ABST
    Figure 2024168917000001_ABST
Patent Text Reader

Abstract

To obtain image data in which image quality unevenness in a background video area is reduced regardless of the arrangement of a background display and an imaging apparatus.SOLUTION: An information processing apparatus 105 acquires an input image obtained by an imaging apparatus picking up images of an object OBJ and background videos displayed on a background display LD, discriminates an object area and a background video area in the input image from each other, acquires arrangement information on at least one of the angle and distance between the background display and the imaging apparatus, and performs image processing according to the arrangement information on the background video area.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device for processing information relating to imaging. [Background technology]

[0002] One imaging technique used in movies and television is virtual production, in which a background image is displayed on a background display that is larger than the subject, and then the subject is imaged together with the background image. However, LEDs, which are the main display elements used in background displays, have viewing angle characteristics. For this reason, if the imaging device is not positioned so as to directly face the background display when imaging, image quality unevenness (brightness unevenness) will occur in the background image area in the image data obtained by imaging.

[0003] Patent Document 1 discloses a method of calculating an angle between an LED display and an imaging device from position information of the imaging device, and correcting the brightness of the LED display using a brightness correction value according to the angle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Publication No. 2022 / 0014728 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method of Patent Document 1, the brightness of the LED display itself is changed according to the angle between it and one imaging device. Therefore, image data in which image quality unevenness in the background image area is reduced is obtained when the image is captured by the one imaging device. However, when an imaging device other than this imaging device captures an image from a different angle at the same time, there is a risk that image data in which image quality unevenness in the background image area is not reduced is obtained.

[0006] The present invention provides an information processing device and an imaging device that are capable of obtaining image data with reduced image quality unevenness in the background image area, regardless of the arrangement of the background display and the imaging device. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present invention is characterized by having an acquisition means for acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device, a discrimination means for discriminating between a subject area and a background image area in the input image, and a processing means for acquiring arrangement information regarding at least one of the angle and the distance between the background display and the imaging device, and performing image processing on the background image area according to the arrangement information. Note that an imaging device having the above-mentioned information processing device also constitutes another aspect of the present invention.

[0008] An information processing method according to another aspect of the present invention includes the steps of acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display using an imaging device, determining an object area and a background image area in the input image, acquiring layout information relating to at least one of an angle and a distance between the background display and the imaging device, and performing image processing on the background image area according to the layout information. Note that a program for causing a computer to execute processing according to the information processing method also constitutes another aspect of the present invention. Effect of the Invention

[0009] According to the present invention, an image with reduced image quality unevenness in the background image area can be obtained regardless of the arrangement of the background display and the imaging device. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a part of a light receiving surface of an image sensor. [Diagram 3]11 is a flowchart showing a process for calculating distance information in the embodiment. [Figure 4] FIG. [Diagram 5] FIG. [Figure 6] 4 is a flowchart showing a process in the first embodiment. [Figure 7] 4A to 4C are diagrams showing an image capturing state in the first embodiment and a diagram showing a subject area and an LED display area in the image data. [Figure 8] FIG. 4 is a diagram showing division of image data in the first embodiment. [Figure 9] 4A and 4B are diagrams showing a correction target block and peripheral blocks in the first embodiment. [Figure 10] 5 is a schematic diagram showing an imaging position and two adjacent blocks on adjacent LED displays in image data in the first embodiment. FIG. [Figure 11] 11 is a flowchart showing a process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, an image capturing device (digital camera) equipped with an information processing device that executes the processes of specific embodiments 1 and 2 described below will be described. Fig. 1 shows the configuration of an image capturing device 100. The image capturing device 100 captures images, processes the images, and records the images.

[0012] Connected to the internal bus 101 are a CPU 102, a ROM 103, a RAM 104, an image processing unit 105, a lens unit 106, an imaging unit 107, a network module 108, a video terminal 109, a recording medium I / F 110, a display unit 114, and an object detection unit 115. The units connected to the internal bus 101 can exchange data with each other via the internal bus 101.

[0013] The lens unit 106 has an optical system including a zoom lens, a focus lens, and an aperture mechanism, and a motor for driving the optical system. An optical image formed by the lens unit 106 is converted into an image signal as an analog electrical signal (i.e., imaged) by an image sensor such as a CCD sensor or a CMOS sensor included in the imaging unit 107. The imaging unit 107 also includes an A / D conversion unit that converts the image signal output from the image sensor into a digital signal (image data). The image sensor is an image plane phase difference sensor, and its details will be described later.

[0014] A CPU 102 as a computer controls each unit of the imaging device 100 according to a program stored in a ROM 103 and uses a RAM 104 as a work memory.

[0015] The ROM 103 is a non-volatile memory, and stores programs for operating the CPU 102, various parameters, and the like. The RAM 104 is a volatile memory. The frame memory 111 is a memory for temporarily storing image data, and the image data can be read out when necessary. By using the frame memory 111, it is possible to perform processing such as combining images obtained by imaging at different times and cutting out a specific area from an image obtained by imaging.

[0016] The image processing unit 105 as an information processing device performs various image processing on image data from the imaging unit 107 or image data stored in the frame memory 111 or the recording medium 112 under the control of the CPU 102. The image processing performed by the image processing unit 105 includes pixel interpolation, encoding, compression, decoding, enlargement / reduction (resizing), noise reduction, color conversion, and the like of the image data. The image processing unit 105 also corrects performance variations in a plurality of pixels of the imaging element, corrects defective pixels, corrects white balance, corrects brightness, and corrects distortion and peripheral light loss caused by lens characteristics. Furthermore, the image processing unit 105 generates distance information, which will be described later, and corrects the brightness value of the image data according to the angle and distance between the background display and the imaging device 100, which will be described later. The image processing unit 105 may be configured as a dedicated image processing computer for performing image processing, or the CPU 102 may have the function of the image processing unit 105.

[0017] The CPU 102 performs optical zoom, focusing, and aperture adjustment by controlling each motor of the lens unit 106 based on the calculation results for lens control obtained from the image processing unit 105. The CPU 102 may also perform image blur correction by moving a lens that constitutes a part of the optical system in a direction perpendicular to the optical axis according to camera shake detected by an attitude detection unit 116, which will be described later.

[0018] The operation unit 113 accepts an operation by a user and outputs an instruction corresponding to the operation to the CPU 102 and the image processing unit 105. The operation unit 113 has operation members such as a switch, a dial, and a button. The operation members include a power switch, a mode switching dial, and a release button. When the user presses the release button halfway, the release button instructs the CPU 102 to start an image capture preparation operation such as autofocus (AF) and exposure control (AE). When the user presses the release button all the way, the release button causes the CPU 102 to start an image capture operation for acquiring image data for recording. The operation unit 113 may be a touch sensor provided on the display unit 114.

[0019] The display unit 114 as a display means displays images processed by the image processing unit 105, setting menus, etc. The display unit 114 has a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display. The display unit 114 may also be provided with the above-mentioned touch sensor.

[0020] The CPU 102 generates text information and icons to inform the user of the setting status of the imaging device 100, as well as images of a setting menu for making various settings of the imaging device 100, and displays these on the display unit 114 by superimposing them on the image processed by the image processing unit 105.

[0021] The video terminal 109 is a video interface terminal such as SDI (Serial Digital Interface), HDMI (High Definition Multimedia Interface) (registered trademark), DisplayPort (registered trademark), etc. The CPU 102 can output video (image data) to an external monitor or the like via the video terminal 109.

[0022] The network module 108 is an interface for inputting and outputting image data and audio signals. The CPU 102 communicates with external devices via the network module 108 over the Internet or the like, and can transmit and receive not only image data and audio signals, but also control signals such as commands. The network module 108 may be of a wireless communication type or a wired communication type.

[0023] The recording medium 112 is a hard disk drive (HDD) or a solid state drive (SSD) capable of recording image data and various setting data, and is detachably attached to the recording medium I / F 110.

[0024] The object detection unit 115 detects an object using artificial intelligence, such as deep learning using a neural network. In object detection using deep learning, the CPU 102 transmits a program, a network structure, weight parameters, and the like stored in the ROM 103 to the object detection unit 115. Examples of the network structure include SSD (Single Shot Multibox Detector) and YOLO (You Only Look Once). The object detection unit 115 performs a process of detecting an object from image data based on various parameters obtained from the CPU 102, and transmits the process result to the RAM 104.

[0025] The attitude detection unit 116 detects the attitude and shake of the imaging device 100 using an angular velocity sensor, an acceleration sensor, etc. This makes it possible to detect the tilt of the imaging device 100 and camera shake due to hand shake or the like.

[0026] FIG. 2A is a schematic diagram showing a part of an image sensor in which red (R), blue (B), and green (Gb, Gr) pixels are arranged in a Bayer array. As shown in FIG. 2B, each pixel has two photodiodes A and B as photoelectric conversion elements, and the two photodiodes A and B are provided with one microlens (not shown). With this configuration, the two photodiodes A and B receive light from two different regions of the exit pupil of the lens unit 106 and output A and B signals. The imaging unit 107 synthesizes the A and B signals output from each of a plurality of pixels on the image sensor, thereby generating a pair of image signals (A and B image signals) for phase difference detection. In addition, an imaging signal for obtaining image data is generated by adding the A and B image signals.

[0027] In the imaging unit 107, in addition to outputting the A and B signals from each pixel, it is also possible to output a value obtained by averaging the A and B signals from multiple pixels that are close to each other. By outputting the averaging value, it is possible to shorten the time required to read out signals from the imaging unit 107 and reduce the bandwidth of the internal bus 101.

[0028] The CPU 102 performs correlation calculations on the pair of image signals to calculate the phase difference between them, and calculates information on the defocus amount from the phase difference. The defocus amount indicates the degree of defocus with respect to the in-focus state of the lens unit 106. The CPU 102 determines that the in-focus state is reached if the defocus amount is within a predetermined range including 0, and determines that the state is front-focused or back-focused if the defocus amount is positive or negative, respectively. The CPU 102 performs image plane phase difference AF by controlling a motor that drives a focus lens in the lens unit 106 according to the positive or negative defocus amount. The CPU 102 also calculates the reliability of the calculated phase difference. The CPU 102 further calculates the distance to the subject from the parallax between the pair of image signals and the optical information of the lens unit 106 using the principle of triangulation.

[0029] Although Fig. 2B shows a case where two photodiodes are provided for each pixel, the number of photodiodes for each pixel may be three or more. Also, the image sensor may be provided with a plurality of pixels having different aperture positions, and the plurality of pixels may be treated as one pixel for focus detection. In this way, the image sensor may be configured to obtain a pair of image signals for phase difference detection.

[0030] Next, the information generating process performed by the image processing unit 105 will be described with reference to the flowcharts of Figures 3 to 5. The image processing unit 105 functions as an acquisition unit, a determination unit, and a processing unit.

[0031] The flowchart in Fig. 3 shows distance calculation processing that the image processing unit 105 executes according to a program in response to the start of imaging in the imaging unit 107. First, in step S301, the image processing unit 105 calculates (acquires) a B image signal by determining the difference between the A image signal and the imaging signal (A image signal + B image signal) output from the imaging unit 107. Note that in this step, the A image signal and the B image signal may each be acquired from the imaging unit 107. In this case, a signal obtained by adding the A image signal and the B image signal can be acquired as the imaging signal. Furthermore, in the case of a stereo camera having two lens units and two imaging elements, the A image signal may be acquired from one of the two imaging elements, and the B image signal may be acquired from the other.

[0032] Next, in step S302, the image processing unit 105 performs shading correction due to optical factors on the A and B image signals.

[0033] Next, in step S303, the image processing unit 105 performs filtering on each of the A and B image signals. The filtering is, for example, high-pass filtering using a finite impulse response (FIR) filter. Below, a case will be described in which the A and B image signals that have been subjected to high-pass filtering are used, but filtering may also be performed using a band-pass filter or low-pass filter with different filter coefficients.

[0034] Next, in step S304, the image processing unit 105 divides each of the A and B image signals that have been subjected to the filter processing in step S303 into minute blocks, and performs correlation calculation for each divided block. There are no particular restrictions on the size or shape of the minute blocks, and adjacent blocks may partially overlap each other.

[0035] The correlation calculation between the A and B image signals will be explained below. The signal sequence of the A image signal at a pixel of interest is assumed to be the A image signal sequence E(1) to E(m), and the signal sequence of the B image signal at a pixel of interest is assumed to be the B image signal sequence F(1) to F(m). While shifting the B image signal sequence F(1) to F(m) relatively with respect to the A image signal sequence E(1) to E(m), the correlation amount C(k) at the shift amount k between these paired image signal sequences is calculated using the following equation (1).

[0036]

number

[0037] In equation (1), the Σ calculation refers to an operation to calculate the sum for n. In the Σ calculation, the range of n and n+k is limited to the range from 1 to m. The amount of shift k is an integer value, and is the relative pixel shift amount in units of the detection pitch (pixel arrangement pitch) of the paired image signal strings. Figure 4 shows the calculation results of equation (1) when the correlation between the paired image signal strings is high in an ideal state without noise.

[0038] As shown in Fig. 4, at a shift amount where the correlation between paired image signal sequences is high (k = kj = 0), the correlation amount C(k) is minimum. The k at which the discrete correlation amount C(k) is minimum is denoted as kj. The pixel shift amount (phase difference) x that gives the minimum value C(x) for continuous correlation amounts is calculated by the three-point interpolation process shown in equations (2) to (4). The pixel shift amount x is a real value, and its unit is pixel.

[0039]

number

[0040]

number

[0041]

number

[0042] The SLOP in equation (4) represents the slope of the change in the correlation amount that is the smallest and local minimum and the correlation amount adjacent to it. C(kj) = C(0) = 1000 C(kj-1)=C(-1)=1700 C(kj+1)=C(1)=1830 In this example, kj=0. From equations (2) to (4), SLOP=830 x=-0.078pixel In the focused state, the pixel shift amount x between the paired image signal rows is ideally 0.00.

[0043] On the other hand, the calculation results when formula (1) is applied to a small block where noise is present are shown in Fig. 5. As shown in Fig. 5, the correlation between paired image signal sequences decreases due to the effect of randomly distributed noise. The minimum value of the correlation amount C(k) becomes larger than the minimum value shown in Fig. 4, and the correlation amount curve becomes generally flat (the absolute value of the difference between the maximum and minimum values ​​becomes smaller).

[0044] In FIG. 5, as a specific example, C(kj) = C(0) = 1300 C(kj-1)=C(-1)=1480 C(kj+1)=C(1)=1800 In this example, kj=0. From equations (2) to (4), SLOP=500 x=-0.32pixel It becomes.

[0045] Compared to the calculation result in a noise-free state shown in FIG. 4, the pixel shift amount x is farther from the ideal value.

[0046] When the correlation between the pair of image signal strings is low, the change in the correlation amount C(k) is small and the correlation amount curve is generally flat, so the SLOP value is small. Similarly, when the subject image has low contrast, the correlation between the pair of image signal strings is low and the correlation amount curve is flat. Based on this property, the reliability of the calculated pixel shift amount x can be determined by the SLOP value. That is, when the SLOP value is large, it can be determined that the correlation between the pair of image signal strings is high, and when the SLOP value is small, it can be determined that no significant correlation has been obtained between the pair of image signal strings. In this embodiment, since formula (1) is used for the correlation calculation, the correlation amount C(k) is minimum and minimal at the shift amount k where the correlation between the pair of image signal strings is highest. Alternatively, a correlation calculation method in which the correlation amount C(k) is maximum and maximal at the shift amount k where the correlation between the pair of image signal strings is highest may be used.

[0047] Next, in step S305, the image processing unit 105 calculates the reliability of the calculated pixel shift amount x. The reliability can be defined by C(kj), which indicates the degree of match between the pair of image signals calculated in step S304, and the value of SLOP.

[0048] Next, in step S306, the image processing unit 105 performs an interpolation process. Although the correlation calculation could be performed in step S304, there are cases where the pixel shift amount x of the pixel of interest cannot be adopted due to low reliability calculated in step S305. In such cases, it is necessary to obtain the pixel shift amount of the pixel of interest by interpolation using the pixel shift amounts calculated for the pixels surrounding the pixel of interest. As a method of interpolation, a median filter may be applied to the pixel shift amount calculated for the surrounding pixels, or a method of reducing the pixel shift amount data and then enlarging it again may be used. Also, color data may be extracted from the imaging signal, and the pixel shift amount may be interpolated using the color data.

[0049] Next, in step S307, the image processing unit 105 calculates the defocus amount by referring to the pixel shift amount x calculated in step S304 or acquired by interpolation in step S306. The defocus amount (DEF) can be calculated by the following formula (5).

[0050]

number

[0051] In equation (5), P is a conversion coefficient determined by the detection pitch and the distance between the projection centers of the two viewpoints in the pair of parallax images (image signal A and image signal B) (the distance between the center of gravity of the image sensor generating image signal A and the center of gravity of the image sensor generating image signal B). The unit of the conversion coefficient is mm / pixel.

[0052] Next, in step S308, the image processing unit 105 calculates the distance from the defocus amount calculated in step S307. When the distance to the subject is Da, the focal position of the lens unit 106 is Db, and the focal length is F, the following formula (6) approximately holds.

[0053]

number

[0054] Therefore, the distance to the subject, Da, is

[0055]

number

[0056] It becomes.

[0057] If Db when DEF=0 is Db0, then equation (7) becomes:

[0058]

number

[0059] The absolute distance Da' to the subject can be calculated. The relative distance Da-Da' is calculated from equations (7) and (8) as follows:

[0060]

number

[0061] It can be found by:

[0062] By the above processing shown in FIG. 3, the pixel shift amount, defocus amount, and distance to the subject can be obtained from the A and B image signals.

[0063] The pixel shift amount and defocus amount thus acquired change depending on the distance from the imaging device 100. Therefore, the pixel shift amount, the defocus amount, and the distance are all information related to the distance from the imaging device 100. In the following description, information related to the distance is referred to as distance information. Furthermore, the information related to the distance is not limited to the distance itself, but may be information that can be converted into or associated with the distance. EXAMPLES

[0064] Next, a first embodiment will be described. Fig. 7(a) shows a state where an image of a subject OBJ located in front of an LED display LD as a background display is captured by an imaging device 100. The LED display LD is configured by arranging a large number of LEDs as display elements in a two-dimensional direction, and displays a background image according to a video signal supplied from a video supply device (not shown). Image data obtained by capturing an image shows the subject OBJ against the background image.

[0065] In FIG. 7(a), the imaging device 100 captures an object OBJ and an LED display LS from a direction different from the direction (direction facing the object) of the normal N of the LED display LD, which is a plane. The background display is not limited to a plane, and may be a curved or spherical surface. When the angle between the imaging direction of the imaging device 100 and the display surface of the LED display LD is significantly different from 90°, the brightness of the light incident from the LED display LD to the imaging device 100 decreases due to the viewing angle characteristics of the LED display LD. As a result, image quality unevenness (brightness unevenness) occurs due to the above angle in the background image region of the image data obtained by the imaging device 100. For this reason, it is desirable to perform brightness value correction as image processing on the image data to reduce this image quality unevenness.

[0066] The flowchart in FIG. 6 shows processing that the image processing unit 105 in the imaging device 100 in the first embodiment executes according to a program.

[0067] First, in step S 601 , the image processing unit 105 acquires image data as an input image obtained by capturing an image of the subject OBJ and the LED display LD, and information on the distance between the subject OBJ and the LED display LD, and stores these in the frame memory 111 .

[0068] Next, in step S602, the image processing unit 105 reads out image data and distance information from the frame memory 111. The image processing unit 105 determines an object area in which an object in the image data is captured and an LED display area (background image area) in which the LED display LD is captured, based on the distance information, as shown in FIG. 7(b), and separates them. Specifically, a threshold value is set for the distance information, and an area in the image data having distance information equal to or greater than the threshold is determined to be an LED display area, and an area in the image data having distance information less than the threshold is determined to be an object area. The threshold value may be arbitrarily set by the user through the operation of the operation unit 113, or may be set by the CPU 102 according to the aperture value, focus position, etc. of the lens unit 106. The image processing unit 105 adds flags for distinguishing between the object area and the LED display area in the image data, and stores the image data in the frame memory 111.

[0069] Next, in step S603, the image processing unit 105 reads out the image data and the distance information from the frame memory 111. The image processing unit 105 divides the image data into a plurality of x blocks in the horizontal direction and a plurality of y blocks in the vertical direction, as shown in Fig. 8. Each block may be a square or may have another shape.

[0070] The image processing unit 105 calculates the average value of the distance information for each divided block. At this time, the image processing unit 105 refers to the flag added to the image data in step S602, calculates the average value of the distance information only for the LED display area, and does not include the distance information of the subject area in the calculation of the average value. The image processing unit 105 stores the average value of the distance information for each block in the frame memory 111.

[0071] Next, in step S604, the image processing unit 105 reads out from the frame memory 111 the average value of the image data and distance information of the correction target block and the average value of the image data and distance information of the left adjacent block adjacent to the correction target block on the left side. For example, the upper left block of all the blocks is initially set as the correction target block, and in step S613 described later, the correction target block is moved one by one in the raster direction. The image processing unit 105 calculates the angle between the area corresponding to the correction target block on the display surface of the LED display LD and the imaging direction (imaging optical axis) of the imaging device 100 for each correction target block as the arrangement information indicating the arrangement relationship between the LED display LD and the imaging device 100. Then, brightness value correction is performed to change the brightness value for each correction target block according to the angle. Note that the angle referred to here may indicate the angle itself, or may be a value that can be converted to or associated with an angle, and may be information related to the angle. In the following description, the above angle is referred to as the angle between the LED display LD (area corresponding to the block) and the imaging device 100.

[0072] First, as shown in FIG. 9(a), the image processing unit 105 calculates the angle between the imaging device 100 and an area corresponding to the left half (adjacent block side) of the block to be corrected on the LED display LD, using the average value of the distance information of the block to be corrected and the adjacent block to the left.

[0073] A method for calculating the angle between the LED display LD and the imaging device 100 will be described with reference to Figs. 10(a) to (c). Fig. 10(a) shows the positional relationship between the imaging device 100 and the LED display in a horizontal plane. In Fig. 10(a), vertex A indicates the imaging position of the imaging device 100, vertex B indicates a point on the LED display that is captured at the center point of the block to be corrected, and vertex C indicates a point on the LED display that is captured at the center point of the block to the left. The distance Lc from vertex A to vertex B and the distance Lb from vertex A to vertex C are known from the average value of the distance information.

[0074] Vertex A is a vertex that connects the center points of two adjacent blocks when image data obtained by imaging at the angle of view of lens unit 106 is divided into x blocks in the horizontal direction, and the imaging position of imaging device 100. Therefore, the angle degA of vertex A is a value obtained by dividing the angle of view of lens unit 106 by x. Image processing unit 105 can calculate the distance La from vertex B to vertex C by the cosine theorem shown in the following equation (10) using the lengths of the two sides of triangle ABC (distances Lc, Lb) and angle degA.

[0075]

number

[0076] Once the lengths of the three sides of triangle ABC (distances La, Lb, Lc) have been determined in this way, the angle degB of vertex B and the angle degC of vertex C can be determined using the sine theorem shown in equation (11) below.

[0077]

number

[0078] Next, set up a triangle ABD with vertex D at the position that bisects the distance La between vertices B and C as shown in Figure 10(b). Since the lengths of the two sides of triangle ABD (distance Lc, La / 2) and the angle degB of vertex B are known, the angle degD of vertex D can be found using the cosine theorem in equation (10) and the sine theorem in equation (11).

[0079] In this way, by using the average value of the distance information of two points, the center point of the block to be corrected and the center point of the adjacent block to the left, it is possible to obtain the angle degD, which is the angle between the imaging device 100 and the area corresponding to the left half of the block to be corrected on the LED display LD.

[0080] When calculating the angle between the LED display LD and the imaging device 100 using the correction target block and its adjacent block to the right as shown in Fig. 9(b), the following can be done. The vertex C in Fig. 10(a) and (b) is the point on the LED display that is reflected at the center point of the correction target block, and the vertex B is the point on the LED display that is reflected at the center point of the adjacent block to the right. When calculating the above angle using vertically adjacent blocks as shown in Fig. 9(c) and (d), since the image data is divided into y pieces in the vertical direction, the angle degA of the vertex A can be set as the value obtained by dividing the angle of view of the lens unit 106 by y.

[0081] Next, in step S605, the image processing unit 105 multiplies the luminance value of the image data of the left half of the block to be corrected, which is indicated by diagonal lines in FIG. 9A, by the luminance correction value corresponding to the angle obtained in step S604. The ROM 103 holds an LUT (look-up table) in which the angle is an input value and the luminance correction value as a luminance correction parameter (image processing parameter) is an output value. The image processing unit 105 reads out the luminance correction value corresponding to the angle obtained in step S604 from the LUT. The LUT of the luminance correction value holds luminance correction values ​​from angles 0° to 180°, and the luminance correction value when the angle is 90° is multiplied by 1, and the luminance correction value decreases toward 0 as the angle moves away from 90°. This luminance correction value is a value that changes (increases) the luminance value of the image data in the opposite direction to the viewing angle characteristic of the LED display LD, in which the luminance of the LED decreases as the observation position moves away from 90°. The image processing unit 105 may calculate and obtain the luminance correction value corresponding to the angle by using a formula or the like. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111 .

[0082] Next, in step S606, the image processing unit 105 reads out the average value of the image data and distance information of the correction target block and the average value of the image data and distance information of the right adjacent block adjacent to the right side of the correction target block from the frame memory 111. The image processing unit 105 calculates the angle between the imaging device 100 and an area of ​​the LED display LD that corresponds to the right half of the correction target block, using the average value of the distance information of the correction target block and the right adjacent block, as shown in FIG.

[0083] Next, in step S607, the image processing unit 105 multiplies the luminance value of the image data of the right half of the correction target block shown by the diagonal lines in FIG. 9B by the luminance correction value according to the angle calculated in step S606. The luminance correction value used in this step is the same as that used in step S605. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111.

[0084] Next, in step S608, the image processing unit 105 reads out the average value of the image data and distance information of the correction target block and the average value of the image data and distance information of the upper neighboring block adjacent to the correction target block on the upper side from the frame memory 111. The image processing unit 105 calculates the angle between the imaging device 100 and a region of the LED display LD that corresponds to the upper half of the correction target block, using the average value of the distance information of the correction target block and the upper neighboring block, as shown in Fig. 9(c).

[0085] Next, in step S609, the image processing unit 105 multiplies the luminance value of the image data of the upper half of the correction target block shown by the diagonal lines in FIG. 9C by the luminance correction value according to the angle calculated in step S608. The luminance correction value may be the same as that used in steps S605 and S607, or may be a value for correcting the luminance value in the opposite direction to the vertical viewing angle characteristic when the vertical viewing angle characteristic of the LED display LD is different from the horizontal viewing angle characteristic. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111.

[0086] Next, in step S610, the image processing unit 105 reads out the average value of the image data and distance information of the correction target block and the average value of the image data and distance information of the lower neighboring block adjacent to the correction target block on the lower side from the frame memory 111. The image processing unit 105 calculates the angle between the imaging device 100 and an area of ​​the LED display LD that corresponds to the lower half of the correction target block, using the average value of the distance information of the correction target block and the lower neighboring block, as shown in FIG.

[0087] Next, in step S611, the image processing unit 105 multiplies the luminance value of the image data of the lower half of the correction target block shown by the diagonal lines in FIG. 9D by the luminance correction value according to the angle calculated in step S610. The luminance correction value used in this step is the same as that used in step S609. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111.

[0088] Next, in step S612, the image processing unit 105 determines whether or not the luminance value correction in steps S604 to S611 has been performed on all the blocks of x horizontal by y vertical, and ends this process if the correction has been performed on all the blocks. If there are any blocks that have not yet been corrected, the process in step S613 is performed.

[0089] In step S613, the image processing unit 105 moves the correction target block in the raster direction, and then the process returns to step S604.

[0090] By carrying out the above processing, it is possible to correct the luminance value of the background image area in the image data (input image) according to the angle between the LED display LD and the imaging device 100. Therefore, it is possible to obtain image data (output image) in which the image quality unevenness of the background image area is reduced. Moreover, since the emission luminance of the LED display LD itself is not changed, even when imaging is performed simultaneously using a plurality of imaging devices arranged at different angles, it is possible to obtain image data in which the image quality unevenness of the background image area is reduced in each imaging device.

[0091] In this embodiment, the angle between the LED display and the imaging device is calculated using the average value of the distance information of each block. However, other distance information, such as distance information of the center position of each block, may be used instead of the average value. EXAMPLES

[0092] Next, a description will be given of Example 2. In Example 2, image quality unevenness caused by the distance between the LED display and the image capturing device 100 is corrected, rather than image quality unevenness caused by the angle between the LED display and the image capturing device 100.

[0093] The flowchart in Fig. 11 shows the process executed by the image processing unit 105 according to a program in the embodiment 2. The process from step S601 to step S603 is the same as the process of the steps with the same reference numerals in the embodiment 1 (Fig. 6).

[0094] After step S603 and in step S1101, the image processing unit 105 reads out the average value of the distance information for each divided block in the image data (input image) from the frame memory 111. At this time, the image processing unit 105 normalizes the average value of the distance information for each block by setting the average value of the distance information for all blocks to 1. The image processing unit 105 stores the normalized average value of the distance information for each block in the frame memory 111 as the arrangement information of the LED display LD and the imaging device 100.

[0095] Next, in step S1102, the image processing unit 105 reads out the image data of the correction target block and the average value of the normalized distance information from the frame memory 111. The image processing unit 105 calculates a luminance correction value from the average value of the normalized distance information. That is, the image processing unit 105 calculates a luminance correction value according to the distance. In this embodiment, the luminance of the image data is inversely proportional to the square of the distance between the LED display and the imaging device 100. Therefore, the image processing unit 105 sets the luminance correction value according to the distance as the reciprocal of the value obtained by squaring the average value of the normalized distance information.

[0096] Next, in step S1103, the image processing unit 105 multiplies the luminance value of the image data of the correction target block by the luminance correction value according to the distance calculated in step S1102. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111.

[0097] Next, in step S1104, the image processing unit 105 determines whether or not the luminance value correction in steps S1102 to S1103 has been performed on all of the horizontal x by vertical y blocks, and ends this process if correction has been performed on all of the blocks. If there are any blocks that have not yet been corrected, the process of step S613 is performed. The process of step S613 is the same as the process of the step of the same symbol in the first embodiment (FIG. 6), but in this embodiment, the process returns to the process of step S1102.

[0098] By carrying out the above processing, it is possible to perform luminance correction of the background image area in the image data (input image) according to the distance of the imaging device 100 from the LED display. Therefore, it is possible to obtain image data (output image) in which image quality unevenness in the background image area is reduced. Moreover, since the emission luminance of the LED display LD itself is not changed, even when imaging is performed simultaneously using a plurality of imaging devices arranged at different distances from each other, it is possible to obtain image data in which image quality unevenness in the background image area is reduced in each imaging device.

[0099] In the first and second embodiments, if the LED display is configured with LEDs of multiple colors, for example, three colors R (red), G (green), and B (blue), it is desirable to prepare a luminance correction value for angle or distance for each of the R, G, and B colors of the image data and perform luminance value correction for each color.

[0100] Furthermore, instead of obtaining a luminance correction value and performing luminance value correction for each block to be corrected, a map of luminance correction values ​​for all blocks may be created, and luminance value correction may be performed using the luminance correction value obtained by performing two-dimensional filtering processing such as low-pass filtering on this map.

[0101] In the above-described first and second embodiments, the distance information is not limited to the process using the image sensor shown in Fig. 3, but may be acquired using a distance measuring device capable of measuring the distance to the subject, which is provided in the image sensor 100 separately from the image sensor. Examples of the distance measuring device include radar and LiDAR (Light Detection and Ranging).

[0102] In the above-mentioned Examples 1 and 2, an LED display was used as the background display. However, the background display is not limited to an LED display, and the luminance value can be corrected according to at least one of the angle and the distance even when a background display using various display elements such as a liquid crystal element or an organic EL element having a viewing angle characteristic in which the luminance changes according to the angle or distance is used.

[0103] In the above embodiments, the image processing unit 105 performs image processing to correct the luminance value of image data according to the angle and distance between the background display LD and the imaging device 100, but other image processing according to the angle and distance may be performed. For example, when the background display LD is a VA (Vertical Alignment) panel or the like, the color displayed is likely to be impaired according to the angle between the background display LD and the imaging device 100, so it is preferable to perform color correction processing according to the angle.

[0104] Furthermore, the image processing unit 105 may add the above-mentioned map of brightness correction values ​​as metadata to image data serving as an input image, and output this image data to the outside from output units such as the network module 108, the video terminal 109, and the recording medium 112. This allows the brightness value correction of the image data to be performed in an external device by referring to the map of brightness correction values.

[0105] Furthermore, in the first and second embodiments, the image processing unit 105 as an information processing device is built into the imaging device 100. However, a personal computer having the same functions as the image processing unit 105 (operating according to a program that realizes the same functions) may be provided as an information processing device separate from the imaging device.

[0106] The above embodiment includes the following configurations.

[0107] (Configuration 1) an acquisition means for acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; A discrimination means for discriminating between a subject area and a background image area in the input image; An information processing device comprising: a processing means for acquiring placement information regarding at least one of the angle and distance between the background display and the imaging device, and performing image processing on the background image area in accordance with the placement information. (Configuration 2) 2. The information processing device according to configuration 1, wherein the image processing is a process for changing a luminance value of the background image area. (Configuration 3) The information processing device described in configuration 2, characterized in that the image processing is a process of changing the luminance value of the background image area in a direction opposite to the change in luminance of the incident light when the luminance of the incident light from the background display to the imaging device changes depending on at least one of the angle and the distance. (Configuration 4) 4. The information processing device according to any one of configurations 1 to 3, wherein the processing means preliminarily holds or calculates image processing parameters corresponding to the layout information, and performs the image processing using the image processing parameters. (Configuration 5) The discrimination means is 5. The information processing device according to any one of configurations 1 to 4, wherein the object region and the background image region are distinguished using information relating to a distance from the imaging device to the object and the background display. (Configuration 6) The processing means includes: Dividing the background image area into a plurality of blocks; 6. The information processing device according to any one of configurations 1 to 5, wherein the arrangement information is acquired for each of the blocks and the image processing is performed. (Configuration 7) The processing means includes: obtaining the arrangement information of a target block and an adjacent block to the target block among the plurality of blocks; 7. The information processing device according to configuration 6, wherein the image processing is performed on a portion of the target block on the side of the adjacent block in accordance with the arrangement information of the target block and the adjacent block. (Configuration 8) 8. The information processing device according to any one of configurations 1 to 7, wherein the processing means obtains information regarding the angle using information regarding a distance from the imaging device to the background display. (Configuration 9) 9. The information processing device according to any one of configurations 1 to 8, wherein, when the background image has a plurality of colors, the image processing is performed on the input image for each of the colors. (Configuration 10) 10. The information processing device according to any one of configurations 1 to 9, wherein the processing means adds image processing parameters corresponding to the layout information to the input image and outputs the image to the outside. (Configuration 11) a plurality of pixels of an image sensor provided in the image capturing device each having a plurality of photoelectric conversion elements that receive light from different regions of a pupil of the optical system; 11. The information processing device according to any one of configurations 1 to 10, wherein the processing means acquires the arrangement information from a phase difference between a pair of image signals generated using signals output from the plurality of photoelectric conversion elements of the plurality of pixels. (Configuration 12) An information processing device according to any one of configurations 1 to 11; and an imaging element.

[0108] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0109] The embodiments described above are merely representative examples, and various modifications and alterations are possible for each embodiment when implementing the present invention. [Explanation of symbols]

[0110] 100 Imaging device 102 CPU 105 Image processing section 106 Lens unit 107 Imaging unit LD LED Display

Claims

1. an acquisition means for acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; A discrimination means for discriminating between a subject area and a background image area in the input image; An information processing device comprising: a processing means for acquiring placement information regarding at least one of the angle and distance between the background display and the imaging device, and performing image processing on the background image area in accordance with the placement information.

2. 2. The information processing apparatus according to claim 1, wherein the image processing is a process for changing a luminance value of the background image area.

3. The information processing device described in claim 2, characterized in that the image processing is a process of changing the brightness value of the background image area in a direction opposite to the change in brightness of the incident light when the brightness of the incident light from the background display to the imaging device changes depending on at least one of the angle and the distance.

4. 2. The information processing apparatus according to claim 1, wherein said processing means holds or calculates image processing parameters corresponding to said layout information in advance, and performs said image processing using said image processing parameters.

5. The discrimination means is 2 . The information processing apparatus according to claim 1 , wherein the subject area and the background image area are distinguished using information about distances from the imaging device to the subject and the background display.

6. The processing means includes: Dividing the background image area into a plurality of blocks; 2. The information processing apparatus according to claim 1, wherein the image processing is performed by acquiring the layout information for each of the blocks.

7. The processing means includes: obtaining the arrangement information of a target block and an adjacent block to the target block among the plurality of blocks; 7. The information processing apparatus according to claim 6, wherein the image processing is performed on a portion of the target block on the adjacent block side in accordance with the arrangement information of the target block and the adjacent block.

8. 2 . The information processing apparatus according to claim 1 , wherein the processing means acquires the information regarding the angle using information regarding a distance from the imaging device to the background display.

9. 2 . The information processing apparatus according to claim 1 , wherein, when the background image has a plurality of colors, the image processing is performed on the input image for each of the colors.

10. 2. The information processing apparatus according to claim 1, wherein said processing means adds image processing parameters corresponding to said layout information to said input image and outputs the input image to an outside source.

11. a plurality of pixels of an image sensor provided in the image capturing device each having a plurality of photoelectric conversion elements that receive light from different regions of a pupil of the optical system; 2. The information processing apparatus according to claim 1, wherein the processing means acquires the arrangement information from a phase difference between a pair of image signals generated using signals output from the plurality of photoelectric conversion elements of the plurality of pixels.

12. An information processing device according to any one of claims 1 to 11; and an imaging element.

13. acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display by an imaging device; Distinguishing a subject area and a background image area in the input image; An information processing method comprising the steps of acquiring placement information regarding at least one of the angle and distance between the background display and the imaging device, and performing image processing on the background image area in accordance with the placement information.

14. A program for causing a computer to execute a process according to the information processing method according to claim 13.

Citation Information

Patent Citations

  • System and Method for Realtime LED Viewing Angle Correction

    US20220014728A1