Information processing device and imaging device
The information processing device addresses image quality issues in virtual production by calculating and correcting brightness based on the angle and distance between the background display and imaging device, ensuring consistent image quality across different imaging setups.
Patent Information
- Application Number
- PCT/JP2024/012563
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-25
- Filing Date
- 2024-03-28
- Publication Date
- 2024-11-28
AI Technical Summary
Existing imaging technologies face challenges in maintaining uniform image quality in the background video area due to the viewing angle characteristics of LEDs in virtual production setups, particularly when imaging devices are not directly facing the background display.
An information processing device and method that acquire input images, determine the angle and distance between the background display and the imaging device, and perform image processing to correct brightness unevenness in the background video area, using techniques such as correlation calculations and brightness value corrections based on the calculated angles or distances.
This approach ensures reduced image quality unevenness in the background video area regardless of the arrangement between the background display and the imaging device, even when multiple devices capture images from different angles, without altering the LED display's brightness characteristics.
Smart Images

Figure JP2024012563_28112024_PF_FP_ABST
Abstract
Description
Information processing device and imaging device
[0001] The present invention relates to an information processing device that processes information related to imaging.
[0002] One imaging technique used in movies and television is virtual production, in which a background image is displayed on a background display that is larger than the subject and then captured together with the subject. However, LEDs, the display elements primarily used in background displays, have viewing angle characteristics. For this reason, unless the imaging device is positioned so as to directly face the background display, image quality variations (brightness variations) will occur in the background image area of the image data obtained by capturing the image.
[0003] Patent Document 1 discloses a method of calculating the angle between an LED display and an imaging device from position information of the imaging device, and correcting the brightness of the LED display using a brightness correction value according to the angle.
[0004] U.S. Patent Publication No. 2022 / 0014728
[0005] In the method of Patent Document 1, the brightness of the LED display itself is changed depending on the angle between it and a single imaging device. As a result, image data with reduced image quality unevenness in the background image area is obtained when capturing images using that single imaging device. However, if another imaging device simultaneously captures images from a different angle, there is a risk that image data with image quality unevenness in the background image area will not be reduced.
[0006] The present invention provides an information processing device, an imaging device, and the like that are capable of obtaining image data with reduced image quality unevenness in the background image area, regardless of the arrangement of the background display and the imaging device.
[0007] An information processing device according to one aspect of the present invention comprises: an acquisition means for acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display using an imaging device; a discrimination means for discriminating between a subject area and a background image area in the input image; and a processing means for acquiring position information relating to at least one of the angle and the distance between the background display and the imaging device, and performing image processing on the background image area in accordance with the position information. Note that an imaging device having the above-described information processing device also constitutes another aspect of the present invention.
[0008] Another aspect of the present invention is an information processing method comprising the steps of: acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display using an imaging device; distinguishing between a subject area and a background image area in the input image; acquiring layout information regarding at least one of the angle and the distance between the background display and the imaging device; and performing image processing on the background image area in accordance with the layout information. Note that a program for causing a computer to execute processing in accordance with the above information processing method also constitutes another aspect of the present invention.
[0009] According to the present invention, an image with reduced image quality unevenness in the background image area can be obtained regardless of the arrangement of the background display and the imaging device.
[0010] 1 is a block diagram showing the configuration of an imaging device according to an embodiment; a diagram showing a part of the light receiving surface of an imaging element; a flowchart showing a process for calculating distance information according to an embodiment; a diagram showing a correlation calculation; another diagram showing a correlation calculation; a flowchart showing a process according to a first embodiment; a diagram showing an imaging state according to the first embodiment and a diagram showing a subject area and an LED display area in image data; a diagram showing division of image data according to the first embodiment; a diagram showing a correction target block and a peripheral block according to the first embodiment; a schematic diagram showing an imaging position according to the first embodiment and two adjacent blocks on adjacent LED displays in image data; a flowchart showing a process according to a second embodiment;
[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, an image capture device (digital camera) equipped with an information processing device that executes the processes of specific embodiments 1 and 2 described below will be described. Figure 1 shows the configuration of the image capture device 100. The image capture device 100 captures images, processes the images, and records the images.
[0012] Connected to the internal bus 101 are a CPU 102, a ROM 103, a RAM 104, an image processing unit 105, a lens unit 106, an imaging unit 107, a network module 108, a video terminal 109, a recording medium I / F 110, a display unit 114, and an object detection unit 115. The units connected to the internal bus 101 can exchange data with each other via the internal bus 101.
[0013] The lens unit 106 has an optical system including a zoom lens, a focus lens, and an aperture mechanism, and a motor that drives the optical system. An optical image formed by the lens unit 106 is converted into an image signal as an analog electrical signal (i.e., captured) by an image sensor such as a CCD sensor or a CMOS sensor included in the image capture unit 107. The image capture unit 107 also includes an A / D converter that converts the image signal output from the image sensor into a digital signal (image data). The image sensor is an image-surface phase difference sensor, the details of which will be described later.
[0014] A CPU 102 as a computer controls each unit of the image capturing apparatus 100 according to a program stored in a ROM 103 and uses a RAM 104 as a work memory.
[0015] The ROM 103 is a non-volatile memory that stores programs and various parameters for operating the CPU 102. The RAM 104 is a volatile memory. The frame memory 111 is a memory that temporarily stores image data and allows the image data to be read out when needed. By using the frame memory 111, it is possible to perform processes such as combining images obtained by capturing images at different times and cutting out specific areas from captured images.
[0016] The image processing unit 105, which serves as an information processing device, performs various image processing operations on image data from the image capture unit 107 or image data stored in the frame memory 111 or the recording medium 112 under the control of the CPU 102. The image processing operations performed by the image processing unit 105 include pixel interpolation, encoding, compression, decoding, enlargement / reduction (resizing), noise reduction, color conversion, and the like. The image processing unit 105 also corrects performance variations among multiple pixels of the image sensor, corrects defective pixels, corrects white balance, corrects brightness, and corrects distortion and peripheral light loss caused by lens characteristics. Furthermore, the image processing unit 105 generates distance information (described below) and corrects brightness values of image data according to the angle and distance between the background display (described below) and the image capture device 100. The image processing unit 105 may be configured as a dedicated image processing computer for performing image processing, or the CPU 102 may have the functions of the image processing unit 105.
[0017] The CPU 102 performs optical zoom, focusing, and aperture adjustment by controlling each motor of the lens unit 106 based on the calculation results for lens control obtained from the image processing unit 105. The CPU 102 may also perform image blur correction by moving a lens that constitutes a part of the optical system in a direction perpendicular to the optical axis in accordance with camera shake detected by an attitude detection unit 116 (described later).
[0018] The operation unit 113 accepts operations by the user and outputs instructions corresponding to the operations to the CPU 102 and the image processing unit 105. The operation unit 113 has operation members such as switches, dials, and buttons. The operation members include a power switch, a mode switching dial, and a release button. When the user presses the release button halfway, it instructs the CPU 102 to start image capture preparation operations such as autofocus (AF) and exposure control (AE). When the user presses the release button all the way, it causes the CPU 102 to start an image capture operation for acquiring image data for recording. The operation unit 113 may be a touch sensor provided on the display unit 114.
[0019] The display unit 114, which serves as a display means, displays images processed by the image processing unit 105, setting menus, etc. The display unit 114 has a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display. The display unit 114 may also be provided with the touch sensor described above.
[0020] The CPU 102 generates text information and icons to inform the user of the setting status of the imaging device 100, as well as images of a setting menu for making various settings of the imaging device 100, and displays these on the display unit 114 by superimposing them on the image processed by the image processing unit 105.
[0021] The video terminal 109 is a video interface terminal such as SDI (Serial Digital Interface), HDMI (High Definition Multimedia Interface) (registered trademark), DisplayPort (registered trademark), etc. The CPU 102 can output video (image data) to an external monitor or the like via the video terminal 109.
[0022] The network module 108 is an interface for inputting and outputting image data and audio signals. The CPU 102 communicates with external devices via the network module 108 over the Internet or the like, and can transmit and receive not only image data and audio signals but also control signals such as commands. The network module 108 may be of a wireless communication type or a wired communication type.
[0023] The recording medium 112 is a hard disk drive (HDD) or a solid state drive (SSD) capable of recording image data and various setting data, and is detachably attached to the recording medium I / F 110 .
[0024] The object detection unit 115 detects objects using artificial intelligence, such as deep learning using a neural network. In object detection using deep learning, the CPU 102 transmits a program, a network structure, weight parameters, and the like stored in the ROM 103 to the object detection unit 115. Examples of the network structure include SSD (Single Shot Multibox Detector) and YOLO (You Only Look Once). The object detection unit 115 performs processing to detect objects from image data based on various parameters obtained from the CPU 102 and transmits the processing result to the RAM 104.
[0025] The orientation detection unit 116 uses an angular velocity sensor, an acceleration sensor, etc. to detect the orientation and shake of the image capturing device 100. This makes it possible to detect the tilt of the image capturing device 100 and camera shake due to hand shake or the like.
[0026] FIG. 2A schematically illustrates a portion of an image sensor in which red (R), blue (B), and green (Gb, Gr) pixels are arranged in a Bayer array. As shown in FIG. 2B, each pixel has two photodiodes A and B as photoelectric conversion elements, each of which is provided with a microlens (not shown). With this configuration, the two photodiodes A and B receive light from two different regions of the exit pupil of the lens unit 106 and output A and B signals. The image capturing unit 107 combines the A and B signals output from each of the multiple pixels on the image sensor to generate a pair of image signals (A and B image signals) for phase difference detection. Furthermore, the A and B image signals are added together to generate an image signal for obtaining image data.
[0027] In addition to outputting an A signal and a B signal from each pixel, the imaging unit 107 can also output an arithmetic average of the A signals and B signals from multiple pixels that are close to each other. Outputting an arithmetic average value can shorten the time required to read signals from the imaging unit 107 and reduce the bandwidth of the internal bus 101.
[0028] The CPU 102 performs a correlation calculation on the pair of image signals to calculate the phase difference between them, and calculates defocus amount information from the phase difference. The defocus amount indicates the degree of defocus relative to the in-focus state of the lens unit 106. The CPU 102 determines the in-focus state if the defocus amount is within a predetermined range including 0, and determines the front-focus state and back-focus state, respectively, if the defocus amount is positive or negative. The CPU 102 performs image plane phase-difference AF by controlling a motor that drives the focus lens in the lens unit 106 according to the positive or negative defocus amount. The CPU 102 also calculates the reliability of the calculated phase difference. The CPU 102 then calculates the distance to the subject using the parallax between the pair of image signals and optical information from the lens unit 106 using the principle of triangulation.
[0029] 2B shows a case where two photodiodes are provided for each pixel, but the number of photodiodes for each pixel may be three or more. Furthermore, the image sensor may be provided with multiple pixels with different aperture positions, and these multiple pixels may be treated as one pixel for focus detection. In this way, the image sensor may be configured to obtain a pair of image signals for phase difference detection.
[0030] 3 to 5, the information generation process performed by the image processing unit 105 will be described. The image processing unit 105 functions as an acquisition unit, a determination unit, and a processing unit.
[0031] 3 shows a distance calculation process that the image processing unit 105 executes according to a program in response to the start of imaging in the imaging unit 107. First, in step S301, the image processing unit 105 calculates (acquires) a B image signal by determining the difference between the A image signal and the imaging signal (A image signal + B image signal) output from the imaging unit 107. Note that in this step, the A image signal and the B image signal may be acquired separately from the imaging unit 107. In this case, a signal obtained by adding the A image signal and the B image signal can be acquired as the imaging signal. Furthermore, in a case where two lens units and two imaging elements are provided, such as in a stereo camera, the A image signal may be acquired from one of the two imaging elements, and the B image signal may be acquired from the other.
[0032] Next, in step S302, the image processing unit 105 corrects shading due to optical factors for the A and B image signals.
[0033] Next, in step S303, the image processing unit 105 performs filtering on each of the A and B image signals. The filtering is, for example, high-pass filtering using a finite impulse response (FIR) filter. The following describes a case where the A and B image signals that have been subjected to high-pass filtering are used, but filtering may also be performed using a band-pass filter or low-pass filter with different filter coefficients.
[0034] Next, in step S304, the image processing unit 105 divides each of the A and B image signals that have been filtered in step S303 into minute blocks, and performs correlation calculations for each divided block. There are no particular restrictions on the size or shape of the minute blocks, and adjacent blocks may partially overlap each other.
[0035] The correlation calculation between the A and B image signals will be described below. The signal sequence of the A image signal at a pixel of interest is designated as A image signal sequence E(1) to E(m), and the signal sequence of the B image signal at a pixel of interest is designated as B image signal sequence F(1) to F(m). While the B image signal sequence F(1) to F(m) is shifted relative to the A image signal sequence E(1) to E(m), the correlation amount C(k) at a shift amount k between these pairs of image signal sequences is calculated using the following equation (1):
[0036]
[0037] In equation (1), the Σ operation refers to an operation to calculate a sum with respect to n. In the Σ operation, the range of n and n+k is limited to the range from 1 to m. The amount of shift k is an integer value, and is the amount of relative pixel shift in units of the detection pitch (pixel arrangement pitch) of the paired image signal sequences. FIG. 4 shows the results of the operation of equation (1) when the paired image signal sequences are highly correlated in an ideal state where no noise is present.
[0038] As shown in Figure 4, the correlation amount C(k) is minimum at a shift amount where the correlation between paired image signal sequences is high (k = kj = 0). The k at which the discrete correlation amount C(k) is minimum is denoted as kj. The pixel shift amount (phase difference) x that gives the minimum value C(x) for continuous correlation amounts is calculated by the three-point interpolation process shown in equations (2) to (4). The pixel shift amount x is a real value, and its unit is pixel.
[0039]
[0040]
[0041]
[0042] SLOP in equation (4) represents the gradient of change in the minimum and local minimum correlation amount and the correlation amount adjacent to it. In FIG. 4, as a specific example, C(kj) = C(0) = 1000, C(kj-1) = C(-1) = 1700, and C(kj+1) = C(1) = 1830 are used. In this example, kj = 0. From equations (2) to (4), SLOP = 830 and x = -0.078 pixels are obtained. In a focused state, the pixel shift amount x between the paired image signal sequences is ideally 0.00.
[0043] On the other hand, the calculation results when equation (1) is applied to a small block containing noise are shown in Fig. 5. As shown in Fig. 5, the correlation between paired image signal sequences decreases due to the influence of randomly distributed noise. The minimum value of the correlation amount C(k) is larger than the minimum value shown in Fig. 4, and the correlation amount curve has an overall flat shape (the absolute value of the difference between the maximum and minimum values is small).
[0044] 5, as a specific example, C(kj) = C(0) = 1300 C(kj-1) = C(-1) = 1480 C(kj+1) = C(1) = 1800 In this example, kj = 0. From equations (2) to (4), SLOP = 500 x = -0.32 pixels.
[0045] Compared with the calculation result in the absence of noise shown in FIG. 4, the pixel shift amount x is farther from the ideal value.
[0046] When the correlation between the paired image signal sequences is low, the change in the correlation amount C(k) is small, and the correlation amount curve is generally flat, resulting in a small SLOP value. Similarly, when the subject image has low contrast, the correlation between the paired image signal sequences is low, and the correlation amount curve is also flat. Based on this property, the reliability of the calculated pixel shift amount x can be determined from the SLOP value. That is, when the SLOP value is large, it can be determined that the correlation between the paired image signal sequences is high, and when the SLOP value is small, it can be determined that no significant correlation has been obtained between the paired image signal sequences. In this embodiment, because Equation (1) is used for correlation calculation, the correlation amount C(k) is minimum and minimal at the shift amount k at which the correlation between the paired image signal sequences is highest. Alternatively, a correlation calculation method in which the correlation amount C(k) is maximum and maximal at the shift amount k at which the correlation between the paired image signal sequences is highest may be used.
[0047] Next, in step S305, the image processing unit 105 calculates the reliability of the calculated pixel shift amount x. The reliability can be defined by the value of SLOP and C(kj), which indicates the degree of match between the pair of image signals calculated in step S304.
[0048] Next, in step S306, the image processing unit 105 performs interpolation processing. Although the correlation calculation was successful in step S304, there may be cases where the pixel shift amount x of the pixel of interest cannot be adopted due to low reliability calculated in step S305. In such cases, the pixel shift amount of the pixel of interest must be obtained by interpolation using pixel shift amounts calculated for the pixels surrounding the pixel of interest. As a method of interpolation, a median filter may be applied to the pixel shift amount calculated for the surrounding pixels, or the pixel shift amount data may be reduced and then re-enlarged. Alternatively, color data may be extracted from the imaging signal, and the pixel shift amount may be interpolated using the color data.
[0049] Next, in step S307, the image processing unit 105 calculates the defocus amount by referring to the pixel shift amount x calculated in step S304 or acquired by interpolation in step S306. The defocus amount (DEF) can be calculated by the following equation (5).
[0050]
[0051] In equation (5), P is a conversion coefficient determined by the detection pitch and the distance between the projection centers of the two viewpoints in the pair of parallax images (image signal A and image signal B) (the distance between the center of gravity of the image sensor that generates image signal A and the center of gravity of the image sensor that generates image signal B). The unit of the conversion coefficient is mm / pixel.
[0052] Next, in step S308, the image processing unit 105 calculates the distance from the defocus amount calculated in step S307. When the distance to the subject is Da, the focal position of the lens unit 106 is Db, and the focal length is F, the following equation (6) approximately holds.
[0053]
[0054] Therefore, the distance Da to the subject is
[0055]
[0056] This becomes:
[0057] If Db when DEF=0 is Db0, then equation (7) becomes:
[0058]
[0059] From the above, the absolute distance Da' to the subject can be calculated. The relative distance Da-Da' can be calculated from the equations (7) and (8) as follows:
[0060]
[0061] It can be found by:
[0062] By the above-described processing shown in FIG. 3, the pixel shift amount, defocus amount, and distance to the subject can be obtained from the A and B image signals.
[0063] The pixel shift amount and defocus amount acquired in this manner change depending on the distance from the imaging device 100. Therefore, the pixel shift amount, defocus amount, and distance are all information relating to the distance from the imaging device 100. In the following description, information relating to distance is referred to as distance information. Furthermore, the information relating to distance is not limited to distance itself, but may also be information that can be converted into or associated with distance.
[0064] Next, a first embodiment will be described. Fig. 7(a) shows how an image of a subject OBJ positioned in front of an LED display LD serving as a background display is captured by an imaging device 100. The LED display LD is configured with a large number of LEDs serving as display elements arranged in a two-dimensional direction, and displays a background image corresponding to a video signal supplied from a video supply device (not shown). Image data obtained by capturing an image shows the subject OBJ against the background image.
[0065] In FIG. 7A , the imaging device 100 captures an image of the subject OBJ and the LED display LS from a direction different from the direction of the normal N (direction facing the LED display LD), which is flat. The background display is not limited to a flat surface, but may be curved or spherical. When the angle between the imaging direction of the imaging device 100 and the display surface of the LED display LD is significantly different from 90°, the luminance of the light incident from the LED display LD to the imaging device 100 decreases due to the viewing angle characteristics of the LED display LD. As a result, image quality variations (luminance variations) due to the angle occur in the background image area of the image data obtained by the imaging device 100. Therefore, it is desirable to perform luminance value correction on the image data as image processing to reduce this image quality variation.
[0066] The flowchart in FIG. 6 shows processing that the image processing unit 105 in the imaging device 100 in the first embodiment executes according to a program.
[0067] First, in step S601, the image processing unit 105 acquires image data as an input image obtained by capturing an image of the object OBJ and the LED display LD, and information on the distance between the object OBJ and the LED display LD, and stores these in the frame memory 111.
[0068] Next, in step S602, the image processing unit 105 reads image data and distance information from the frame memory 111. Based on the distance information, the image processing unit 105 determines and separates the subject area in the image data, which includes the subject, and the LED display area (background image area) in which the LED display LD is displayed, as shown in FIG. 7B . Specifically, a threshold value is set for the distance information, and an area in the image data having distance information equal to or greater than the threshold is determined to be the LED display area, and an area in the image data having distance information less than the threshold is determined to be the subject area. The threshold value may be arbitrarily set by the user through operation of the operation unit 113, or may be set by the CPU 102 based on the aperture value, focus position, etc. of the lens unit 106. The image processing unit 105 adds flags to the subject area and the LED display area in the image data to distinguish them, and stores the image data in the frame memory 111.
[0069] Next, in step S603, the image processing unit 105 reads the image data and distance information from the frame memory 111. As shown in Fig. 8, the image processing unit 105 divides the image data into a plurality of x blocks in the horizontal direction and a plurality of y blocks in the vertical direction. Each block may be square or may have another shape.
[0070] The image processing unit 105 calculates the average value of the distance information for each divided block. At this time, the image processing unit 105 refers to the flag added to the image data in step S602 and calculates the average value of the distance information for only the LED display area, and does not include the distance information for the subject area in the calculation of the average value. The image processing unit 105 stores the average value of the distance information for each block in the frame memory 111.
[0071] Next, in step S604, the image processing unit 105 reads from the frame memory 111 the average values of the image data and distance information of the correction target block and the average values of the image data and distance information of the left-side block adjacent to the correction target block. For example, the upper left block of all blocks is initially set as the correction target block, and in step S613 (described later), the correction target blocks are moved one by one in the raster direction. The image processing unit 105 calculates, for each correction target block, the angle between the area on the display surface of the LED display LD corresponding to the correction target block and the imaging direction (imaging optical axis) of the imaging device 100 as layout information indicating the layout relationship between the LED display LD and the imaging device 100. Then, brightness value correction is performed to change the brightness value of each correction target block according to the angle. Note that the angle referred to here may represent the angle itself, or may be a value convertible to or associated with an angle, as long as it is information related to the angle. In the following description, the above angle is referred to as the angle between the LED display LD (the area corresponding to the block) and the imaging device 100.
[0072] First, as shown in Figure 9 (a), the image processing unit 105 calculates the angle between the imaging device 100 and an area corresponding to the left half (adjacent block side) of the block to be corrected on the LED display LD, using the average value of the distance information of the block to be corrected and the block adjacent to it on the left.
[0073] A method for calculating the angle between the LED display LD and the imaging device 100 will be described with reference to Figures 10(a) to 10(c). Figure 10(a) shows the positional relationship between the imaging device 100 and the LED display in a horizontal plane. In Figure 10(a), vertex A indicates the imaging position of the imaging device 100, vertex B indicates a point on the LED display that is captured at the center point of the block to be corrected, and vertex C indicates a point on the LED display that is captured at the center point of the block to the left. The distance Lc from vertex A to vertex B and the distance Lb from vertex A to vertex C are known from the average value of the distance information.
[0074] Vertex A is the vertex that connects the center points of two adjacent blocks when image data obtained by capturing an image at the angle of view of lens unit 106 is divided horizontally into x blocks, and the capturing position of imaging device 100. Therefore, the angle degA of vertex A is the value obtained by dividing the angle of view of lens unit 106 by x. Using the lengths of the two sides of triangle ABC (distances Lc and Lb) and the angle degA, image processing unit 105 can calculate the distance La from vertex B to vertex C using the cosine theorem shown in the following equation (10).
[0075]
[0076] Once the lengths of the three sides of triangle ABC (distances La, Lb, Lc) have been determined in this way, the angle degB of vertex B and the angle degC of vertex C can be determined using the sine law shown in equation (11) below.
[0077]
[0078] Next, as shown in Figure 10(b), a triangle ABD is set with vertex D at the position that bisects the distance La between vertices B and C. Since the lengths of the two sides of triangle ABD (distance Lc, La / 2) and the angle degB of vertex B are known, the angle degD of vertex D can be found from the cosine theorem of equation (10) and the sine theorem of equation (11).
[0079] In this way, the angle degD between the area corresponding to the left half of the block to be corrected on the LED display LD and the imaging device 100 can be obtained by using the average value of the distance information of two points, the center point of the block to be corrected and the center point of the adjacent block to the left.
[0080] Note that, when calculating the angle between the LED display LD and the imaging device 100 using the correction target block and its adjacent block to the right as shown in Figure 9(b), the following can be done. Vertex C in Figures 10(a) and 10(b) is defined as the point on the LED display that is projected at the center point of the correction target block, and vertex B is defined as the point on the LED display that is projected at the center point of the adjacent block to the right. Furthermore, when calculating the angle using vertically adjacent blocks as shown in Figures 9(c) and 9(d), since the image data is divided into y parts in the vertical direction, the angle of view of the lens unit 106 divided by y can be set as the angle degA of vertex A.
[0081] Next, in step S605, the image processing unit 105 multiplies the luminance value of the image data of the left half of the correction target block (shown by the diagonal lines in FIG. 9A ) by a luminance correction value corresponding to the angle calculated in step S604. A look-up table (LUT) is stored in advance in the ROM 103. The LUT uses angles as input values and luminance correction values as luminance correction parameters (image processing parameters) as output values. The image processing unit 105 reads the luminance correction value corresponding to the angle calculated in step S604 from the LUT. The LUT for luminance correction values stores luminance correction values for angles from 0° to 180°. The luminance correction value is multiplied by 1 for an angle of 90°, and decreases toward 0 as the angle moves away from 90°. This luminance correction value changes (increases) the luminance value of the image data in the opposite direction to the viewing angle characteristic of the LED display LD, in which the luminance of the LED decreases as the viewing position moves away from 90°. The image processing unit 105 may calculate and acquire the brightness correction value corresponding to the angle using a calculation formula, etc. The image processing unit 105 stores the image data multiplied by the brightness correction value in the frame memory 111.
[0082] Next, in step S606, the image processing unit 105 reads the average value of the image data and distance information of the block to be corrected and the average value of the image data and distance information of the block adjacent to the right of the block to be corrected from the frame memory 111. The image processing unit 105 calculates the angle between the imaging device 100 and an area of the LED display LD corresponding to the right half of the block to be corrected, using the average values of the distance information of the block to be corrected and the block to the right, as shown in FIG.
[0083] Next, in step S607, the image processing unit 105 multiplies the luminance value of the image data of the right half of the correction target block, shown by the diagonal lines in FIG. 9B, by the luminance correction value corresponding to the angle calculated in step S606. The luminance correction value used here is the same as that used in step S605. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111.
[0084] Next, in step S608, the image processing unit 105 reads the average value of the image data and distance information of the block to be corrected and the average value of the image data and distance information of the block adjacent to the block to be corrected above from the frame memory 111. The image processing unit 105 calculates the angle between the imaging device 100 and an area of the LED display LD that corresponds to the upper half of the block to be corrected, using the average values of the distance information of the block to be corrected and the block adjacent to the block above, as shown in Figure 9(c).
[0085] Next, in step S609, the image processing unit 105 multiplies the luminance values of the image data of the upper half of the correction target block, shown by the diagonal lines in Figure 9(c), by a luminance correction value corresponding to the angle calculated in step S608. This luminance correction value may be the same as that used in steps S605 and S607, or, if the vertical viewing angle characteristics of the LED display LD differ from those in the horizontal direction, a value that corrects the luminance values in the opposite direction to the vertical viewing angle characteristics may be used. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111.
[0086] Next, in step S610, the image processing unit 105 reads the average value of the image data and distance information of the block to be corrected and the average value of the image data and distance information of the block adjacent below the block to be corrected from the frame memory 111. The image processing unit 105 calculates the angle between the imaging device 100 and an area of the LED display LD that corresponds to the lower half of the block to be corrected, using the average values of the distance information of the block to be corrected and the block adjacent below, as shown in FIG.
[0087] Next, in step S611, the image processing unit 105 multiplies the luminance value of the image data of the lower half of the correction target block, shown by the diagonally shaded area in FIG. 9D, by the luminance correction value corresponding to the angle calculated in step S610. The luminance correction value used here is the same as that used in step S609. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111.
[0088] Next, in step S612, the image processing unit 105 determines whether or not the luminance value correction in steps S604 to S611 has been performed on all of the horizontal x by vertical y blocks, and if correction has been performed on all of the blocks, the process ends. If there are any blocks that have not yet been corrected, the process proceeds to step S613.
[0089] In step S613, the image processing unit 105 moves the correction target block in the raster direction, and then returns to the processing of step S604.
[0090] By performing the above processing, the luminance value of the background image area in the image data (input image) can be corrected according to the angle between the LED display LD and the imaging device 100. Therefore, it is possible to obtain image data (output image) in which image quality unevenness in the background image area is reduced. Moreover, because the light emission luminance of the LED display LD itself is not changed, even when imaging is performed simultaneously using multiple imaging devices arranged at different angles, it is possible to obtain image data in which image quality unevenness in the background image area is reduced in each imaging device.
[0091] In this embodiment, the angle between the LED display and the imaging device is calculated using the average value of the distance information for each block, but other distance information, such as distance information for the center position of each block, may be used instead of the average value.
[0092] Next, a description will be given of Example 2. In Example 2, image quality unevenness due to the distance between the LED display and the image capture device 100 is corrected, rather than image quality unevenness due to the angle between the LED display and the image capture device 100.
[0093] 11 shows the process executed by the image processing unit 105 according to a program in the second embodiment. The processes from step S601 to step S603 are the same as the processes of the steps with the same reference numerals in the first embodiment (FIG. 6).
[0094] After step S603 and in step S1101, the image processing unit 105 reads the average value of the distance information for each divided block of the image data (input image) from the frame memory 111. At this time, the image processing unit 105 normalizes the average value of the distance information for each block, assuming that the average value of the distance information for all blocks is 1. The image processing unit 105 stores the normalized average value of the distance information for each block in the frame memory 111 as the arrangement information of the LED display LD and the imaging device 100.
[0095] Next, in step S1102, the image processing unit 105 reads the image data of the correction target block and the average value of the normalized distance information from the frame memory 111. The image processing unit 105 calculates a luminance correction value from the average value of the normalized distance information. That is, it calculates a luminance correction value according to the distance. In this embodiment, the luminance of the image data is inversely proportional to the square of the distance between the LED display and the imaging device 100. Therefore, the image processing unit 105 sets the luminance correction value according to the distance as the reciprocal of the square of the average value of the normalized distance information.
[0096] Next, in step S1103, the image processing unit 105 multiplies the luminance value of the image data of the correction target block by the luminance correction value corresponding to the distance calculated in step S1102. The image processing unit 105 stores the image data multiplied by the luminance correction value in the frame memory 111.
[0097] Next, in step S1104, the image processing unit 105 determines whether the luminance value correction in steps S1102 and S1103 has been performed on all of the horizontal x by vertical y blocks, and ends this process if correction has been performed on all of the blocks. If there are any blocks that have not yet been corrected, the process of step S613 is performed. The process of step S613 is the same as the process of the step of the same symbol in embodiment 1 (FIG. 6), but in this embodiment, the process returns to step S1102.
[0098] By performing the above processing, it is possible to correct the brightness of the background image area in the image data (input image) according to the distance of the imaging device 100 from the LED display. Therefore, it is possible to obtain image data (output image) in which image quality unevenness in the background image area is reduced. Moreover, because the light emission brightness of the LED display LD itself is not changed, even when imaging is performed simultaneously using multiple imaging devices located at different distances from each other, it is possible to obtain image data in which image quality unevenness in the background image area is reduced in each imaging device.
[0099] In the first and second embodiments, if the LED display is configured with LEDs of multiple colors, for example, three colors of R (red), G (green), and B (blue), it is desirable to prepare brightness correction values for the angle or distance for each of the R, G, and B colors of the image data and perform brightness value correction for each color.
[0100] Furthermore, instead of obtaining a brightness correction value and correcting the brightness value for each block to be corrected, a map of brightness correction values for all blocks may be created, and brightness value correction may be performed using the brightness correction values obtained by performing two-dimensional filtering processing such as low-pass filtering on this map.
[0101] 3 using the image sensor, the distance information may be acquired using a distance measuring device capable of measuring the distance to the subject, which is provided in the image sensing device 100 separately from the image sensor. Examples of distance measuring devices include radar and LiDAR (Light Detection and Ranging).
[0102] Furthermore, in the above-described Examples 1 and 2, an LED display was used as the background display. However, the background display is not limited to an LED display. When using a background display that uses various display elements such as liquid crystal elements or organic EL elements having viewing angle characteristics in which the brightness changes depending on the angle and distance, the brightness value can be corrected depending on at least one of the angle and the distance.
[0103] In the above embodiments, the image processing unit 105 performs image processing to correct the brightness value of image data according to the angle and distance between the background display LD and the imaging device 100. However, other image processing according to the angle and distance may be performed. For example, when the background display LD is a VA (Vertical Alignment) panel or the like, the displayed colors are likely to be impaired depending on the angle between the background display LD and the imaging device 100, so it is preferable to perform color correction processing according to the angle.
[0104] Furthermore, the image processing unit 105 may add the above-described map of brightness correction values as metadata to image data as an input image, and output this image data to the outside from an output unit such as the network module 108, the video terminal 109, and the recording medium 112. This allows an external device to perform brightness value correction of the image data by referring to the map of brightness correction values.
[0105] Furthermore, in the first and second embodiments, the image processing unit 105 as an information processing device is built into the imaging device 100. However, a personal computer having the same functions as the image processing unit 105 (operating according to a program that realizes the same functions) may be provided as an information processing device separate from the imaging device.
[0106] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.
[0107] The embodiments described above are merely representative examples, and various modifications and alterations are possible to each embodiment when implementing the present invention.
Claims
1. An information processing device comprising: an acquisition means for acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display using an imaging device; a discrimination means for discriminating between a subject area and a background image area in the input image; and a processing means for acquiring positional information relating to at least one of the angle and distance between the background display and the imaging device, and performing image processing on the background image area in accordance with the positional information.
2. The information processing device according to claim 1, wherein said image processing is a process for changing a luminance value of said background image area.
3. The information processing device described in claim 2, characterized in that the image processing is a process of changing the brightness value of the background image area in a direction opposite to the change in brightness of the incident light when the brightness of the incident light from the background display to the imaging device changes depending on at least one of the angle and the distance.
4. An information processing device according to any one of claims 1 to 3, characterized in that the processing means stores or calculates image processing parameters corresponding to the layout information in advance, and performs the image processing using the image processing parameters.
5. An information processing device as described in any one of claims 1 to 4, characterized in that the discrimination means discriminates between the subject area and the background image area using information regarding the distance from the imaging device to the subject and the background display.
6. An information processing device according to any one of claims 1 to 5, characterized in that the processing means divides the background image area into a plurality of blocks, obtains the layout information for each of the blocks, and performs the image processing.
7. The information processing device according to claim 6, characterized in that the processing means acquires the arrangement information of a target block among the plurality of blocks and an adjacent block relative to the target block, and performs the image processing on the portion of the target block on the adjacent block side in accordance with the arrangement information of the target block and the adjacent block.
8. An information processing device according to any one of claims 1 to 7, characterized in that the processing means obtains information relating to the angle using information relating to the distance from the imaging device to the background display.
9. The information processing device according to any one of claims 1 to 8, characterized in that, when the background image has a plurality of colors, the image processing is performed on the input image for each of the colors.
10. An information processing apparatus according to any one of claims 1 to 9, characterized in that said processing means adds image processing parameters corresponding to said layout information to said input image and outputs the image to the outside.
11. An information processing device as described in any one of claims 1 to 10, characterized in that each of the multiple pixels of an imaging element provided in the imaging device has multiple photoelectric conversion elements that receive light from different areas of the pupil of the optical system, and the processing means acquires the arrangement information from the phase difference of a pair of image signals generated using signals output from the multiple photoelectric conversion elements of the multiple pixels.
12. An imaging device comprising: an information processing device according to any one of claims 1 to 11; and an imaging element.
13. An information processing method comprising the steps of: acquiring an input image obtained by capturing an image of a subject and a background image displayed on a background display using an imaging device; determining a subject area and a background image area in the input image; acquiring positional information regarding at least one of the angle and distance between the background display and the imaging device; and performing image processing on the background image area in accordance with the positional information.
14. A program for causing a computer to execute processing according to the information processing method set forth in claim 13.
Citation Information
Patent Citations
Image processor, image processing method, and image processing program
JP2010067223A
Imaging apparatus
JP2011028177A
Imaging apparatus, imaging method, program and recording medium
JP2015033020A
Image processing apparatus, image processing method, and program
JP2022140057A
Digital photographing apparatus and method of operating the same
US20170150067A1
