Image processing device, image processing method, and program

Abstract

To provide an image processing device capable of selectively outputting a result of image processing.SOLUTION: An image processing device comprises: image acquisition means which acquires images of multiple view points different from one another; computing means which performs an operation for converting the acquired image to distance information; multiple output means which are capable of outputting the distance information; and control means which performs control of outputting an operation result by the computing means. The control means distributes multiple operation results with different operation amounts obtained in an operation process by the computing means to the multiple output means, and selectively switches an operation result to distribute, according to the performance situation of the image processing device.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Currently, one of the image compositing techniques commonly used in movies and television is chromakey compositing, which makes certain colored parts of an image transparent and composites a different image into them. When shooting an image for chromakey compositing, a specific color background such as a green or blue screen must be prepared, and work such as smoothing out wrinkles and adjusting the lighting must be done to prevent color unevenness in the background. In addition, if light reflected from the green screen is reflected onto the subject, it may be difficult to separate the background, so post-processing after shooting may be required to remove the reflected light. This series of work takes a great deal of time and manpower.

[0003] On the other hand, a recent image synthesis technique is to use information on the distance to the subject to separate the subject from the background and synthesize the subject with another image. This method eliminates the need to prepare a background such as a green screen, and also eliminates the need for the complicated work required for chromakey synthesis mentioned above.

[0004] As a method for acquiring distance information to a subject, for example, there is a method for measuring the distance to a subject using a distance measuring device equipped with a distance sensor, separate from an image processing device for image capture. In this case, the image processing device and the distance measuring device have different angles of view and resolutions, and the optical axes of the image processing device and the distance measuring device do not coincide, so that a calibration work needs to be performed before capture in order to perform image synthesis in post-processing.

[0005] Meanwhile, as another method for acquiring distance information to a subject, for example, the technology of Patent Document 1 is known. Patent Document 1 discloses a technology for generating distance information indicating the distance from an image processing device to a subject by using an image sensor having a distance measuring function based on an image plane phase difference method. With this technology, image capture and distance information generation can be performed simultaneously with a single image sensor, making the above-mentioned calibration work unnecessary. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2021-048560 A Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, all the calculations are performed within the imaging device, and the distance information is provided to the user. However, generally, the amount of calculations required to perform all the calculations within the imaging device tends to be large. In particular, when applied to video processing, it may be impossible to achieve without reducing the resolution or the frame rate. On the other hand, in the current video production industry, there is a view that only the minimum role is assigned to the imaging device, the captured image is output outside the imaging device, and various correction processes are performed in the workflow of a subsequent video processing PC, etc. In view of this current situation, when operating as a system, it is desirable to be able to arbitrarily select how much is assigned to the imaging device and how much is assigned to the subsequent video processing workflow according to the needs and environment of the user.

[0008] The present invention has been made in consideration of the above circumstances, and has an object to provide an image processing device that can selectively output the results of image processing. [Means for solving the problem]

[0009] An image processing device according to an embodiment of the present invention includes: An image acquisition means for acquiring images from a plurality of different viewpoints; a lens unit having a lens for forming an optical image on the image acquisition means; A calculation means for performing a calculation to convert the acquired image into distance information; A plurality of output means capable of outputting the distance information; a control unit for controlling output of a calculation result by the calculation unit from the plurality of output units, The control means distributes a plurality of calculation results with different calculation amounts obtained in the calculation process by the calculation means to the plurality of output means, and selectively switches the calculation results to be distributed according to an operating status of the image processing device. , along with When the lens is operated by a user, the calculation results to be distributed are selectively switched. It is characterized by: Effect of the Invention

[0010] According to the present invention, it is possible to provide an image processing device that can selectively output the results of image processing. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a schematic block diagram illustrating an example of a hardware configuration of an image processing device. [Diagram 2] 2A is a diagram showing an example of color filters arranged on the light receiving surface of an image sensor, and FIG. 2B is a diagram showing an example in which two photoelectric conversion units (photodiodes) are arranged in each pixel in correspondence with the color filter arrangement in FIG. 2A. [Diagram 3] 11 is a flowchart illustrating a process of generating distance information. [Figure 4] FIG. 4 is a diagram showing the calculation result of equation (1) when correlation between a pair of image signal strings is high in an ideal state in which no noise is present. [Diagram 5] FIG. 11 is a diagram showing the calculation result when formula (1) is applied to a small block containing noise. [Figure 6] FIG. 2 is a diagram of a captured image in the first embodiment. [Figure 7] 4A to 4C are diagrams showing image A and B signals in the first embodiment. [Figure 8] 4A to 4C are diagrams of an image shift image, a defocused image, and a distance image in the first embodiment. [Figure 9] 4A and 4B are diagrams of a distance layer image in the first embodiment. [Figure 10] 13(A) to 13(D) are diagrams of output switching patterns in the first embodiment. [Figure 11] 2 is a flowchart according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] First Embodiment Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the following embodiments do not limit the invention according to the claims, and not all of the features described in the following embodiments are necessarily essential to the present invention. Note that images in the following embodiments include not only still images, but also images such as moving images.

[0013] First, a system on which the present invention is based will be described. Fig. 1 is a schematic block diagram showing an example of the hardware configuration of an image processing device 100. In the figure, the image processing device 100 is a device capable of inputting, outputting, and recording video.

[0014] 1, a CPU 102 as a computer, a ROM 103, a RAM 104, an image processing unit 105, a lens unit 106, an imaging unit 107, a network module 108, and an image output unit 109 are connected to an internal bus 101. A recording medium I / F (interface) 110, an object detection unit 115, etc. are also connected to the internal bus 101. The blocks connected to the internal bus 101 are configured to be able to exchange data with each other via the internal bus 101.

[0015] Some of the blocks shown in Fig. 1 are realized by causing a CPU, which serves as a computer included in the image processing device, to execute a computer program stored in a memory, such as a ROM, which serves as a storage medium. However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP), etc., may be used. Furthermore, each block shown in Fig. 1 does not have to be built into the same housing, and may be configured by separate devices connected to each other via signal paths.

[0016] The lens unit 106 is a unit consisting of a group of lenses including a zoom lens and a focus lens, an aperture mechanism, a drive motor, and the like. An optical image that passes through the lens unit 106 is formed on the light receiving surface of the imaging section 107. The imaging section 107 functions as an image acquisition means for acquiring an image including a subject, and includes an imaging element such as a CCD image sensor or a CMOS image sensor, and plays a role in converting an optical signal into an electrical signal. Since the electrical signal obtained here is an analog value, it also has a function of converting it into a digital value. The imaging element in this embodiment is a sensor having an image plane phase difference detection function, and the details of this will be described later.

[0017] The CPU 102 uses the RAM 104 as a work memory in accordance with a computer program stored in the ROM 103 to control each unit of the image processing device 100. The ROM 103 is a non-volatile semiconductor memory in which computer programs for operating the CPU 102, various adjustment parameters, etc. are recorded. The RAM 104 is a volatile memory that uses semiconductor elements, and generally has a lower speed and a smaller capacity than the frame memory 111.

[0018] The frame memory 111 is a semiconductor element that temporarily stores image signals and can read them out when necessary. Image signals are huge amounts of data, so a high-bandwidth, large-capacity memory is required. Here, a DDR4-SDRAM (Dual Data Rate4-Synchronous Dynamic RAM) or the like is used. By using this frame memory 111, it becomes possible to perform processes such as synthesizing images that differ over time and cutting out only the required area.

[0019] Under the control of the CPU 102, the image processing unit 105 performs various types of image processing on data from the imaging unit 107 or image data stored in the frame memory 111 or the recording medium 112. The image processing performed by the image processing unit 105 includes pixel interpolation, encoding, compression, decoding, enlargement / reduction (resizing), noise reduction, color conversion, and the like of the image data.

[0020] The image processing unit 105 also performs correction processes such as correcting variations in pixel characteristics of the imaging unit 107, correcting defective pixels, correcting white balance, correcting brightness, and correcting distortion and peripheral light loss caused by lens characteristics. The image processing unit 105 also generates distance information, the details of which will be described later. The image processing unit 105 may be configured with a dedicated circuit block for performing specific image processing. Depending on the type of image processing, the CPU 102 may also perform image processing according to a program without using the image processing unit 105.

[0021] Based on the calculation results obtained by the image processing unit 105, the CPU 102 controls the lens unit 106 to optically adjust the image enlargement, focal length, aperture that adjusts the amount of light, etc. Also, camera shake may be corrected by moving a part of the lens group within a plane perpendicular to the optical axis.

[0022] The operation unit 113 receives operations from the user as an interface with the outside of the device. The operation unit 113 is composed of elements such as mechanical buttons and switches, and includes a power switch, a mode change switch, and the like.

[0023] The display unit 114 has a function for displaying images, and is a display device that can be visually recognized by the user. For example, the image processed by the image processing unit 105, a setting menu, and the operating status of the image processing device 100 can be confirmed. The display unit 114 uses a small, low-power device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) as a display device. Furthermore, the display unit 114 may have a touch panel structure using a resistive or capacitive thin-film element and be used as a part of the operation unit 113.

[0024] The CPU 102 generates character strings for informing the user of the setting status of the image processing device 100 and a menu for setting the image processing device 100, and displays these on the display unit 114, superimposed on the image processed by the image processing unit 105. In addition to character information, shooting assist displays such as a histogram, vector scope, waveform monitor, zebra, peaking, false color, etc. can also be superimposed.

[0025] Another interface is an image output unit 109. Representative interfaces include SDI (Serial Digital Interface) and HDMI (High Definition Multimedia Interface), both of which are registered trademarks. Alternatively, an interface such as Display Port (registered trademark) may be used. A real-time image can be displayed on an external monitor or the like via the image output unit 109.

[0026] The camera also includes a network module 108 that can transmit not only images but also control signals. The network module 108 is an interface for inputting and outputting image and audio signals. The network module 108 can also communicate with external devices via the Internet or the like, and transmit and receive various types of data such as files and commands. The network module 108 may be wireless or wired.

[0027] The image processing device 100 has a function of not only outputting images to the outside, but also recording images within the main body. The recording medium 112 is capable of recording image data and various setting data, and a large-capacity storage element is used. For example, there is a hard disc drive (HDD) or a solid state drive (SSD), which can be attached to the recording medium I / F 110.

[0028] The object detection unit 115 is a block for detecting objects, and performs object detection using artificial intelligence, such as deep learning using a neural network. When performing object detection using deep learning, the CPU 102 transmits to the object detection unit 115 a program for processing stored in the ROM 103, a network structure such as SSD or YOLO, weight parameters, and the like.

[0029] Note that SSD is an abbreviation for Single Shot Multibox Detector, and YOLO is an abbreviation for You Only Look Once. The object detection unit 115 performs processing for detecting an object from an image signal based on various parameters obtained from the CPU 102, and loads the processing result in the RAM 104.

[0030] Fig. 2(A) is a diagram showing an example of color filters arranged on the light receiving surface of an image sensor. Fig. 2(A) shows an example of a Bayer array of red (R), blue (B), and green (Gb, Gr), in which multiple pixels are arranged two-dimensionally on the image sensor, and a color filter of R, B, Gb, or Gr is arranged in front of each pixel as shown in Fig. 2(A). Although Fig. 2(A) shows only two rows of a color filter array, this color filter array is repeatedly arranged in two rows in the vertical scanning direction.

[0031] In addition, a microlens is arranged in front of the color filter arranged in front of each pixel of the image sensor, and each pixel has two photoelectric conversion units (photodiode A, photodiode B) arranged side by side in the horizontal scanning direction.

[0032] Fig. 2(B) is a diagram showing an example in which two photoelectric conversion units (photodiodes) are arranged in each pixel in correspondence with the color filter arrangement in Fig. 2(A). In Fig. 2(B), each pixel is composed of a pair of photodiode A and photodiode B, and color filters of the same color are arranged for the two photodiodes in the pair. Note that photodiode A and photodiode B each receive light beams from different exit pupils of the optical system via a microlens.

[0033] In the image sensor of this embodiment, an A image signal can be obtained from a plurality of photodiodes A of pixels arranged in the row direction. Similarly, a B image signal can be obtained from a plurality of photodiodes B of pixels arranged in the row direction. These A and B image signals are processed as signals for phase difference detection.

[0034] That is, for example, CPU 102 or image processing unit 105 performs a correlation calculation between the A and B image signals, detects the phase difference between the A and B image signals, and calculates the subject distance based on the phase difference. That is, CPU 102 and image processing unit 105 function as distance information acquisition means that acquires distance information indicating the distance to the subject.

[0035] It is also possible to obtain an image signal (A image signal+B image signal) by adding together the signals from the two photodiodes A and B of each pixel. This added image signal is processed by the image processing unit 105 as a color image signal according to the Bayer array shown in FIG. 2(A).

[0036] In the imaging unit 107, it is possible to output a phase difference detection signal (A image signal, B image signal) for each pixel, but it is also possible to average the A image signals of multiple adjacent pixels and output an average value of the B image signals of multiple adjacent pixels. Outputting the average value makes it possible to shorten the time required to read out signals from the imaging unit 107 and reduce the bandwidth of the internal bus 101.

[0037] Using the output signal from the imaging unit 107 having such an image sensor, the CPU 102 and image processing unit 105 perform correlation calculations of the two image signals and calculate information such as the defocus amount, parallax information, and various types of reliability based on the phase difference between the two image signals. The defocus amount on the light receiving surface is calculated based on the deviation (phase difference) between the A and B image signals. The defocus amount has positive and negative values, and whether the focus is front or back can be determined depending on whether the defocus amount is positive or negative.

[0038] The absolute value of the defocus amount indicates the degree of focus, and if the defocus amount is 0, the focus is achieved. That is, the CPU 102 calculates information on whether the focus is front or back based on the positive or negative defocus amount, and calculates focus degree information, which is the degree of focus (amount of focus deviation), based on the absolute value of the defocus amount. The information on whether the focus is front or back is output when the defocus amount exceeds a predetermined value, and outputs information indicating that the focus is achieved when the absolute value of the defocus amount is within the predetermined value.

[0039] The CPU 102 performs focus adjustment by controlling the lens unit 106 in accordance with the defocus amount. The CPU 102 also calculates the distance to the subject from the phase difference information and lens information of the lens unit 106 using the principle of triangulation.

[0040] 2, an example has been described in which pixels, each having two photodiodes as photoelectric conversion units arranged for one microlens, are arranged in an array. However, each pixel may have three or more photodiodes as photoelectric conversion units arranged for one microlens. Also, all pixels do not have to have the above configuration. For example, pixels for distance detection may be discretely arranged among a plurality of pixels for image detection arranged in a two-dimensional shape.

[0041] In this case, the pixel for distance detection may have the above-mentioned two photodiodes, or each pixel for distance detection may have only one of the photodiode A and the photodiode B. When only one of the photodiode A and the photodiode B is included, the photodiode A and the photodiode B are arranged so that images of different exit pupils of the lens unit are incident thereon. Alternatively, one of the light beams is shielded. In this way, this embodiment is not limited to the above-mentioned pixel structure as long as it is configured to obtain two image signals capable of detecting the phase difference, such as an image signal A and an image signal B. In addition, the imaging unit 107 may be a so-called stereo camera consisting of two imaging elements having parallax.

[0042] Next, the process of generating distance information will be described with reference to Fig. 3 to Fig. 5. Fig. 3 is a flowchart for explaining the process of generating distance information. Note that the operation of each step in the flowchart of Fig. 3 is performed by CPU 102 as a computer executing a computer program stored in ROM 103 as a storage medium.

[0043] 3, first, two signals, an (A image signal+B image signal) for imaging and an A image signal for phase difference detection, are obtained by reading them from the imaging unit 107. Next, in step S301, the image processing unit 105 calculates the B image signal for phase difference detection by finding the difference between the (A image signal+B image signal) and the A image signal.

[0044] In the above steps S300 and S301, an example has been described in which the B signal is calculated by reading out (A image signal+B image signal) and the A image signal and calculating the difference. However, the A image signal and the B image signal may be read out from the imaging unit 107. In addition, in a case where two image sensors are provided, such as a stereo camera, the image signals output from the respective image sensors may also be processed as the A image signal and the B image signal.

[0045] In step S302, optical shading correction is performed on each of the A image signal for phase difference detection and the B image signal for phase difference detection. In step S303, filter processing is performed on each of the A image signal for phase difference detection and the B image signal for phase difference detection. For example, low frequencies are cut using a high-pass filter configured with FIR. Note that the signals may also be passed through a band-pass filter or a low-pass filter with different filter coefficients. FIR stands for finite impulse response.

[0046] Next, in step S304, the A image signal for phase difference detection and the B image signal for phase difference detection that have been subjected to the filter processing in step S303 are divided into minute blocks and correlation calculations are performed. Note that there is no restriction on the size or shape of the minute blocks, and adjacent blocks may have overlapping areas.

[0047] The correlation calculation for the A and B image signals, which are a pair of images, will be explained below. The signal sequence of the A image signal at a pixel position of interest will be denoted as E(1) to E(m), and the signal sequence of the B image signal at a pixel position of interest will be denoted as F(1) to F(m). While shifting the signal sequence of the B image signal F(1) to F(m) relative to the signal sequence of the A image signal E(1) to E(m), the correlation amount C(k) at the shift amount k between the two signal sequences is calculated using equation (1).

[0048]

number

[0049] Fig. 4 is a diagram showing the calculation result of equation (1) when the correlation between a pair of image signal sequences is high in an ideal state where there is no noise. As shown in Fig. 4, at a shift amount (k = kj = 0) where the correlation between a pair of image signal sequences is high, the correlation amount C(k), which is the difference, is minimum. Hereinafter, k when the discrete correlation amount C(k) is minimum is represented as kj. By the three-point interpolation process shown in equations (2) to (4), x that gives the minimum value C(x) for the continuous correlation amount is calculated. Note that the pixel shift amount x is a real value and its unit is pixel.

[0050]

number

number

number

[0051] The SLOP in equation (4) represents the minimum and local minimum correlation amount and the slope of the change in the adjacent correlation amount. C(kj) = C(0) = 1000 C(kj-1)=C(-1)=1700 C(kj+1)=C(1)=1830 Let us assume that.

[0052] In this example, kj = 0. From equations (2) to (4), SLOP=830 x=-0.078pixel In addition, in the case of a focused state, the pixel shift amount x for the signal sequence of the image A and the signal sequence of the image B is ideally 0.00.

[0053] On the other hand, Fig. 5 is a diagram showing the calculation results when formula (1) is applied to a small block containing noise. As shown in Fig. 5, the correlation between the signal sequence of image A and the signal sequence of image B decreases due to the influence of randomly distributed noise. The minimum value of the correlation amount C(k) is larger than the minimum value shown in Fig. 4, and the correlation amount curve has a generally flat shape (a shape in which the absolute difference between the maximum and minimum values ​​is small).

[0054] In FIG. 5, as a specific example, C(kj) = C(0) = 1300 C(kj-1)=C(-1)=1480 C(kj+1)=C(1)=1800 Let us assume that.

[0055] In this example, kj = 0. From equations (2) to (4), SLOP=500 x=-0.32pixel That is, compared to the calculation result in the absence of noise shown in FIG.

[0056] When the correlation between a pair of image signal series is low, the amount of change in the correlation amount C(k) is small and the correlation amount curve is generally flat, so the SLOP value is small. Similarly, when the subject image has low contrast, the correlation between the pair of image signal series is low and the correlation amount curve is flat. Based on this property, the reliability of the calculated pixel shift amount x can be determined by the SLOP value. That is, when the SLOP value is large, it can be determined that the correlation between the pair of image signal series is high, and when the SLOP value is small, it can be determined that no significant correlation has been obtained between the pair of image signal series.

[0057] In this embodiment, since the formula (1) is used for the correlation calculation, the correlation amount C(k) is minimum and minimal at the amount of deviation where the correlation between the pair of image signal sequences is highest. However, a correlation calculation method may be used where the correlation amount C(k) is maximum and maximal at the amount of deviation where the correlation between the pair of image signal sequences is highest.

[0058] Next, in step S305, the reliability is calculated. As described above, the reliability can be calculated based on C(kj), which indicates the degree of match between the two images calculated in step S304, and the value of SLOP. Next, in step S306, an interpolation process is performed. Even if the correlation calculation was performed in step S304, there are cases in which the reliability calculated in step S305 is low and cannot be used as the pixel shift amount.

[0059] In that case, the pixel shift amount is interpolated from the pixel shift amount calculated for the surrounding pixels. The interpolation method may be a median filter, or a calculation may be performed to reduce the pixel shift amount data and then enlarge it again. Also, color data may be extracted from the (A image signal + B image signal) used for imaging, and the pixel shift amount may be interpolated using the color data.

[0060] Next, in step S307, the defocus amount is calculated by referring to the amount x calculated in step S304. Specifically, the defocus amount (denoted as DEF) can be obtained by the following formula (5).

number

[0061] Next, in step S308, the distance is calculated from the defocus amount calculated in step S307. When the distance to the subject is Da, the focal position is Db, and the focal length is F, the following formula (6) approximately holds.

[0062]

number

number

[0063] Therefore, if Db when DEF=0 is Db0, equation (7) becomes the following equation (8):

number

[0064] On the other hand, the relative distance is Da-Da', and can be calculated from equations (7) and (8) using the following equation (9).

number

[0065] As described above, by performing correlation calculations according to the flowchart in Fig. 3, it is possible to calculate the pixel shift amount, defocus amount, and distance information from the A image signal for phase difference detection and the B image signal for phase difference detection. That is, it is possible to obtain distance information based on the phase difference between the outputs of multiple photoelectric conversion units. Note that the distance information may be the distance data itself, or may be the shift amount or defocus amount, or may include these.

[0066] Up to this point, the calculation of distance information using the A image signal for phase difference detection and the B image signal for phase difference detection has been described. Hereinafter, in order to explain this embodiment, the information generated in each process will be defined as follows. The captured image is (A image signal + B image signal) for imaging. The AB independent image is a video signal that is not particularly corrected in the A image signal for phase difference detection and the B image signal for phase difference detection. The image shift image is information obtained by performing various corrections on the AB independent images and imaging the image shift amount obtained by formula (1) in two dimensions. The defocus image is information obtained by multiplying the image shift image by the pixel pitch and imaging the defocus amount in two dimensions. The distance image is distance information obtained by adding information on the imaging optical system, such as lens position information, to the defocus image. The distance layer image is information expressed like a layer by setting multiple thresholds for the numerical information of the distance image and rounding off the numerical values ​​near the thresholds to the thresholds. In the image processing device 100, the CPU 102 functions as a calculation means and a control means, and realizes these calculations and image generation by controlling the image processing unit 105.

[0067] In the image processing device 100 of this embodiment, the imaging unit 107 acquires images from a plurality of different viewpoints, and the CPU 102 performs a calculation to convert the images acquired by the imaging unit 107 into distance information. The image processing device 100 also has an image output unit 109 composed of a plurality of output means capable of outputting the acquired images and distance information. The CPU 102 performs the calculation and also controls the output of the calculation results from the image output unit 109.

[0068] 6 is a diagram of a captured image in the first embodiment. The captured image in this embodiment will be described by taking as an example a landscape image in which trees stand from the foreground to the background. For example, it is assumed that the focus is on the second tree 1003 from the front, and the foreground tree 1001 and the background tree 1002 are slightly out of focus.

[0069] Fig. 7 is a diagram of the A image signal and the B image signal in the first embodiment. When the above-mentioned landscape painting is displayed as A and B independent images, it can be shown as in Fig. 7. Comparing the two images, the second tree 1003 from the front is in focus, so the A image signal and the B image signal are displayed at approximately the same position in the image. The frontmost tree 1001 and the backmost tree 1002 are out of focus, so there is a misalignment due to parallax between the A image signal for phase difference detection and the B image signal for phase difference detection.

[0070] FIG. 8 is a diagram of an image shift image, a defocus image, and a distance image in the first embodiment. When the image shift image after the correlation calculation is visualized, it can be displayed as shown in FIG. 8. In the example of FIG. 8, the image shift amount (unit Pixel) is expressed by brightness, and gray is the center value with the image shift amount being zero. Therefore, the brighter or darker it is, the larger the image shift amount is indicated. In this example, the area where the image shift amount cannot be detected is expressed in black. By multiplying the image shift image by the pixel pitch, the unit can be converted from Pixel to mm, and it can be treated as a defocus image. It can be displayed in the same way as FIG. 8. A distance image can be generated by adding information about the imaging optical system, such as the position information of the lens, to the defocus amount. This can also be displayed in the same way as FIG. 8. In addition to brightness information, FIG. 8 can also be displayed by changing the color, such as a heat map.

[0071] 9(A)-(B) are diagrams of distance layer images in the first embodiment. The distance layer image shown in FIG. 9 is displayed by adding the concept of layers to the distance image and applying subjects within a certain distance range to a single layer. Generally, a photographed subject has unevenness, so when it is difficult to distinguish from a distance image, displaying a distance layer image makes it easier to understand. FIG. 9(A) is an example of a distance layer image displayed using brightness information, and FIG. 9(B) is an example of a distance layer image displayed using colors and markers.

[0072] So far, we have explained how to visualize information in each process until distance information is acquired. Next, we will explain the image processing device 100 as shown in FIG. 1 as a camera device. First, we will focus on the display unit 114, the image output unit 109, and the network module 108. The display unit 114 functions as a display means for displaying various calculation results, and is a display unit directly equipped on the image processing device 100. The display unit 114 is also called an EVF (Electric View Finder), and is an eyepiece monitor on which a cameraman constantly checks a captured image. Character information called an OSD (On Screen Display) is superimposed on the captured image. In general, the display on the display unit 114 is often displayed at a reduced size compared to the capture resolution of the imaging unit 107.

[0073] The image output unit 109 is connected to a monitor or the like installed at a distance by a cable, and displays the same captured image as the cameraman is looking at, and is used by the director of photography, the person who gives instructions for exposure adjustment, and the like. In general, the image processing device 100 is equipped with a plurality of output terminals as a plurality of output means of the image output unit 109. For example, the first SDI terminal transmits the captured image and OSD as with the EVF, and the second SDI terminal transmits the image for recording at the highest resolution without superimposing the OSD or the like. Other terminals such as HDMI are also provided, and are often used to transmit images for recording in the same manner. In some cases, the image processing device 100 may be equipped with an additional small sub-monitor or the like, and may be used for the cameraman to check the image as a display means separate from the display unit 114.

[0074] Although the network module 108 often has a large transmission delay and lacks real-time performance, it is used for transmitting images to distant locations and receiving control signals. Networks include Ethernet cable connections and wireless connections called Wi-Fi (registered trademark). When using a network, images may be reduced in size and transmitted at a slower frame rate in order to reduce the data rate. These output means are controlled by the CPU 102 of the image processing device 100, but the manner in which images are processed and transmitted is not limited to the above.

[0075] In this embodiment, in order to improve the convenience when synthesizing the subject with another image, distance information to the subject and information obtained during the process are output outside the device. In particular, during shooting, the various output means and the images displayed thereon change in use depending on the user and the device connected to the latter stage. Therefore, whether to distribute various images to the output means is determined according to the operation settings of the image processing device 100. The output means here are the above-mentioned EVF, SDI, HDMI, and network terminal, and the various images are the above-mentioned shot image, AB independent image, image shift image, defocus image, distance image, and distance layer image. The calculation results related to these images are images of multiple different viewpoints, parallax information, image shift information, defocus information, distance information, and distance layer information.

[0076] 10A to 10D are diagrams of output switching patterns in the first embodiment. As an example, how to distribute the images to be output is managed in a table such as that shown in FIG. 10 according to the operation settings of the image processing device 100.

[0077] Fig. 10(A) is an example of a table for switching the output method according to the calculation mode. By using Fig. 10(A), it is possible to selectively switch the calculation results to be distributed according to the set value of the amount of calculation set by the user. The high load mode is a mode in which the image processing device 100 is set to perform all calculations up to the calculation of distance information. The EVF displays the captured image so that the cameraman can check it, the SDI-1 outputs a distance image for external recording or input to an external PC, and the SDI-2 outputs the captured image for external recording or input to an external PC. The HDMI outputs a distance layer image so that the sense of distance to the subject can be checked while viewing it.

[0078] The low-load mode is a mode in which the image processing device 100 is set to execute only the minimum amount of calculation processing. As described above, various calculation processes are required before distance information is calculated. For this reason, when a highly accurate result is required, it becomes necessary to increase the number of calculation types in the image processing unit 105 or to use additional information from the object detection unit 115 or the like. In this case, unless the processed image is reduced in size or the frame rate is reduced, video shooting may be hindered. For this reason, the low-load mode can also be selected, and it is possible to switch to an output that assumes that only AB independent images with a small amount of calculation are output and connected to an external PC or the like, and image processing and distance information calculation calculation are performed by the PC.

[0079] For ease of explanation, only two types of modes, high load mode and low load mode, have been described here, but the method of reducing the amount of calculation is not limited to these. For example, various ideas can be considered, such as reducing the relative shift amount of the correlation calculation or omitting auxiliary calculations that increase reliability. In addition, although the high load mode places a high load on the image processing device 100, it is possible to obtain useful information from other installed functional blocks, and therefore perform calculations with higher accuracy than an external PC.

[0080] FIG. 10B is an example of a table for switching the output method depending on whether the image processing device 100 is in a non-recording state or a recording state. As shown in FIG. 10B, the output method may be switched depending on the operating status of the image processing device 100. By using FIG. 10B, it is possible to selectively switch the calculation results to be distributed depending on whether the calculation results are to be saved (recorded). Non-recording is a so-called standby state, which is a state in which actual shooting (recording) has not yet been performed, such as setting the parameters of the device, checking the composition to be shot, and rehearsing. This can be applied to cases where monitoring is performed using a distance layer image with good visibility during non-recording, and distance images with a large amount of information are saved in an external recording device during recording. When a high-definition monitor is attached to the external recording device, the convenience of such an output method is improved.

[0081] FIG. 10(C) is an example of a table that switches depending on whether the attached lens can be recognized (supported) by the image processing device 100. As shown in FIG. 10(C), the output method table may be switched depending on the lens unit 106 of the image processing device 100. If the type of the attached lens can be recognized, the optical parameters stored in the ROM 103 in advance can be used, so that the generation of a distance image is possible. However, if there is no lens information, it becomes difficult to generate a distance image, and only AB independent images can be supported. Even if optical parameters are available, it may be difficult to generate a distance image depending on the type of lens, such as a super telephoto lens. For this reason, a structure that imposes restrictions as necessary may be used. In this way, by using FIG. 10(C), it is possible to control the selection of a calculation result that can be output depending on the lens characteristics or the subject situation. Here, the lens characteristics or the subject situation include the type of lens, the F-number state, the subject contrast, and the like.

[0082] FIG. 10(D) is an example of a table that switches the output method depending on whether the attached lens is being operated (operated) or not (non-operated). By using FIG. 10(D), it is possible to selectively switch the calculation results to be distributed when the lens is being operated. It is mainly the cameraman who operates the lens, and in addition to adjusting the angle of view, the cameraman also performs focusing (focusing). At this time, it is easier to adjust the focus on a defocused image than on a captured image. In this way, it is also possible to switch the output method table depending on the state of the lens.

[0083] Although not described, a network terminal can also be used as the output terminal for the image signal. The user may set these tables as desired, and they may be configured to switch under conditions other than those described above. Although not described, the CPU 102 may selectively switch the calculation results to be distributed according to the readout mode in the sensor. The above describes switching the output method according to various settings and operation modes.

[0084] Fig. 11 is a flowchart in the first embodiment. The control of switching the output will be described with reference to Fig. 11. The operation of each step in the flowchart in Fig. 11 is performed by the CPU 102 executing a program stored in the ROM 103 or the like serving as a storage medium.

[0085] In step S1501, the CPU 102 of the image processing device 100 communicates with the lens unit 106. Here, it is checked whether the lens attached to the lens unit 106 is the same as the lens stored in the ROM 103. If communication is not possible or if a non-stored, incompatible lens is attached, the process proceeds to step S1502. On the other hand, if communication is possible and it is determined that the lens is a compatible lens in step S1501, the process proceeds to step S1503.

[0086] In step S1502, the CPU 102 reads out a setting table for the non-compatible lens terminal output from the ROM 103. Then, the CPU 102 performs control for switching the operations of the image processing unit 105 and the image output unit 109.

[0087] In step S1503, the CPU 102 reads out a setting table of the initial terminal output from the ROM 103, and performs control to switch the operations of the image processing unit 105 and the image output unit 109. Then, the process proceeds to step S1504.

[0088] In step S1504, it is determined whether the calculation mode is set to high load or low load. If the low load mode is selected to reduce the processing load of the image processing device 100, the process proceeds to step S1505. On the other hand, if the high load mode is selected in step S1504, the process proceeds to step S1506.

[0089] In step S 1505 , the CPU 102 reads out a table of low-load calculation terminal outputs from the ROM 103 , and performs control to switch the operations of the image processing unit 105 and the image output unit 109 .

[0090] In step S1506, the CPU 102 reads out a setting table for non-recording terminal output from the ROM 103, and performs control for switching the operations of the image processing unit 105 and the image output unit 109.

[0091] In step S1507, it is determined whether recording has started. If recording has not started yet, the process proceeds to step S1508. On the other hand, if it is determined that recording has started, the process proceeds to step S1511.

[0092] In step S1508, lens operation is detected. If the lens has not been operated, the process returns to step S1506. If the lens has been operated, the process proceeds to step S1509.

[0093] In step S1509, the CPU 102 reads out a setting table for the lens operation terminal output from the ROM 103, and performs control for switching the operations of the image processing unit 105 and the image output unit 109. After that, the process proceeds to step S1510.

[0094] In step S1510, it is determined whether the lens operation is continuing or has ended. If it is continuing, the process returns to step S1509. On the other hand, if it has ended, the process returns to step S1506.

[0095] In step S 1511 , the CPU 102 reads out a setting table for terminal output during recording from the ROM 103 , and performs control for switching the operations of the image processing unit 105 and the image output unit 109 .

[0096] In this embodiment, the CPU 102 distributes a plurality of calculation results with different amounts of calculation obtained during the calculation process to a plurality of output means, and selectively switches the calculation results to be distributed according to the operating status of the image processing device 100. Here, the plurality of calculation results include an initial result, an intermediate result, and a final result. The plurality of calculation results can be stored in the recording medium 112, and can be recorded in different areas. For this reason, for example, the area for storing the initial results may be the first storage means, the area for storing the intermediate results may be the second storage means, and the area for storing the final results may be the third storage means.

[0097] The initial result is images from at least a plurality of viewpoints that are different from each other. The intermediate result is information obtained in the process from when the CPU 102 acquires the disparity information to when it calculates the distance information. The final result is distance information. The image processing device 100 also has a conversion means for converting the distance information into distance layer information, and is capable of outputting the distance layer information from an output means. The CPU 102 can arbitrarily change the allocation destination of each of the calculation results. Here, the change in allocation destination may be performed based on a setting by the user.

[0098] In this embodiment, the CPU 102 may perform control not to perform auxiliary calculations to improve the reliability of the distance information, depending on the amount of calculations and the frame rate that are set. By generating the distance information and the distance layer information in this manner, it is possible to improve the frame rate while reducing the amount of calculations.

[0099] In this embodiment, the image processing device 100 may be battery-powered. Here, when the image processing device 100 is battery-powered, the display unit 114 may display that a calculation result with a smaller amount of calculation can be displayed. By reducing the amount of calculation, it is possible to reduce the power consumption of the battery.

[0100] <Second embodiment> A second embodiment of the present invention will be described. In this embodiment, the same components as those in the above-mentioned embodiment are given the same reference numerals and the description thereof will be omitted. In the first embodiment, it was described that the operation of the image processing unit 105 and the image output unit 109 is switched using a table according to the operation setting of the image processing device 100. In this embodiment, the concept of reliability is incorporated when adopting the table. This makes it possible to improve the convenience of the output method of the output terminal.

[0101] For example, suppose that CPU 102 judges values ​​such as SLOP described in Fig. 4 and Fig. 5 and judges that the reliability of the entire screen is low. In this case, if CPU 102 judges that the currently displayed image is a display of the calculation result such as a distance image, it displays a warning to prompt the user to switch the display method. Alternatively, CPU 102 performs processing such as forcibly switching to a captured image. The reason for this is that, for example, in Fig. 8, since black is displayed when there are many areas where calculation is not possible, if the reliability is low as a result, the image will be completely black or will be noisy. In this case, usability will be poor, so processing is performed to switch the display method.

[0102] The user may decide how to use the reliability. For example, if a user wants to take a picture with emphasis on people, the object detection unit 115 or the like is used to identify the area of ​​the person, and the reliability of the image is determined using the reliability within the area of ​​the person. If it is determined that the image has low reliability, a warning may be displayed or a forced switch may be performed.

[0103] The reliability information is obtained from an intermediate result among the multiple calculation results. The CPU 102 functions as a reliability information acquisition means for acquiring the reliability information from the intermediate result, and switches the calculation result to be distributed to the output means according to the reliability information. The reliability information is determined by the correlation between images from multiple different viewpoints.

[0104] As explained above, by switching the output method according to various settings and operation modes, it is possible to select the output method appropriate for the user, the environment, and the system being used. This makes it possible to efficiently carry out a series of tasks, such as separating the subject from the background and combining the subject with another image.

[0105] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0106] The present invention can also be realized by supplying a program for implementing one or more functions of the embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more functions. [Explanation of symbols]

[0107] 100 Image processing device 102 CPU 105 Image processing section 107 Imaging unit 109 Image output section 112 Recording media 114 Display section 1001 Foreground subject out of focus 1002 Out of focus subject in the background 1003 Subject in focus

Claims

1. An image acquisition means for acquiring images from a plurality of different viewpoints; a lens unit having a lens for forming an optical image on the image acquisition means; A calculation means for performing a calculation to convert the acquired image into distance information; A plurality of output means capable of outputting the distance information; a control unit for controlling output of a calculation result by the calculation unit from the plurality of output units, The control means distributes a plurality of calculation results with different calculation amounts obtained in a calculation process by the calculation means to the plurality of output means, and selectively switches the calculation results to be distributed according to an operating status of the image processing device; When the lens is operated by a user, the calculation results to be distributed are selectively switched.

13. An image processing device comprising:

2. The plurality of operation results include an initial result, an intermediate result, and a final result.

2. The image processing device according to claim 1,

3. The control means controls the allocation of the plurality of calculation results to the plurality of output means based on a setting made by a user.

3. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing device.

4. The control means performs control to selectively switch the calculation results to be distributed based on a predetermined table when the lens is operated by a user.

4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

5. An image acquisition means for acquiring images from a plurality of different viewpoints; a lens unit having a lens for forming an optical image on the image acquisition means; A calculation means for performing a calculation to convert the acquired image into distance information; A plurality of output means capable of outputting the distance information; a control unit for controlling output of a calculation result by the calculation unit from the plurality of output units, The control means distributes a plurality of calculation results with different calculation amounts obtained in the calculation process by the calculation means to the plurality of output means, and selectively switches the calculation results to be distributed according to an operating status of the image processing device. The control means selects an outputtable calculation result from among the plurality of calculation results in accordance with the lens characteristics of the lens or the subject situation by using a predetermined table.

13. An image processing device comprising:

6. The control means selectively switches the calculation results to be distributed in accordance with a setting value of the amount of calculation of the calculation means set by a user.

6. The image processing device according to claim 1,

7. A storage means for storing the plurality of calculation results, The control means selectively switches the calculation results to be allocated depending on whether or not the plurality of calculation results are to be stored.

7. The image processing device according to claim 1,

8. The control means selectively switches the calculation results to be distributed according to a read mode in the sensor.

8. The image processing device according to claim 1,

9. The control means controls the calculation means not to perform auxiliary calculation for increasing the reliability of the distance information in accordance with the amount of calculation and the frame rate set in the calculation means.

9. The image processing device according to claim 1,

10. A display means for displaying the plurality of calculation results, When the image processing device is battery-powered, the control means displays on the display means that a calculation result with a smaller amount of calculation can be displayed.

10. The image processing device according to claim 1,

11. The initial result is images of at least a plurality of different viewpoints.

3. The image processing device according to claim 2.

12. The intermediate result is information obtained in the process in which the calculation means acquires the parallax information and calculates the distance information.

3. The image processing device according to claim 2.

13. The final result is the distance information.

3. The image processing device according to claim 2.

14. A conversion means for converting the distance information into distance layer information.

14. The image processing device according to claim 1,

15. The control means can arbitrarily change the allocation destination of each of the calculation results.

15. The image processing device according to claim 1,

16. An image acquisition means for acquiring images from a plurality of different viewpoints; a lens unit having a lens for forming an optical image on the image acquisition means; A calculation means for performing a calculation to convert the acquired image into distance information; A plurality of output means capable of outputting the distance information; a control means for controlling output of the calculation results by the calculation means from the plurality of output means; a reliability information acquiring means for acquiring reliability information from an intermediate result obtained in a calculation process by the calculation means; In an image processing device comprising: The control means distributes a plurality of calculation results with different calculation amounts obtained in a calculation process by the calculation means to the plurality of output means, and selectively switches the calculation results to be distributed according to an operating status of the image processing device; Switching the calculation results to be distributed according to the reliability information; The plurality of operation results include an initial result, the intermediate result, and a final result.

13. An image processing device comprising:

17. 17. The image processing apparatus according to claim 16, wherein the reliability information is determined based on a correlation between images taken from a plurality of different viewpoints.

18. 1. An image processing method for an image processing device having a plurality of output means capable of outputting distance information, comprising: an image acquisition step of acquiring images from a plurality of different viewpoints via a lens that forms an optical image; a calculation step of performing a calculation to convert the acquired image into distance information; an output step of allocating a plurality of calculation results with different calculation amounts obtained in the calculation process to the plurality of output means, and outputting the calculation results from the plurality of output means by selectively switching the calculation results to be allocated according to the operating status of the image processing device; When the lens is operated by a user, the calculation results to be distributed are selectively switched.

13. An image processing method comprising:

19. A program for controlling each unit of the image processing apparatus according to any one of claims 1 to 17 by a computer.

Images