Image processing device, image processing method, and computer program

Abstract

To provide an image processing device suitable for outputting distance information, etc. externally that can be used for image composition, etc.SOLUTION: The image processing device has image acquisition means acquiring via a lens unit an image including a subject, distance information acquisition means acquiring distance information indicating distance to the subject, auxiliary data generation means generating auxiliary data related to the distance information, data stream generation means generating a data stream in which the image, the distance information, and the auxiliary data are superimposed, and output means for outputting the data stream externally.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a computer program, and the like. [Background technology]

[0002] Currently, one of the image synthesis techniques commonly used in movies and television is chromakey synthesis, which makes certain colored parts of an image transparent and synthesizes a different image there. When shooting an image for chromakey synthesis, it is necessary to prepare a background of a specific color such as a green or blue screen, and work such as smoothing out wrinkles and adjusting the lighting to prevent color unevenness in the background is also required. Also, if reflected light from the green screen is reflected on the subject, it may be difficult to separate the background. Therefore, work such as removing the reflected light in post-processing after shooting may be required.

[0003] On the other hand, a recent image synthesis technique is to use information on the distance to the subject to separate the subject from the background and synthesize the subject with another image. This method eliminates the need to prepare a background such as a green screen, and also eliminates the need for the complicated work required for chromakey synthesis mentioned above.

[0004] As a method for creating distance information to a subject, for example, there is a method for measuring the distance to the subject using a device equipped with a distance sensor separate from the imaging device for image capture. In this case, since the angle of view and resolution differ for each device, a calibration work needs to be performed before shooting in order to perform image synthesis in post-processing.

[0005] Meanwhile, as described in Patent Document 1, a technology is disclosed that generates distance information indicating the distance from an imaging device to a subject by using an imaging element having a distance measuring function based on an imaging surface phase difference method. With this technology, image capture and distance information generation can be performed simultaneously with a single imaging element, making the above-mentioned calibration work unnecessary. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-48560 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in Patent Document 1, no consideration is given to how distance information should be output in order to perform image synthesis or the like in post-processing.

[0008] The present invention has been made in view of the above circumstances, and has an object to provide an image processing device suitable for outputting to the outside distance information and the like that can be used for image synthesis and the like. [Means for solving the problem]

[0009] In order to solve the above problems, the image processing device of the present invention comprises: an image capture means for capturing an image including a subject through a lens unit; a distance information acquisition means for acquiring distance information indicating a distance to the subject; an auxiliary data generating means for generating auxiliary data relating to the distance information; a data stream generating means for generating a data stream in which the image, the distance information, and the auxiliary data are superimposed; an output means for outputting the data stream to an outside; The present invention is characterized by having the following. Effect of the Invention

[0010] According to the present invention, it is possible to realize an image processing device suitable for outputting to the outside distance information and the like that can be used for image synthesis and the like. [Brief description of the drawings]

[0011] [Figure 1]1 is a block diagram of an image processing apparatus according to a first embodiment. [Diagram 2] 2(A) is a diagram showing an example of color filters arranged on the light receiving surface of an image sensor, and (B) is a diagram showing an example in which two photoelectric conversion units (photodiodes) are arranged in each pixel in correspondence with the color filter arrangement in FIG. 2(A). [Diagram 3] 11 is a flowchart illustrating a process of generating distance information according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing the calculation result of equation (1) when correlation between a pair of image signal strings is high in an ideal state in which no noise is present. [Diagram 5] FIG. 11 is a diagram showing the calculation result when formula (1) is applied to a small block containing noise. [Figure 6] FIG. 2 is a diagram illustrating a data structure of an SDI according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of superimposition of distance information in the first embodiment. [Figure 8] 4 is a diagram showing a packet structure of ancillary data superimposed on a blanking area in the first embodiment. FIG. [Figure 9] 11 is a flowchart of a stream generation process in the first embodiment. [Figure 10] 10 is a flowchart showing an example of a process for outputting distance information so that it is easily visible on a display in the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of superimposition of distance information in the second embodiment. [Figure 12] FIG. 11 is a diagram showing an example of display of distance information in the second embodiment. [Figure 13] FIG. 11 is a diagram showing another example of superimposing distance information in the second embodiment. [Figure 14] FIG. 13 is a diagram illustrating a data structure of an SDI according to the third embodiment. [Figure 15] 13 is a flowchart of a stream generation process according to the third embodiment. [Figure 16] FIG. 13 is a diagram showing an example of superimposition of distance information in the third embodiment. [Figure 17]13 is a flowchart illustrating an example of an output mode selection process for distance information in the fourth embodiment. [Figure 18] 13 is a flowchart illustrating an example of an output mode selection process for distance information in the fifth embodiment. [Figure 19] FIG. 13 is a diagram showing an example of superimposition of distance information in a high-precision distance information output mode in the fifth embodiment. [Figure 20] FIG. 13 is a diagram showing an example of superimposition of distance information in a grayscale output mode in the fifth embodiment. [Figure 21] FIG. 13 is a diagram showing an example of displaying distance information in the fifth embodiment. [Figure 22] FIG. 13 is a diagram showing an example of superimposing distance information in a heat map output mode in the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the present invention will be described by way of examples with reference to the drawings. However, the present invention is not limited to the following embodiments and examples. In each drawing, the same members or elements are given the same reference numbers, and duplicated descriptions are omitted or simplified.

[0013] In the embodiment, an example in which the image processing device is applied to a digital camera will be described. However, the image processing device also includes electronic devices having an imaging function, such as a network camera, a smartphone with a camera, a tablet computer with a camera, an in-vehicle camera, a drone camera, and a camera mounted on a robot. In the embodiment, the images include not only still images but also moving images and the like.

[0014] <Example 1> 1 is a block diagram of an image processing device according to Example 1. In the figure, an image processing device 100 is a device capable of inputting, outputting, and recording an image, and is configured as, for example, a digital camera.

[0015] 1, a CPU 102 as a computer, a ROM 103, a RAM 104, an image processing unit 105, a lens unit 106, an imaging unit 107, a network module 108, and an image output unit 109 are connected to an internal bus 101. A recording medium I / F (interface) 110, an object detection unit 115, etc. are also connected to the internal bus 101. The blocks connected to the internal bus 101 are configured to be able to exchange data with each other via the internal bus 101.

[0016] Some of the blocks shown in Fig. 1 are realized by causing a CPU, which is a computer included in the image processing device, to execute a computer program stored in a memory such as a ROM, which is a storage medium. However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP), etc., may be used. Moreover, each block shown in Fig. 1 does not have to be built into the same housing, and may be configured by separate devices connected to each other via signal paths.

[0017] The lens unit 106 is a unit consisting of a group of lenses including a zoom lens and a focus lens, an aperture mechanism, a drive motor, etc. An optical image that passes through the lens unit 106 is formed on the light receiving surface of an imaging section 107. The imaging section 107 functions as an image acquisition means for acquiring an image including a subject, and includes an imaging element such as a CCD image sensor or a CMOS image sensor. The optical image formed on the light receiving surface of the imaging element is converted into an imaging signal, which is further converted into a digital signal and output. The imaging element in this embodiment is a sensor having an image plane phase difference detection function, and details of which will be described later.

[0018] The CPU 102 as a computer controls each unit of the image processing device 100 using the RAM 104 as a work memory in accordance with a computer program stored in the ROM 103. Also, in accordance with the computer program stored in the ROM 103, the CPU 102 executes the processes shown in the flowcharts of Figures 3, 9, 10, 15, 17, and 18, which will be described later. The ROM 103 is a non-volatile semiconductor memory, and stores therein the computer program for operating the CPU 102, various adjustment parameters, and the like.

[0019] The RAM 104 is a volatile semiconductor memory, and generally has a lower speed and a smaller capacity than the frame memory 111. The frame memory 111 is a semiconductor memory that temporarily stores image signals and can read them out when necessary. Since image signals are a huge amount of data, a high-bandwidth and large-capacity memory is required. Here, a DDR4-SDRAM (Dual Data Rate4-Synchronous Dynamic RAM) or the like is used. By using this frame memory 111, it becomes possible to perform processes such as synthesizing images that differ over time and cutting out only the required area.

[0020] Under the control of the CPU 102, the image processing unit 105 performs various types of image processing on data from the imaging unit 107 or image data stored in the frame memory 111 or the recording medium 112. The image processing performed by the image processing unit 105 includes pixel interpolation, encoding, compression, decoding, enlargement / reduction (resizing), noise reduction, color conversion, and the like of the image data.

[0021] Furthermore, the image processing unit 105 performs correction processes such as correcting variations in pixel characteristics of the imaging unit 107, correcting defective pixels, correcting white balance, correcting brightness, and correcting distortion and peripheral light loss caused by lens characteristics. The image processing unit 105 also generates a distance map, the details of which will be described later. The image processing unit 105 may be configured with a dedicated circuit block for performing specific image processing. Depending on the type of image processing, the CPU 102 may perform image processing according to a program without using the image processing unit 105.

[0022] Based on the image processing results in the image processing unit 105, the CPU 102 controls the lens unit 106 and adjusts the aperture that optically adjusts the image enlargement, focal length, and light amount, etc. Camera shake may also be corrected by moving a part of the lens group in a plane perpendicular to the optical axis. Reference numeral 113 denotes an operation unit that receives user operations as an interface with the outside of the device, and is composed of elements such as mechanical buttons and switches, and includes a power switch, a mode change switch, etc.

[0023] Reference numeral 114 denotes a display unit for displaying an image, and for example, an image processed by the image processing unit 105, a setting menu, and the operating status of the image processing device 100 can be confirmed on the display unit. The display unit 114 uses a small, low-power device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) as a display device. Furthermore, the display unit 114 may have a touch panel structure using a resistive or capacitive thin-film element and be used as a part of the operation unit 113.

[0024] The CPU 102 generates character strings for informing the user of the setting status of the image processing device 100 and a menu for setting the image processing device 100, and displays these on the display unit 114, superimposed on the image processed by the image processing unit 105. In addition to character information, shooting assist displays such as a histogram, vector scope, waveform monitor, zebra, peaking, false color, etc. can also be superimposed.

[0025] Reference numeral 109 denotes an image output unit, and an interface therefor may be SDI (Serial Digital Interface) or HDMI (High Definition Multimedia Interface), or an interface such as Display Port (registered trademark) may also be used. A real-time image can be displayed on an external display device (display) via the image output unit 109.

[0026] Also provided is a network module 108 capable of transmitting not only images but also control signals. The network module 108 is an interface for inputting and outputting image signals and audio signals. The network module 108 can also communicate with external devices via the Internet or the like, and transmit and receive various types of data such as files and commands. The network module 108 may be wireless or wired.

[0027] The image processing device 100 has a function of not only outputting an image to the outside, but also recording the image within the main body. The recording medium 112 is a large-capacity storage element such as a hard disk drive (HDD) or a solid state drive (SSD) capable of recording image data and various setting data, and is mountable to the recording medium I / F 110.

[0028] The object detection unit 115 is a block for detecting objects, and performs object detection using artificial intelligence, such as deep learning using a neural network. When performing object detection using deep learning, the CPU 102 transmits to the object detection unit 115 a program for processing stored in the ROM 103, a network structure such as SSD or YOLO, weight parameters, and the like.

[0029] In addition, SSD is an abbreviation for Single Shot Multibox Detector, and YOLO is an abbreviation for You Only Look Once. The object detection unit 115 performs processing for detecting an object from an image signal based on various parameters obtained from the CPU 102, and loads the processing result in the RAM 104.

[0030] Fig. 2(A) is a diagram showing an example of color filters arranged on the light receiving surface of an image sensor. Fig. 2(A) shows an example of a Bayer array of red (R), blue (B), and green (Gb, Gr), in which multiple pixels are arranged two-dimensionally on the image sensor, and a color filter of R, B, Gb, or Gr is arranged in front of each pixel as shown in Fig. 2(A). Although Fig. 2(A) shows only two rows of a color filter array, this color filter array is repeatedly arranged in two rows in the vertical scanning direction.

[0031] In addition, a microlens is arranged in front of the color filter arranged in front of each pixel of the image sensor, and each pixel has two photoelectric conversion units (photodiode A, photodiode B) arranged side by side in the horizontal scanning direction.

[0032] Fig. 2(B) is a diagram showing an example in which two photoelectric conversion units (photodiodes) are arranged in each pixel in correspondence with the color filter arrangement in Fig. 2(A). In Fig. 2(B), each pixel is composed of a pair of photodiode A and photodiode B, and color filters of the same color are arranged for the two photodiodes in the pair. Note that photodiode A and photodiode B each receive light beams from different exit pupils of the optical system via a microlens.

[0033] In the image sensor of this embodiment, an A image signal can be obtained from a plurality of photodiodes A of pixels arranged in the row direction. Similarly, a B image signal can be obtained from a plurality of photodiodes B of pixels arranged in the row direction. These A and B image signals are processed as signals for phase difference detection.

[0034] That is, for example, CPU 102 or image processing unit 105 performs a correlation calculation between the A and B image signals, detects the phase difference between the A and B image signals, and calculates the subject distance based on the phase difference. That is, CPU 102 and image processing unit 105 function as distance information acquisition means for acquiring distance information indicating the distance to the subject.

[0035] Incidentally, an image signal (A image signal+B image signal) can also be obtained by adding up the signals from the two photodiodes A and B of each pixel, and this added image signal is processed by the image processing unit 105 as a color image signal according to the Bayer array shown in FIG. 2(A).

[0036] In the imaging unit 107, it is possible to output a phase difference detection signal (A image signal, B image signal) for each pixel, but it is also possible to average the A image signals of multiple adjacent pixels and output an average value of the B image signals of multiple adjacent pixels. Outputting the average value makes it possible to shorten the time required to read out signals from the imaging unit 107 and reduce the bandwidth of the internal bus 101.

[0037] Using the output signal from the imaging unit 107 having such an image sensor, the CPU 102 and image processing unit 105 perform correlation calculations of the two image signals and calculate information such as the defocus amount, parallax information, and various types of reliability based on the phase difference between the two image signals. The defocus amount on the light receiving surface is calculated based on the deviation (phase difference) between the A and B image signals. The defocus amount has positive and negative values, and whether the focus is front or back can be determined depending on whether the defocus amount is positive or negative.

[0038] The absolute value of the defocus amount indicates the degree of focus, and if the defocus amount is 0, the focus is achieved. That is, the CPU 102 calculates information on whether the focus is front or back based on the positive or negative defocus amount, and calculates focus degree information, which is the degree of focus (amount of focus deviation), based on the absolute value of the defocus amount. The information on whether the focus is front or back is output when the defocus amount exceeds a predetermined value, and outputs information indicating that the focus is achieved when the absolute value of the defocus amount is within the predetermined value.

[0039] The CPU 102 controls the lens unit 106 in accordance with the defocus amount to perform focus adjustment. Furthermore, the CPU 102 calculates the distance to the subject from the phase difference information and lens information of the lens unit 106 using the principle of triangulation.

[0040] 2, an example has been described in which pixels, each having two photodiodes as photoelectric conversion means arranged for one microlens, are arranged in an array. However, each pixel may have three or more photodiodes as photoelectric conversion means arranged for one microlens. Also, all pixels do not have to have the above configuration. For example, pixels for distance detection may be discretely arranged among a plurality of pixels for image detection arranged two-dimensionally.

[0041] In this case, the pixel for distance detection may have a structure having two photodiodes as described above, or each pixel for distance detection may have a structure having only one of photodiode A and photodiode B. When only one of photodiode A and photodiode B is provided, photodiode A and photodiode B are arranged so that images of different pupil regions (exit pupils) of the lens unit are incident thereon. Alternatively, one of the light beams is shielded. In this way, the imaging unit 107 of this embodiment is not limited to the pixel structure as described above as long as it is configured to obtain two image signals such as an A image signal and a B image signal, which are capable of detecting a phase difference. Also, the imaging unit 107 may be a so-called stereo camera consisting of two imaging elements having parallax.

[0042] Next, the process of generating distance information will be described with reference to FIGS. FIG. 3 is a flowchart illustrating a process of generating distance information according to the first embodiment. The operations of the steps in the flowchart of FIG. 3 are performed by a CPU 102 serving as a computer executing a computer program stored in a ROM 103 or the like serving as a storage medium.

[0043] In the flowchart of FIG. 3, first, in step S300, two signals, that is, (A image signal+B image signal) for imaging and A image signal for phase difference detection, are read out and acquired from the imaging unit 107. Next, in step S301, the image processing unit 105 calculates a B image signal for phase difference detection by determining the difference between (A image signal+B image signal) and the A image signal.

[0044] In the above steps S300 and S301, an example has been described in which the B signal is calculated by reading out (A image signal+B image signal) and the A image signal and calculating the difference. However, the A image signal and the B image signal may be read out from the imaging unit 107, respectively. In addition, in a case where two image sensors are provided, such as a stereo camera, the image signals output from the respective image sensors may also be processed as the A image signal and the B image signal.

[0045] In step S302, optical shading correction is performed on each of the A image signal for phase difference detection and the B image signal for phase difference detection. In step S303, a filter process is performed on each of the A image signal for phase difference detection and the B image signal for phase difference detection. For example, a high-pass filter configured with an FIR filter is used to cut low frequencies. Note that the signals may also be passed through a band-pass filter or a low-pass filter with different filter coefficients.

[0046] Next, in step S304, the A image signal for phase difference detection and the B image signal for phase difference detection that have been subjected to the filter processing in step S303 are divided into minute blocks, and correlation calculations are performed. Note that there is no restriction on the size or shape of the minute blocks, and adjacent blocks may have overlapping areas.

[0047] The correlation calculation for the A and B image signals, which are a pair of images, will be explained below. The signal sequence of the A image signal at a pixel position of interest will be denoted as E(1) to E(m), and the signal sequence of the B image signal at a pixel position of interest will be denoted as F(1) to F(m). While shifting the signal sequence of the B image signal F(1) to F(m) relative to the signal sequence of the A image signal E(1) to E(m), the correlation amount C(k) at the shift amount k between the two signal sequences is calculated using equation (1).

[0048]

number

[0049] FIG. 4 is a diagram showing the calculation result of equation (1) when the correlation between a pair of image signal strings is high in an ideal state in which no noise exists. As shown in Fig. 4, at a shift amount (k = kj = 0) where the correlation between a pair of image signal sequences is high, the correlation amount C(k), which is the difference, is minimum. Hereinafter, k when the discrete correlation amount C(k) is minimum will be denoted as kj. The shift amount x that gives the minimum value C(x) for the continuous correlation amount is calculated by the three-point interpolation process shown in equations (2) to (4). The shift amount x is a real value and its unit is pixels.

[0050]

number

number

number

[0051] The SLOP in equation (4) represents the minimum and local minimum correlation amount and the slope of the change in the adjacent correlation amount. C(kj) = C(0) = 1000 C(kj-1)=C(-1)=1700 C(kj+1)=C(1)=1830 Let us assume that.

[0052] In this example, kj = 0. From equations (2) to (4), SLOP=830 x=-0.078pixel It becomes. In addition, in the case of a focused state, the shift amount x for the signal sequence of the image A and the signal sequence of the image B is ideally 0.00.

[0053] On the other hand, FIG. 5 is a diagram showing the calculation results when formula (1) is applied to a small block containing noise. As shown in Fig. 5, the effect of randomly distributed noise reduces the correlation between the signal sequence of image A and the signal sequence of image B. The minimum value of the correlation amount C(k) becomes larger than the minimum value shown in Fig. 4, and the correlation amount curve has a generally flat shape (a shape in which the absolute difference between the maximum and minimum values ​​is small).

[0054] In FIG. 5, as a specific example, C(kj) = C(0) = 1300 C(kj-1)=C(-1)=1480 C(kj+1)=C(1)=1800 Let us assume that.

[0055] In this example, kj = 0. From equations (2) to (4), SLOP=500 x=-0.32pixel It becomes. That is, compared to the calculation result in the absence of noise shown in FIG. 4, the shift amount x is farther from the ideal value.

[0056] When the correlation between a pair of image signal series is low, the amount of change in the correlation amount C(k) is small and the correlation amount curve is generally flat, so the SLOP value is small. Similarly, when the subject image has low contrast, the correlation between the pair of image signal series is low and the correlation amount curve is flat. Based on this property, the reliability of the calculated shift amount x can be judged by the SLOP value. That is, when the SLOP value is large, it can be judged that the correlation between the pair of image signal series is high, and when the SLOP value is small, it can be judged that no significant correlation has been obtained between the pair of image signal series.

[0057] In this embodiment, since the formula (1) is used for the correlation calculation, the correlation amount C(k) is minimum and minimal at the amount of deviation where the correlation between the pair of image signal sequences is highest. However, a correlation calculation method may be used where the correlation amount C(k) is maximum and maximal at the amount of deviation where the correlation between the pair of image signal sequences is highest.

[0058] Next, in step S305, the reliability is calculated. As described above, the SLOP value calculated in step S304 is defined as the reliability. Next, an interpolation process is performed in step S306. Even if the correlation calculation was performed in step S304, the calculation in step S305 may not be reliable and therefore cannot be used as the shift amount.

[0059] In that case, the shift amount is interpolated from the shift amount calculated for the surrounding pixels. As a method of interpolation, a median filter may be applied, or a calculation may be performed to reduce the shift amount data and then enlarge it again. Also, color data may be extracted from the (A image signal + B image signal) used for imaging, and the shift amount may be interpolated using the color data.

[0060] Next, in step S307, the defocus amount is calculated by referring to the shift amount x calculated in step S304. Specifically, the defocus amount (denoted as DEF) can be obtained by the following formula (5).

number

[0061] Next, in step S308, the distance is calculated from the defocus amount calculated in step S307. When the distance to the subject is Da, the focal position is Db, and the focal length is F, the following formula (6) approximately holds.

[0062]

number

number

[0063] Therefore, if Db when DEF=0 is Db0, equation (7) becomes the following equation (8):

number

[0064] On the other hand, the relative distance is Da-Da', and can be calculated from equations (7) and (8) using the following equation (9).

number

[0065] As described above, by performing correlation calculations according to the flowchart of Fig. 3, distance information such as the shift amount, defocus amount, and distance data can be calculated from the A image signal for phase difference detection and the B image signal for phase difference detection. That is, distance information can be acquired based on the phase difference between the outputs of multiple photoelectric conversion units. Note that the distance information in this embodiment may be the distance data itself, or may be the shift amount or defocus amount, and the distance information includes these.

[0066] In this embodiment, the distance information calculated as described above is superimposed on auxiliary data related to the distance information to generate a data stream, which is output to the outside via the image output unit 109 as output means. This allows the synthesis of highly accurate CG (computer graphics) images to be performed accurately and efficiently by an external device. Details of the auxiliary data will be described later. In the following, in this embodiment, an example of a method for outputting the above-mentioned distance information and auxiliary data related to the distance information to the outside using the SDI transmission standard will be described. Note that, since the SDI transmission standard has prohibited codes, in this embodiment, a part of the distance information is converted and then superimposed so that it does not become a prohibited code.

[0067] First, the structure of an HD-SDI data stream when the frame rate is 29.97 fps, the horizontal resolution is 1920, and the vertical resolution is 1080 will be described with reference to Fig. 6. The HD-SDI data stream is defined in SMPTE STEST292-1.

[0068] 6 is a diagram for explaining the data structure of SDI in the first embodiment, in which a data stream for one line is divided into two data streams with 10 bits per word. One frame has 1125 lines of this data stream. The Y (luminance) stream and C (color) stream each have 10 bits per word and are made up of 2200 words.

[0069] From the 1920th word, an EAV (End of Active Video) identifier for identifying the boundary of the image signal is superimposed on the data, followed by an LN (Line Number) and a CRCC (Cycle Redundancy Check Code), which is data for checking transmission errors.

[0070] Then, a data area (hereinafter referred to as blanking area) where ancillary data may be superimposed continues with 268 words, after which an identifier SAV (Start of Active Video) is superimposed to identify the delimiter of the image signal, just like EAV. Then, 1920 words of Y data (luminance data) and C data (color data) are superimposed and transmitted. If the frame rate changes, the number of words per line changes, and so does the number of words in the blanking area.

[0071] Next, the process of superimposing distance information in this embodiment will be described with reference to FIG. FIG. 7 is a diagram showing an example of superimposing distance information in the first embodiment, and shows how distance information is superimposed for each word at the superimposing positions of the Y data and C data in FIG.

[0072] Fig. 7 shows an example in which high-precision distance information of a total of 18 bits, from D

[17] to D[9] as distance information to be superimposed on Y data, and from D[8] to D[0] as distance information to be superimposed on C data, is calculated for each pixel in step S308 in Fig. 3. In addition, the distance information is divided and superimposed on multiple data streams.

[0073] In the SDI transmission standard, 0, 1, 2, 3 and 1020, 1021, 1022, 1023 are prohibited codes. Therefore, as shown in Figure 7, the inverted value of D

[17] , "D

[17] ", is superimposed on the Y[9] position, and the values ​​of D

[17] to D[9] are superimposed on the Y[8] to Y[0] positions. In addition, the inverted value of D[8], "D[8]", is superimposed on the C[9] position, and the values ​​of D[8] to D[0] are superimposed on the C[8] to C[0] positions, so that the distance information is converted and superimposed so that it does not become a prohibited code.

[0074] Here, an example is shown in which the distance information is 18 bits high precision distance information, but the bit depth of the distance information may be 18 bits or less. In that case as well, the value to be superimposed at the Y[9] position is the inverted value of the value to be superimposed at the Y[8] position, and the value to be superimposed at the C[9] position is the inverted value of the value to be superimposed at the C[8] position. This allows the distance information to be superimposed without becoming a prohibited code.

[0075] As described above, in this embodiment, various auxiliary data are attached to the distance information as metadata and output. For example, in order to use the distance information with high accuracy when performing post-processing such as CG image synthesis in an external device, it is desirable to be able to acquire, as auxiliary data (ancillary data), various setting data and data related to the state of the image processing device. Here, the various setting data of the image processing device includes, for example, exposure setting information, and the data related to the state of the image processing device includes, for example, data on the inclination of the image processing device.

[0076] Furthermore, it is desirable for the auxiliary data to include lens information regarding the lens unit 106 (for example, at least one of the focal length, focus position, and aperture information of the lens unit 106). Furthermore, it is desirable for the auxiliary data to include data (e.g., a time code) for synchronizing the captured image with the distance information, and information indicating whether or not various correction processes have been performed before calculating the distance information.

[0077] It is also desirable to include data indicating attributes of the distance information. Here, the attributes of the distance information include information indicating whether the distance information is distance data itself, defocus information, or shift amount data. When defocus information is output as the distance information, information on the focal length of the lens unit is required as auxiliary data to calculate the subject distance from the defocus amount. When the shift amount is output as the distance information, information on the deviation amount k is also required as auxiliary data.

[0078] In addition, the attributes of the distance information include the number of effective bits (bit depth) of the distance information, the unit of the distance information, information indicating whether the distance information is an absolute distance or a relative distance, and if the distance information includes a decimal point, how many bits are below the decimal point. In addition, the attributes of the distance information include information indicating whether the distance information is signed or not if the distance information is a relative distance, information indicating how many centimeters 1 LSB of the distance information is, and information indicating the distance of the focus position if the distance is a relative distance and is not expressed in a signed form.

[0079] In addition, it is desirable for the auxiliary data to include information such as the values ​​of the closest distance (distance to the nearest subject) and the farthest distance (distance to the farthest subject), and the resolution at which distance measurement is possible. Furthermore, when the above-mentioned images, distance information, and auxiliary data are superimposed and output from the image processing device 100, the auxiliary data must include information regarding the data structure (data format) of the superimposed data stream.

[0080] In this embodiment, the superposition may be multiplexing (for example, inserting into a gap such as a blanking period) or mixing and encoding data, and the superposition in this embodiment includes both methods. Also, the data stream in this embodiment is not limited to a data stream such as video data consisting of multiple frames, and includes, for example, still image data of only one frame.

[0081] Therefore, in this embodiment, as described above, when the distance information is output, the above-mentioned various information (hereinafter, referred to as auxiliary data or ancillary data) is attached as metadata to the distance information and output, thereby improving the accuracy and efficiency of post-processing such as image synthesis. In this embodiment, various auxiliary data are packetized, superimposed on blanking areas, and output together with distance information, resulting in good transmission efficiency.

[0082] FIG. 8 is a diagram showing a packet structure of ancillary data superimposed on a blanking area in the first embodiment, and shows a packet structure for storing auxiliary data (ancillary data) defined in SMPTE ST 291. In FIG. The auxiliary data (ancillary data) is stored in a User Data Words area consisting of 10 bits each in the Type 2 ancillary data packet (ANC Data packet) shown in Fig. 8. The stored data may be in any format as long as the information can be obtained as a value.

[0083] The above-mentioned auxiliary data is stored in this user data word. In addition, a predetermined value indicating that the information is related to the data being transmitted is set for each packet in the DID (Data ID) and SDID (Secondary Data ID), which are header areas for identifying the type of ancillary data packet. Although a maximum of 255 user data words can be stored in one ancillary data packet, the above-mentioned auxiliary data may be divided into a plurality of ancillary data packets, superimposed, and output instead of storing up to 255 words.

[0084] Next, Fig. 9 is a flowchart of the stream generation process in the embodiment 1, and the stream generation process in the embodiment 1 will be described with reference to Fig. 9. Note that the operation of each step in the flowchart in Fig. 9 is performed by a CPU 102 as a computer expanding a computer program stored in a ROM 103 or the like as a storage medium into a RAM 104 and executing the program. In FIG. 9, in step S901, the CPU 102 generates and superimposes EAV, LN, and CRCC in accordance with SMPTE ST 292-1.

[0085] In step S902, the CPU 102 determines whether to superimpose an ancillary data packet storing each of the above-mentioned auxiliary data (ancillary data) on a blanking area. Since the superimposition position differs for each ancillary data packet, when a predetermined line and a predetermined sample timing are reached, a predetermined ancillary data packet is superimposed in step S903. Here, step S903 functions as an auxiliary data generating step (auxiliary data generating means) that generates auxiliary data related to distance information.

[0086] In step S904, data indicating the black level of the image signal is superimposed on the blanking area where no ancillary data packet is superimposed. The data indicating the black level is, for example, 64 for Y data and 512 for C data. In step S905, the CPU 102 generates and superimposes an SAV in accordance with SMPTE ST 292-1.

[0087] In step S906, CPU 102 determines whether or not the line is a valid pixel data superimposition line. For example, in the case of a progressive image, line 42 is the start line of the valid pixel data superimposition line, and the valid pixel data superimposition lines continue up to line 1121. In the case of an interlaced image, the valid pixel data superimposition lines of the first field are from line 21 to line 560, and the valid pixel data of the second field are from line 584 to line 1123. If it is determined that the line is a valid pixel data superimposition line, the process proceeds to step S907. On the other hand, if the line is not a valid pixel data superimposition line, CPU 102 proceeds to step S908.

[0088] In step S907, distance information and the like are superimposed as described with reference to Fig. 7. In step S908, black data is superimposed in the same manner as in step S904. The above-mentioned processing is performed for each line, and then an output step is executed in which the data is serialized and output as specified in SMPTE ST 292-1.

[0089] This enables the image processing device 100 to superimpose the image, distance information, and auxiliary data, and output the superimposed image to the outside via the image output unit 109. Here, steps S1301 to S1308 function as a data stream generating step (data stream generating means) for generating a data stream in which the image, distance information, and auxiliary data are superimposed.

[0090] In this embodiment, an example in which distance information itself is output has been described. However, instead of the distance information itself, for example, a shift amount may be output, and information indicating that the output information is not distance information but a shift amount, and information required to calculate distance information from the shift amount may be output as auxiliary data. In this way, distance information can be calculated in a post-processing step.

[0091] <Example 2> In the first embodiment, distance information and auxiliary data required for image synthesis and the like are output to the outside, so that image synthesis can be performed in post-processing, but even if the output is connected to an external display, it is not displayed as an image that allows a person to easily recognize the distance information. Therefore, in the second embodiment, an example will be described in which distance information is output so that an external display is connected and the user can easily check the acquisition status of the distance information.

[0092] 10 is a flowchart showing an example of a process for outputting distance information so as to be easily visible on a display in Example 2. Each process in this flowchart is realized by CPU 102 expanding a program stored in ROM 103 into RAM 104 and executing it. Note that in this example, the same reference numerals are used for configurations and steps that are the same as or similar to those in Example 1, and duplicated explanations will be omitted.

[0093] In step S1001, CPU 102 performs limit processing on the upper 10 bits of the distance information calculated in step S308 in Fig. 3 to convert it so that it does not become a prohibited code. Specifically, if the upper 10 bits of the distance information are 0, 1, 2, or 3, they are changed to 4, and if they are 1023, 1022, 1021, or 1020, they are changed to 1019.

[0094] Then, in step S1002, the upper 10 bits (D'[9] to D'[0]) of the distance information that has been limit-processed for each word by the CPU 102 are superimposed at the superimposition position of the Y data in Fig. 6 as shown in Fig. 11. Fig. 11 is a diagram showing an example of superimposition of distance information in the second embodiment. On the other hand, in the superimposition positions of the C data, a fixed value such as 1 is superimposed on the most significant bit C[9] and 0 is superimposed on the other positions C[8] to C[0].

[0095] By performing the above processing and having the image processing device 100 output the distance information via the image output unit 109 in accordance with the provisions of SMPTE ST 292-1, the distance information can be displayed in grayscale on a display capable of receiving HD-SDI as shown in Fig. 12. Fig. 12 is a diagram showing an example of how distance information is displayed in the second embodiment. 12, the closer the subject is, the higher the luminance is displayed, and the farther the subject is, the lower the luminance is displayed. In this way, the distance information is expressed by gradations of luminance, so that the user can easily visually recognize the acquisition status of the distance information.

[0096] In this embodiment, an example in which distance information is superimposed on the position of Y data has been described, but in step S1002, distance information may be superimposed on the position of C data and a fixed value may be superimposed on the position of Y data as shown in Fig. 13. Fig. 13 is a diagram showing another example of superimposing distance information in the second embodiment. When superimposing the distance information at the position of data C as shown in FIG. 13, the distance information can be displayed in shades of color like a heat map on an external display.

[0097] Here, when the image format to be transmitted is YCbCr422, the format is such that Cb and Cr are alternately superimposed on the C data for each word. Therefore, for example, by superimposing a fixed value on Cb and superimposing distance information only on Cr, the horizontal resolution of the distance information is halved, but the distance information may be displayed on the external display with a reddish gradation. Conversely, by superimposing a fixed value on Cr and superimposing distance information only on Cb, the distance information may be displayed on the external display with a blued gradation.

[0098] Alternatively, a predetermined threshold may be set, and if the distance information exceeds the threshold, the distance information is superimposed on Cb and a fixed value is superimposed on Cr, and if the distance information is below the threshold, the distance information is superimposed on Cr and the fixed value is superimposed on Cb. Conversely, if the distance information exceeds the threshold, the distance information is superimposed on Cr and a fixed value is superimposed on Cb, and if the distance information is below the threshold, the distance information is superimposed on Cb and the fixed value is superimposed on Cr. Alternatively, the distance information may be superimposed on the positions of both the Y data and the C data. In this case, the distance information is displayed on the external display so that both the brightness and color change according to the value of the distance information.

[0099] <Example 3> In the first and second embodiments, a conventional method for transmitting distance information by superimposing the image signal at the position where the image signal is superimposed has been described, but in this case, an output terminal must be prepared in addition to the terminal for outputting the distance information. Therefore, in the third embodiment, an example in which an image signal, distance information, and auxiliary data are output from a single output terminal will be described.

[0100] Fig. 14 is a diagram for explaining the data structure of SDI in the third embodiment, and the structure of a 3G-SDI data stream in the case where the frame rate is 29.97 fps, the horizontal resolution is 1920, and the vertical resolution is 1080 will be explained with reference to Fig. 14. The 3G-SDI data stream is specified in SMPTE ST 425-1.

[0101] 14, one line's worth of data stream is shown divided into four data streams, each with 10 bits per word. One frame has 1125 lines of this data stream. The R, G, B, and A streams each consist of 2200 words, each with 10 bits per word. The A stream corresponds to the alpha channel data stream, and in this embodiment, distance information is superimposed on the A stream, which is the alpha channel data stream.

[0102] EAV is superimposed on the data from the 1920th word, followed by LN and CRCC. A blanking area lasts for 268 words, followed by SAV. 1920 words of R (red) data, G (green) data, B (blue) data, and A (distance information) data are superimposed and transmitted. If the frame rate changes, the number of words per line changes, and so does the number of words in the blanking area.

[0103] Next, FIG. 15 is a flowchart of a stream generation process in the third embodiment, and FIG. 16 is a diagram showing an example of superimposing distance information in the third embodiment. The stream generation process in the third embodiment will be described using the flowchart in Fig. 15 and the overlapping example in Fig. 16. Each process in this flowchart is realized by the CPU 102 expanding a program stored in the ROM 103 into the RAM 104 and executing it. In this embodiment, the same or similar configurations and steps as those in the second embodiment are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0104] In step S1501, at the superimposition position of the A (distance information) data in Fig. 14, the upper 10 bits (D'[9] to D'[0]) of the distance information that has been limit-processed for each word by CPU 102 are superimposed as shown in Fig. 16. On the other hand, at the superimposition position of each RGB data, the image signal is superimposed as it is in the RGB format.

[0105] By carrying out the above-described processing and outputting in accordance with the provisions of SMPTE ST 425-1, it is possible to output the image signal, distance information, and auxiliary data from a single output terminal. In this embodiment, an example has been described in which image signals are superimposed in RGB format, but it is also possible to superimpose them in a format other than RGB, such as YCbCr, and to superimpose distance information and output it as A data, as shown in Figure 16.

[0106] <Example 4> In the first and second embodiments, examples of the distance information output method have been described, but the image processing device 100 may be configured so that the user can select the method and output the information according to the user's own purpose. In the fourth embodiment, an example of switching the output method in each embodiment and outputting the information will be described.

[0107] 17 is a flowchart showing an example of distance information output mode selection processing in Example 4. Each process in this flowchart is realized by the CPU 102 loading a program stored in the ROM 103 into the RAM 104 and executing it.

[0108] When a user operates the operation unit 113 to turn on a power supply unit (not shown), the CPU 102 performs shooting standby processing in step S1701. In the shooting standby processing, the CPU 102 displays, on the display unit 114, an image captured by the imaging unit 107 and processed by the image processing unit 105, and a menu for setting the image processing device 100.

[0109] In step S1702, the user operates operation unit 113 to perform various settings while looking at display unit 114. CPU 102 accepts settings according to the above operations and performs corresponding processes in each processing unit of image processing device 100. Here, the user can set an output mode of distance information using operation unit 113 as a selection means.

[0110] That is, the user can select a setting such as a high-precision distance information output mode as the output mode. The high-precision distance information output mode refers to a mode in which distance information is divided into multiple data streams, superimposed, and output, as described in the first embodiment. Note that the user may be able to use the operation unit 113 to select whether or not to superimpose the distance information on the alpha channel data stream (A stream). In addition, the user can select an output mode other than the high-precision distance information output mode, either the grayscale output mode for grayscale display described in Example 2, or the heat map output mode for heat map display.

[0111] In this manner, in this embodiment, it is possible to select whether to split and superimpose distance information onto multiple data streams or to superimpose distance information onto one data stream by using operation unit 113 as a selection means. Furthermore, when superimposing distance information onto one data stream, it is possible to select by operation unit 113 whether to superimpose distance information onto the luminance data stream or onto the color data stream.

[0112] In step S1703, CPU 102 determines whether the mode selected in step S1702 is the high-precision distance information output mode. If the mode for outputting distance information is the high-precision distance information output mode, the process proceeds to step S1704, and if not, the process proceeds to step S1705.

[0113] In step S1704, the processing described in the first embodiment is performed, and information indicating that the mode for outputting distance information is the high-precision distance information output mode and auxiliary data regarding the output data structure are superimposed on the distance information as shown in Fig. 7 and output from image output unit 109. That is, in step S1704, the distance information is divided into multiple data streams of Y (brightness) data and C (color) data and then superimposed.

[0114] On the other hand, in step S1705, CPU 102 determines whether the mode selected in step S1702 is the grayscale output mode or the heatmap output mode. If the selected mode is the grayscale output mode, the process proceeds to step S1706, and if the selected mode is the heatmap output mode, the process proceeds to step S1707.

[0115] In step S1706, the process described in the second embodiment is performed, and information indicating that the selected mode is the grayscale output mode and auxiliary data regarding the data structure to be output are superimposed on the distance information as shown in Fig. 11, and output from the image output unit 109. That is, in step S1706, the distance information is superimposed on the luminance data stream.

[0116] Furthermore, in step S1707, the processing described in the second embodiment is performed, and information indicating that the selected mode is the heat map output mode and auxiliary data regarding the output data structure are superimposed on the distance information as shown in Fig. 13, and output from the image output unit 109. That is, in step S1707, the distance information is superimposed on the color data stream.

[0117] By carrying out the above-described processing, the user can output distance information in a desired format. In this embodiment, the output method described in the first and second embodiments is selected and output, but the output method described in the third embodiment may be selected and output.

[0118] <Example 5> In the first to fourth embodiments, examples have been described in which 10-bit or 18-bit distance information is output. However, for example, when a process to be performed in post-processing image synthesis is a process of inserting a CG image between multiple photographed subjects, it may be easier to perform the post-processing by outputting distance information with a bit depth of less than 10 bits.

[0119] Alternatively, there are cases where reducing the bit depth makes it easier to view when displayed on an external display. In view of this, in the fifth embodiment, an example will be described in which the user is allowed to change the bit depth (number of effective bits) using the operation unit 113, and distance information according to the set bit depth is output.

[0120] 18 is a flowchart showing an example of distance information output mode selection processing in Example 5. Each process in this flowchart is realized by CPU 102 expanding a program stored in ROM 103 into RAM 104 and executing it. Note that in this example, the same reference numerals are used for configurations and steps that are the same as or similar to those in Example 4, and duplicated explanations will be omitted.

[0121] In step S1801, the user operates the operation unit 113 to change the bit depth (number of effective bits) of the distance information, etc., while looking at the display unit 114. The CPU 102 performs settings and processing according to the above operation on each processing unit of the image processing device 100. Here, the operation unit 113 functions as an effective bit number selection means that can select the effective bit number of the distance information.

[0122] An example in which the number of effective bits of distance information is set to 3 will be described below. In step S1802, the same process as in the first embodiment is performed, and auxiliary data indicating that the mode for outputting distance information is the high-precision distance information output mode and auxiliary data indicating the number of effective bits of the distance information are superimposed on the distance information as shown in Fig. 19. Then, the image is output from the image output unit 109.

[0123] 19 is a diagram showing an example of superimposing distance information in high-precision distance information output mode in the fifth embodiment, where the inverted value of the value to be superimposed at the Y[8] position is superimposed at the Y[9] position, and the inverted value of the value to be superimposed at the C[8] position is superimposed at the C[9] position. Then, the upper bits of the distance information of the number of effective bits set in step S1801 are superimposed from Y[8] to Y[0] and from C[8] to C[0], and 0 data is superimposed on the other lower bits.

[0124] In step S1803, the processing described in the second embodiment is performed, and auxiliary data indicating that the selected mode is a grayscale output mode and auxiliary data indicating the number of effective bits of the distance information are superimposed on the distance information as shown in FIG. 20 and output from the image output unit 109.

[0125] Fig. 20 is a diagram showing an example of superimposition of distance information in grayscale output mode in Example 5. Distance information D'' shown in Fig. 20 is a value obtained by extracting upper bits of the distance information by the number of effective bits set in step S1801, filling the remaining lower bits with 0 data, and then performing the same process as the limit process performed in step S1001.

[0126] Figure 21 is a diagram showing an example of the display of distance information in Example 5. When the distance information output in step S1803 is received and displayed on an external display, multiple layers divided by specified distances are displayed with different brightnesses, as shown in Figure 21.

[0127] In step S1804, the process described in the second embodiment is performed, and the distance information is superimposed as shown in Fig. 22 together with auxiliary data indicating that the selected mode is the heat map output mode and auxiliary data indicating the number of effective bits of the distance information. Then, the image is output from the image output unit 109.

[0128] Here, Fig. 22 is a diagram showing an example of superimposing distance information in the heat map output mode in the embodiment 5. The distance information D'' shown in Fig. 22 is a value obtained by extracting the most significant bits of the distance information by the number of effective bits set in step S1801, filling the other least significant bits with 0 data, and then performing the same process as the limit process performed in step S1001.

[0129] By performing the processing in this embodiment as described above, the user can select and set the bit depth, and distance information according to the set bit depth can be output.

[0130] In the above embodiment, an example has been described in which distance information is superimposed on an output format defined in SMPTE ST 292-1 and SMPTE ST 425-1 with a frame rate of 29.97 fps, a horizontal resolution of 1920, and a vertical resolution of 1080. However, the frame rate and resolution are not limited to the above examples, and distance information may be superimposed on another output format and output.

[0131] In addition, in each of the above-mentioned embodiments, the case where there is one transmission path has been described, but this is not limiting, and a configuration in which multiple transmission paths are prepared and distance information data is output using a transmission path separate from the image may be adopted. Furthermore, the transmission technology is not limited to SDI, and may be a transmission technology capable of transmitting images such as HDMI (registered trademark), DisplayPort (registered trademark), USB, or LAN, and multiple transmission paths may be prepared by combining these. Incidentally, the output means in this embodiment includes a means for outputting (supplying) a data stream in which an image, distance information, and auxiliary data are superimposed to the outside via a memory.

[0132] Also, for example, the first to fifth embodiments may be combined as appropriate. Also, the image processing device 100 may be configured so that the user can appropriately select the functions described in the first to fifth embodiments or combinations thereof by displaying a menu on the display unit 114 of the image processing device 100.

[0133] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.

[0134] A computer program for implementing all or part of the control in this embodiment may be supplied to an image processing device or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0135] 100: Image processing device 101: Internal bus 102:CPU 103:ROM 104:RAM 105: Image processing unit 106: Lens unit 107: Imaging unit 108: Network module 109: Image output unit 110: Recording medium I / F 111: Frame memory 112: Recording media 113:Operation unit 114: Display section 115: Object detection unit

Claims

1. an image capture means for capturing an image including a subject through a lens unit; a distance information acquisition means for acquiring distance information indicating a distance to the subject; an auxiliary data generating means for generating auxiliary data relating to the distance information; a data stream generating means for superimposing the image, the distance information and the auxiliary data to generate a data stream conforming to the SDI transmission standard; an output means for outputting the data stream to an external device using the SDI transmission standard; 13. An image processing device comprising:

2. An image capture means for capturing an image including a subject through a lens unit; a distance information acquisition means for acquiring distance information indicating a distance to the subject; an auxiliary data generating means for generating auxiliary data relating to the distance information; a data stream generating means for generating a data stream in which the image, the distance information, and the auxiliary data are superimposed; an output means for outputting the data stream to an outside; a selection means for selecting whether to split and superimpose the distance information onto a first number of data streams or to superimpose the distance information onto a second number of data streams which is smaller than the first number; 13. An image processing device comprising:

3. the image acquisition means includes an image sensor in which a plurality of photoelectric conversion units are arranged for each pixel, the photoelectric conversion units receiving light beams passing through different pupil regions of the lens unit, The image processing device according to claim 1 , wherein the distance information is obtainable based on a phase difference between outputs of the plurality of photoelectric conversion units.

4. The image processing device according to claim 3 , wherein the image is acquired by adding up outputs from the plurality of photoelectric conversion units.

5. 5. The image processing apparatus according to claim 1, wherein the data stream generating means divides and superimposes the distance information into a plurality of data streams.

6. 5. The image processing apparatus according to claim 1, wherein the data stream generating means superimposes the distance information on a luminance data stream.

7. 5. The image processing apparatus according to claim 1, wherein the data stream generating means superimposes the distance information on a color data stream.

8. 2. The image processing apparatus according to claim 1, wherein said data stream generating means superimposes said distance information on an alpha channel data stream.

9. 2. The image processing apparatus according to claim 1, further comprising a selection unit capable of selecting whether to split and superimpose the distance information onto a first number of data streams or to superimpose the distance information onto a second number of data streams which is less than the first number.

10. 10. The image processing apparatus according to claim 9, wherein the selection means is capable of selecting whether to superimpose the distance information on a luminance data stream or to superimpose the distance information on a color data stream.

11. 11. The image processing apparatus according to claim 9, wherein the selection means is capable of selecting whether or not to superimpose the distance information on an alpha channel data stream.

12. 12. The image processing device according to claim 1, further comprising an effective bit number selection means for selecting an effective bit number of the distance information.

13. 13. The image processing apparatus according to claim 1, wherein the data stream generating means superimposes significant bits of the distance information on a significant bit number and superimposes zeros on significant bits.

14. 14. The image processing device according to claim 1, wherein the auxiliary data includes setting data or data relating to a state of the image processing device.

15. 15. The image processing device according to claim 1, wherein the auxiliary data includes lens information relating to the lens unit.

16. 16. The image processing device according to claim 15, wherein the lens information includes at least one of a focal length, a focus position, and an aperture information.

17. 17. The image processing device according to claim 1, wherein the auxiliary data includes data for synchronizing the image and the distance information.

18. 18. The image processing device according to claim 1, wherein the auxiliary data includes data indicating an attribute of the distance information.

19. 19. The image processing device according to claim 1, wherein the auxiliary data includes information regarding a data structure of the data stream.

20. 20. The image processing apparatus according to claim 1, wherein said output means converts a part of said distance information so that it does not become a prohibited code in the SDI transmission standard.

21. an image capturing step of capturing an image including a subject through a lens unit; a distance information acquisition step of acquiring distance information indicating a distance to the subject; an auxiliary data generating step of generating auxiliary data related to the distance information; a data stream generating step of superimposing the image, the distance information, and the auxiliary data to generate a data stream conforming to the SDI transmission standard; and an output step of outputting the data stream to an outside using the SDI transmission standard.

22. A computer program for controlling each unit of the image processing apparatus according to any one of claims 1 to 20 by a computer.

Images