Data processing device, imaging device body, imaging device, and operation method for data processing device

Abstract

The present invention provides a data processing device, an imaging device body, an imaging device, and an operation method for the data processing device. This data processing device has a processor that acquires first data, second data, and third data from an imaging element of an imaging device and acquires information that indicates the relationship between the first data, the second data, and the third data on the basis of a first characteristic acquired on the basis of the first data and the second data and a second characteristic acquired on the basis of the first data and the third data. The first data is signal values for color pixels to which a color filter is provided in a region of the imaging element that includes specific pixels, the second data is signal values for first phase difference pixels that are pixels in the region that have been shielded on one side of a light reception unit, and the third data is signal values for second phase difference pixels that are pixels in the region that have been shielded on the opposite side of the light reception unit from the one side.

Description

Data processing device, imaging device body, imaging device, and method of operating data processing device

[0001] The present invention relates to a data processing device that handles data acquired from an imaging element, an imaging device body, an imaging device, and an operating method of a data processing device, and more particularly to a technique for handling data acquired from color pixels and phase difference pixels.

[0002] [Distance Information Estimation Technology] With regard to technology for handling data acquired from an image sensor, for example, Patent Document 1 describes a distance measurement device that calculates distance information of a subject based on a first color image and a second color image. Also, Patent Document 2 describes distance detection using a stereo vision method from input images acquired by an image capture unit with parallax.

[0003] Methods for outputting a distance map (a two-dimensional map of distance information) using AI (Artificial Intelligence) include a monocular inference method that infers distance from a single image and a stereo inference method that infers distance from stereo images. The AI ​​that inputs the stereo images performs the calculations for the correlation calculations performed on the left and right phase-difference images used in phase-difference AF (AF: Auto Focus). In the monocular inference method, depth is a relative value within the screen and differs from the actually measured distance. However, in the stereo inference method, depth is the amount of deviation between the left and right images, so a value close to the actually measured distance is obtained. As shown in FIG. 21 , by using the stereo inference AI to infer the left and right images of phase-difference AF as the left and right images of a stereo image, a distance map that can be compared with the results of phase-difference AF can be generated.

[0004] In the following description, the acquisition of distance information by correlation calculation of left and right phase-contrast images and the control performed using the acquired distance information are collectively referred to as "phase-contrast AF." "Phase-contrast AF" includes, but is not limited to, automatic focus control in an imaging device or the like. Furthermore, the "information about distance" may be the distance itself or other information corresponding to the distance.

[0005] [Resolution in Phase-Difference AF and AI Distance Estimation] Phase-difference AF includes a method using a phase-difference sensor and an image-plane phase-difference method. In the image-plane phase-difference method, for example, a large number of normal pixels (e.g., composed of color pixels with color filters) are arranged over the entire imaging surface of the image sensor, and phase-difference pixels (e.g., composed of pixels with one side shielded and pixels with the other side shielded, with no color filters) are intermittently arranged among them (see examples in Figures 3, 5 to 8, etc., described below). In this case, as described below, the resolution of phase-difference AF and the resolution of AI distance estimation differ.

[0006] Figure 22 shows examples of resolution in phase-difference AF and AI distance estimation. Part (a) of the figure is the original image. With phase-difference AF, as shown in part (b) of Figure 22, distance information is obtained in area units (rectangles in the figure) depending on the array pitch of phase-difference pixels (low resolution). On the other hand, with AI distance estimation (using image data obtained from normal pixels), an inferred distance (pixel shift amount) is obtained in pixel units (high resolution) as shown in part (c) of the figure.

[0007] [AI Learning Distance Map] The AI ​​learning distance map can be generated by a method in which the left and right images are input independently into AI (for example, a trained model such as a neural network can be used, but is not limited to this), and the amount of deviation between the left and right images is converted into a cost volume, which is then visualized as two-dimensional data (see, for example, the example in FIG. 23). A method for generating a distance map (or a disparity map) in this way is described, for example, in Non-Patent Document 1 below.

[0008] In recent years, many methods have been used to calculate cost volumes after extracting features from inputs of both left and right images with the same weight (see, for example, the example in FIG. 24). In the example in FIG. 24, "Group-Corr Volume" has a similar effect to the correlation calculation of phase-difference AF. Such a method is described, for example, in Non-Patent Document 2 below.

[0009] [Phase-difference AF and stereo matching] The correlation calculation part of phase-difference AF is basically the same as stereo matching. In phase-difference AF, a pixel misalignment amount is calculated by performing a correlation calculation on pairs of left and right phase-difference pixels, and a focus drive amount is calculated from the pixel misalignment amount. In stereo matching, the correlation calculation result may not be calculated correctly in areas with low contrast and no texture, saturated pixel values, repetitive patterns, or a lot of noise. In such scenes, distance miscalculation and distance hunting occur in the distance map. Therefore, measures to prevent distance miscalculation and distance hunting are required when using control using distance map information. Figure 25 shows an example of a scene in which the correlation calculation result is not calculated correctly. Note that parts (a1) and (a2) of Figure 25 show left-viewpoint images, and parts (b1) and (b2) of the same figure show right-viewpoint images corresponding to the left-viewpoint images of parts (a1) and (a2), respectively.

[0010] For such difficult scenes, phase-difference AF can take measures by using conditional branching algorithms to handle the difficult scenes, but AI cannot explicitly branch according to the difficult scenes, making it difficult to take measures. Figure 26 shows an example of inappropriate distance estimation in a distance map generated by AI. In Figure 26, as a result of performing distance estimation on the input image shown in part (a), the sky is erroneously estimated as being close, as shown in part (b) of the same figure.

[0011] The problem that "stereo matching is likely to result in areas where the distance is uncertain" also applies to phase-difference AF. Furthermore, if a color filter is not provided on the phase-difference pixels, the left and right phase-difference images are monochrome images, which lack more information than color images, making distance uncertainty more likely to occur. Figure 27 shows an example of a color scheme in which there is a difference in hue in color (part (a) of the figure), but the brightness difference disappears when the image is converted to a monochrome image (part (b) of the figure).

[0012] Japanese Patent No. 6555990 Japanese Patent Application Laid-Open No. 2012-123296

[0013] "From the Fundamentals to the Cutting Edge of Stereo Matching in 3D Image Measurement," OMRON TECHNICS Vol. 53 No. 2 (Issue 165) 2021, Taniai, Ryusuke, [Retrieved January 13, 2024], Internet (https: / / www.omron.com / jp / ja / technology / omrontechnics / 2021 / 20211119-taniai.html). "Introducing the MIRU2020 Award-Winning Paper 'Unsupervised Domain Adaptation for Stereo Matching,'" SenseTime Japan TECH blog, Sakuma, [Retrieved January 13, 2024], Internet (https: / / tech.sensetime.jp / ?p=67).

[0014] One embodiment of the technique of the present disclosure provides a data processing device, an imaging device body, an imaging device, and a method for operating a data processing device.

[0015] A data processing device according to a first aspect of the present invention is a data processing device including a processor, wherein the processor acquires, from an imaging element of an imaging device, signal values ​​of color pixels having color filters arranged thereon in a region including specific pixels of the imaging element as first data, acquires, as second data, signal values ​​of first phase difference pixels in the region that are pixels having one side of their light receiving units shielded, and acquires, as third data, signal values ​​of second phase difference pixels in the region that are pixels having one side of their light receiving units shielded opposite to the one side, and acquires information indicating a relationship between the first data, the second data, and the third data based on a first feature acquired based on the first data and the second data and a second feature acquired based on the first data and the third data.

[0016] In the first aspect and each of the following aspects, the "area including a specific pixel" may be a single point (one pixel), or may be a one-dimensional or two-dimensional area including the specific pixel.

[0017] The data processing device according to the first aspect may be realized as a device that acquires signal values ​​of color pixels, signal values ​​of first phase difference pixels, and signal values ​​of second phase difference pixels from an external device and generates the above-mentioned “information indicating the relationship”, or may be realized as a data processing device including a data acquisition unit or a processor part of an imaging device.

[0018] A data processing device according to a second aspect of the present invention is the first aspect, wherein the processor acquires a first feature based on color image data generated from signal values ​​of color pixels and first phase difference image data generated from signal values ​​of first phase difference pixels, and acquires a second feature based on the color image data and second phase difference image data generated from signal values ​​of second phase difference pixels.

[0019] In the data processing device of the third aspect, in the second aspect, the processor acquires a first feature based on a recognition result of a target pixel in a color image, which is an image represented by color image data, and acquires a second feature based on the recognition result and second phase difference image data.

[0020] In the data processing device of the fourth aspect, in the second or third aspect, the processor determines whether to use the first phase difference image data or the second phase difference image data to obtain information indicating the relationship in position based on the position of a specific pixel in the imaging element, and based on the result of the determination, generates information indicating the relationship in position using the color image data and one of the first phase difference image data and the second phase difference image data.

[0021] In a data processing device according to a fifth aspect, in the fourth aspect, the processor acquires information indicating the relationship at a position using the phase difference image data having a higher signal value at the position between the first phase difference image data and the second phase difference image data.

[0022] A data processing device according to a sixth aspect is any one of the first to fifth aspects, wherein the processor acquires information relating to a distance from the imaging device to the subject as information indicating the relationship.

[0023] A data processing device according to a seventh aspect is the sixth aspect, wherein the processor acquires information indicating the reliability of the distance as information relating to the distance.

[0024] The data processing device according to an eighth aspect is the data processing device of the sixth or seventh aspect, wherein the processor obtains information about distance using an estimator constructed using a machine learning algorithm.

[0025] A ninth aspect of the present invention is the data processing device of any one of the first to eighth aspects, wherein the processor causes the output device to output the information indicating the relationship as a two-dimensional map.

[0026] A data processing device according to a tenth aspect is any one of the second to fifth aspects, wherein the color filters include a red color filter that transmits light in the red wavelength band, a green color filter that transmits light in the green wavelength band, and a blue color filter that transmits light in the blue wavelength band, and the processor generates color image data based on signal values ​​in the red wavelength band, signal values ​​in the green wavelength band, and signal values ​​in the blue wavelength band.

[0027] A data processing device according to an eleventh aspect is the tenth aspect, wherein the color filter further includes a white color filter that transmits light in the white wavelength band, and the processor generates color image data based on signal values ​​in the red wavelength band, signal values ​​in the green wavelength band, signal values ​​in the blue wavelength band, and light in the white wavelength band.

[0028] An imaging device main body according to a twelfth aspect is an imaging device main body including a data processing device according to any one of the first to eleventh aspects and an imaging element, wherein the imaging element includes color pixels arranged two-dimensionally and having color filters disposed thereon, first phase-difference pixels which are two-dimensionally arranged pixels not having color filters disposed thereon and which are shielded on one side, and second phase-difference pixels which are two-dimensionally arranged pixels not having color filters disposed thereon and which are shielded on one side and on the side opposite to the optical axis of the imaging element, and wherein the processor acquires color image data generated from signal values ​​of the color pixels as first data, acquires first phase-difference image data generated from signal values ​​of the first phase-difference pixels as second data, and acquires second phase-difference image data generated from signal values ​​of the second phase-difference pixels as third data.

[0029] An imaging device body according to a thirteenth aspect is the twelfth aspect, wherein an arrangement pitch of the first phase difference pixels and the second phase difference pixels in the imaging element is wider than an arrangement pitch of the color pixels in the imaging element.

[0030] The imaging device body according to the fourteenth aspect is the twelfth or thirteenth aspect, in which the processor adjusts the size of the image indicated by the color image data to the size of the image indicated by the first phase difference image data and / or the size of the image indicated by the second phase difference image data based on the relationship between the arrangement pitch of the color pixels and the arrangement pitch of the first phase difference pixels and the second phase difference pixels.

[0031] An imaging device according to a fifteenth aspect comprises an imaging device body according to any one of the twelfth to fourteenth aspects, and a single optical system that can be attached to the imaging device body and forms an optical image of a subject on an imaging element, and the imaging element is a single imaging element.

[0032] A method for operating a data processing device according to a sixteenth aspect is a method for operating a data processing device including a processor, in which the processor acquires, from an imaging element of the imaging device, signal values ​​of color pixels having color filters arranged thereon in a region including specific pixels of the imaging element as first data, acquires, as second data, signal values ​​of first phase difference pixels in the region that are pixels having one side of their light receiving units shielded, acquires, as third data, signal values ​​of second phase difference pixels in the region that are pixels having one side of their light receiving units shielded opposite to the one side, and acquires information indicating the relationship between the first data, the second data, and the third data based on a first feature acquired based on the first data and the second data and a second feature acquired based on the first data and the third data.

[0033] FIG. 1 is a diagram illustrating a configuration of an imaging device according to a first embodiment. FIG. 2 is a diagram illustrating a configuration of a data processing unit. FIG. 3 is a diagram illustrating an example of a pixel arrangement in an image sensor. FIG. 4 is a diagram illustrating an example of the configuration of normal pixels and phase difference pixels. FIG. 5 is a diagram illustrating an example of a two-dimensional array of color pixels and phase difference pixels. FIG. 6 is a diagram illustrating another example of a pixel arrangement in an image sensor. FIG. 7 is a diagram illustrating another example of a pixel arrangement in an image sensor. FIG. 8 is a diagram illustrating yet another example of a pixel arrangement in an image sensor. FIG. 9 is a diagram illustrating stereo matching in Example 1. FIG. 10 is a diagram illustrating stereo matching in Example 2. FIG. 11 is a diagram illustrating the relationship between the focus position and left-right image misalignment. FIG. 12 is a diagram illustrating the relationship between the focus position and subject blur. FIG. 13 is a diagram illustrating the relationship between the position on the image sensor and the output of the phase difference pixels. FIG. 14 is a diagram illustrating stereo matching performed between images with high S / N ratios. FIG. 15 is a diagram illustrating the reliability of the stereo matching result. FIG. 16 is a diagram illustrating acquisition of a final distance map based on a symmetry determination result. FIG. 17 is a diagram showing how reliability determination is performed by AI (determiner). FIG. 18 is a diagram showing a modified example of pixel arrangement in an image sensor. FIG. 19 is a flowchart showing how processing conditions are switched depending on whether the conditions result in a deterioration of the S / N ratio. FIG. 20 is a diagram showing how segmentation and resizing are performed simultaneously by AI. FIG. 21 is a diagram showing how distance is inferred from a stereo image by AI. FIG. 22 is a diagram showing an example of resolution in phase difference AF and AI distance estimation. FIG. 23 is a diagram showing an example of an AI learning-type distance map. FIG. 24 is another diagram showing an example of an AI learning-type distance map. FIG. 25 is a diagram showing an example of a scene in which the correlation calculation result is not calculated correctly. FIG. 26 is a diagram showing an example of inappropriate distance estimation in a distance map generated by AI. FIG. 27 is a diagram showing an example of a subject that is difficult for a monochrome image sensor to capture.

[0034] [First Embodiment] [Overall Configuration of Imaging Device] Fig. 1 is a diagram showing the configuration of an imaging device 10 (data processing device, imaging device main body, imaging device) according to the first embodiment. The imaging device 10 is composed of an interchangeable lens 100 (single optical system, monocular optical system) and an imaging device main body 200 (imaging device main body, data processing device), and forms a subject image (optical image) on an imaging element 202 (imaging element, single imaging element) using a photographing lens including a zoom lens 102 (described later). The interchangeable lens 100 and the imaging device main body 200 are attachable (and detachable) via mounts (not shown).

[0035] [Configuration of Interchangeable Lens] The interchangeable lens 100 includes a zoom lens 102, a focus lens 104, an aperture 106, and a lens driver 110. The lens driver 110 drives the zoom lens 102 and the focus lens 104 forward and backward in response to commands from a data processor 210 (optical system driver 230 in FIG. 2 ) to perform zoom (optical zoom) adjustment and focus adjustment. The zoom and focus adjustments may be performed in response to commands from the data processor 210 or in response to zoom and focus operations (such as rotation of a zoom ring or focus ring, not shown) performed by the user. The lens driver 110 also controls the aperture 106 in response to commands from the data processor 210 to adjust exposure. Information such as the positions of the zoom lens 102 and the focus lens 104 and the aperture of the aperture 106 is input to the data processor 210. The interchangeable lens 100 has an optical axis L.

[0036] [Configuration of Imaging Device Main Body] The imaging device main body 200 includes an imaging element 202 (imaging element, single imaging element), an AFE (AFE: Analog Front End) 204, an A / D converter 206 (A / D: Analog to Digital, imaging unit), a data processing unit 210 (processor), an operation unit 260, a recording unit 270, and a monitor 280. The imaging device main body 200 may include a shutter (not shown) for blocking light incident on the imaging element 202. If a shutter is included, it is preferable that the shutter speed be variable.

[0037] The image sensor 202 has a light-receiving surface on which a large number of light-receiving elements are arranged in a two-dimensional matrix. The light-receiving surface of the image sensor 202 is provided with color pixels and phase-difference pixels, as described in detail below, and is capable of acquiring color image data and phase-difference image data of a subject. A color image based on the color image data and a phase-difference image based on the phase-difference image data can be recorded in a recording unit 270 (output device), displayed on a monitor 280 (output device), or output to an external device 300 (output device). Subject light transmitted through the zoom lens 102, focus lens 104, and aperture 106 is focused on the light-receiving surface of the image sensor 202 and converted into electrical signals by each light-receiving element. The detailed configuration of the image sensor 202 and the acquisition of image data will be described later. The image sensor 202 can be any of a variety of photoelectric conversion elements, such as a complementary metal-oxide semiconductor (CMOS) or a charge-coupled device (CCD).

[0038] The AFE 204 performs noise removal, amplification, etc. on the analog image signal output from the image sensor 202, and the A / D converter 206 converts the captured analog image signal into a digital image signal with a wide gradation range.

[0039] 2 is a diagram showing the configuration of the data processing unit 210. The data processing unit 210 (processor) includes a processor 220, a ROM 240 (Read Only Memory), and a RAM 250 (Random Access Memory). The processor 220 includes a color image data acquisition unit 222, a phase difference image data acquisition unit 224, a relationship information acquisition unit 226, an optical system driver 230, and an input / output controller 234. Details of the processing performed by these functions will be described later.

[0040] The processor 220 is configured with various processors and electrical circuits, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a PLD (Programmable Logic Device), etc. When these processors and electrical circuits execute software (programs), computer-readable code of the software to be executed (for example, various processors and electrical circuits that constitute the processor, and / or a combination thereof) is stored in a non-transitory and tangible recording medium such as the ROM 240, and the computer references the software.

[0041] The software stored in the non-transitory, tangible recording medium includes various programs (such as the operating method of the data processing device according to the present invention) and data used in executing them. Instead of the ROM 240, the code may be recorded in a non-transitory, tangible recording medium such as a flash ROM or an EEPROM (Electronically Erasable and Programmable Read Only Memory). Note that this "non-transitory, tangible recording medium" does not include non-tangible recording media such as carrier signals or propagation signals themselves. When processing using the software, the RAM 250 is used as a temporary storage area or working area.

[0042] [Configuration of Operation Unit and Monitor] The operation unit 260 has a release button, operation buttons, dials, switches, etc. (not shown), and allows a user to perform various operations such as acquiring color images (color image data) and phase difference images (phase difference image data), generating phase difference maps and distance maps, and outputting these results. Note that the monitor 280 may be configured as a touch panel device, and this device may be used as the operation unit 260. The operation unit 260 may also include a microphone and speaker (not shown).

[0043] The monitor 280 (display device) is configured with a touch panel type liquid crystal display panel and can display normal moving images, still images, phase difference image data, phase difference maps, distance maps, etc. The monitor 280 can be disposed on the back side, top side, etc. of the imaging device main body 200.

[0044] [Configuration of the Recording Unit] The recording unit 270 (recording device) is composed of a non-transitory, tangible recording medium such as various types of magneto-optical recording medium or semiconductor memory, and its control circuit, and records color images (color image data), phase difference images (phase difference image data), distance maps, distance reliability maps, etc. Data acquired from the external device 300 may be recorded in the recording unit 270. The input / output control unit 234 (processor) can display the data recorded in the recording unit 270 on the monitor 280 or output it to the external device 300 in response to a user instruction via the operation unit 260, or automatically without a user instruction. The recording medium used in the recording unit 270 may be a type that is detachable from the imaging device main body 200, such as various memory cards.

[0045] [External Device] An external device 300 can be connected to the imaging device 10 to input and output information. The connection may be via wired or short-range wireless communication, or via a network. Various display devices or recording devices can be used as the external device 300, and an imaging device other than the imaging device 10 may also be used as the external device 300. Furthermore, a vehicle or a moving object may be used as the external device 300, and data output from the imaging device 10 (such as normal moving images or still images, or distance information such as a distance map) may be used to control the vehicle, moving object, or other device (e.g., focus control, maintaining a safe distance, route control, collision prevention, and avoidance of dangerous objects). If the external device 300 is equipped with an optical system, the optical system may be used in place of the interchangeable lens 100. Furthermore, if the external device can acquire color image data and phase-contrast image data, the data may be input to the data processing unit 210 for processing.

[0046] [Configuration of Image Sensor] [Pixel Arrangement Example (Part 1)] FIG. 3 is a diagram showing an example of a pixel arrangement in the image sensor 202 (a state in which a portion of the light receiving surface of the image sensor 202 is viewed from the subject side), and FIG. 4 is a diagram showing an example of the arrangement of normal pixels and phase difference pixels. As shown in FIG. 3 , the image sensor 202 includes phase difference pixels and normal pixels (color pixels). In the example of FIG. 3 , the image sensor 202 includes color pixels 202R, 202G, and 202B (color pixels, normal pixels), and these color pixels are provided with red color filters, green color filters, and blue color filters that transmit light in the red wavelength band, green wavelength band, and blue wavelength band, respectively. A white color filter that transmits light in the white wavelength band may also be provided (see the example of FIG. 8 described later). These color filters constitute "multiple optical filters that transmit light in at least some different wavelength bands."

[0047] In the example of FIG. 3, the pixel arrangement is a Bayer array, but other arrangements such as a diagonal Bayer array (double Bayer array), X-Trans (registered trademark), or quad Bayer array may also be used. The pixel arrangement in X-Trans (registered trademark) will be described later with reference to FIGS. 6 and 7. The data processing unit 210 (processor 220) can generate color image data (first data) from the signal values ​​(first data) output from these color pixels. In generating the color image data, the data processing unit 210 performs a synchronization process (also called a demosaic process) according to the pixel arrangement.

[0048] In this embodiment, the color image data does not necessarily need to be output as an image (color image), but a color image corresponding to the generated color image data may be output (for example, displayed on the monitor 280). Note that, hereinafter, color image data and color image data may be referred to as RGB image data and RGB image data, respectively, but these terms also include cases where the color pixels include W pixels, which will be described later.

[0049] In the following description, the color pixels 202R, 202G, and 202B may be referred to as "R pixel, G pixel, and B pixel," respectively. Microlenses are provided in the color pixels and the phase difference pixels (not shown in FIG. 3, see FIG. 4).

[0050] [Arrangement of Phase Difference Pixels (Intermittent Arrangement)] As shown in part (a) of FIG. 3 , the image sensor 202 includes phase difference pixels 201 and 203 (first phase difference pixel and second phase difference pixel). The phase difference pixel 201 (first phase difference pixel) has an opening 201A on the left side of the pixel (the left side as you face the figure) and functions as a light receiving unit, and the right side of the pixel (the right side as you face the figure; one side of the light receiving unit) is shielded from light by a mask 201B. On the other hand, the phase difference pixel 203 (second phase difference pixel) has an opening 203A on the right side of the pixel (the right side as you face the figure) and functions as a light receiving unit, and the left side of the pixel (the left side as you face the figure; the side opposite the mask 201B of the phase difference pixel 201) is shielded from light by a mask 203B. The phase difference image data acquisition unit 224 (processor) can acquire image data generated from the signal values ​​(second data) of the phase difference pixels 201 as first phase difference image data (second data), and can acquire image data generated from the signal values ​​(third data) of the phase difference pixels 203 as second phase difference image data (third data).

[0051] Hereinafter, the first phase-contrast image data (and the first phase-contrast image) may be referred to as "phase-contrast image A" or "phase-contrast image A," and the second phase-contrast image data (and the second phase-contrast image) may be referred to as "phase-contrast image B" or "phase-contrast image B."

[0052] 3A shows an example in which one pixel is divided and shielded in the horizontal direction, but as shown in FIG. 3B, it may also be divided and shielded in the vertical direction. In the example shown in FIG. 3B, the phase difference pixel 201 has an opening 201D on the upper side (upper side as viewed in the figure) of the pixel, which functions as a light receiving unit, and the lower side (lower side as viewed in the figure; one side of the light receiving unit) of the pixel is shielded from light by a mask 201C. On the other hand, the phase difference pixel 203 has an opening 203D on the lower side (lower side as viewed in the figure) of the pixel, which functions as a light receiving unit, and the upper side (upper side as viewed in the figure; the other side of the light receiving unit) of the pixel is shielded from light by a mask 203C. In this case, the phase difference image data acquisition unit 224 (processor) can acquire image data generated from the signal output by the phase difference pixel 201 as first phase difference image data (second data), and can acquire image data generated from the signal output by the phase difference pixel 203 as second phase difference image data (third data). The intervals between the phase difference pixels may be different from the example shown in Fig. 3. That is, the phase difference pixels may be adjacent to each other or may be spaced apart (the same applies to other examples of pixel arrangements).

[0053] 3, the phase difference pixels are arranged at the positions of the G pixels, but the phase difference pixels may be arranged at the positions of the R pixels or the B pixels. Furthermore, although color filters are not arranged in the phase difference pixels in the example of FIG. 3, color filters may be arranged in the phase difference pixels (the same applies to the following arrangement examples).

[0054] As shown in part (a) of Fig. 4, the color pixel includes a microlens ML and a photodiode PD (color filters are not shown). On the other hand, as shown in part (b) of Fig. 4, the phase difference pixel includes a microlens ML, a photodiode PD, and a mask 202A. As described above with reference to Fig. 3, the position of the mask 202A can be left / right or up / down. The position, shape, and size of the mask 202A can be set according to the position, shape, and size of the pupil to be directed.

[0055] FIG. 5 is a diagram showing an example of a two-dimensional array of color pixels and phase difference pixels. In the example shown in the figure, horizontally divided phase difference pixels 202X and vertically divided phase difference pixels 202Y are arranged in orthogonal directions, resulting in a two-dimensional array of the phase difference pixels as a whole. Color pixels are arranged at pixel positions other than the phase difference pixels. In FIG. 5 , the x-direction array pitch of the color pixels is dx1 and the y-direction array pitch is dy1, and the x-direction array pitch of the phase difference pixels is dx2 and the y-direction array pitch is dy2. The relationship between these array pitches is dx2>dx1 and dy2>dy1, and the array pitch of the phase difference pixels 201 (first phase difference pixels) and the phase difference pixels 203 (second phase difference pixels) in the image sensor 202 is wider than the array pitch of the color pixels 202R, 202G, and 202B (color pixels) in the image sensor 202. This is because the number of phase difference pixels is smaller than that of color pixels, and the lines on which the phase difference pixels exist are thinned out in the vertical direction as well, for example, to one out of every 12 lines, etc. Therefore, as described above with reference to Fig. 22 , the resolution of the phase difference image data (area units corresponding to the array pitches dx2 and dy2) is lower than the resolution of the color image data (pixel units corresponding to the array pitches dx1 and dy1).

[0056] Although FIG. 5 shows the phase difference pixels 202X and the phase difference pixels 202Y in one row, in order to generate a highly accurate phase difference map or distance map, it is preferable to arrange the phase difference pixels in the vertical and horizontal directions over the entire surface of the image sensor 202.

[0057] Furthermore, for pixel positions where color pixels cannot be placed due to the placement of phase difference pixels, the signal at that position can be obtained by interpolation calculation using signals from surrounding pixels.

[0058] [Pixel Arrangement Example (Part 2)] FIGS. 6 and 7 are diagrams showing other examples of pixel arrangements in the image sensor 202. These diagrams show examples of pixel arrangements (a state in which a part of the light receiving surface of the image sensor 202 is viewed from the subject side) when the color filter arrangement is X-Trans (registered trademark). FIG. 6 shows an example in which phase difference pixels 201 and 203 (first phase difference pixels and second phase difference pixels) divided in the left-right direction are provided at the positions of G pixels, and FIG. 7 shows an example in which phase difference pixels 201 and 203 (first phase difference pixels and second phase difference pixels) divided in the up-down direction are provided at the positions of G pixels. Even with such pixel arrangements, color image data and phase difference image data can be acquired.

[0059] [Pixel Arrangement Example (Part 3)] FIG. 8 is a diagram showing yet another example of a pixel arrangement in the image sensor 202. In the above-described "Pixel Arrangement Examples (Part 1) and (Part 2)," color pixels are provided with color filters that transmit light in the red, green, and blue wavelength bands. However, as shown in FIG. 8, white color filters that transmit light in the white wavelength band may be provided in the color pixels. In the example shown in FIG. 8, the most pixels are those with white color filters (W pixels), followed by the most G pixels. The number of R pixels and B pixels is equal. The phase difference pixels 201 and 203 can be provided at the positions of the W pixels as in the example of FIG. 8, but may also be provided at the positions of the G pixels, R pixels, or B pixels. Color image data and phase difference image data can also be acquired with this pixel arrangement. 8 illustrates a state in which phase difference pixels 201 and 203 (first phase difference pixel and second phase difference pixel) divided in the left-right direction are provided. However, similar to the portion (b) of FIG. 3 and the portion (b) of FIG. 7 described above, it is preferable to also provide phase difference pixels divided in the left-right direction.

[0060] [Adding Color Information to Phase-Contrast Images] As described above, the problem that "stereo matching is prone to producing areas with indeterminate distances" also applies to phase-contrast AF. Furthermore, in the pixel arrangement examples (1) to (3) described above, color filters are not provided on the phase-contrast pixels, and the left and right phase-contrast images are monochrome images, which means that information is missing compared to color images, making it more likely that indeterminate distances will occur. Therefore, in this embodiment, as will be described in the following examples, RGB information (color information) is added to stereo matching using phase-contrast images, thereby improving and stabilizing the stereo-matching performance of phase-contrast images.

[0061] [Example 1] Fig. 9 is a diagram showing the stereo matching process in Example 1. In a first mode, left and right phase-contrast images P1L and P1R are superimposed with a color image C1 (RGB image) and input as input data for stereo matching. A color image data acquisition unit 222 (processor) acquires color image data (first data), and a phase-contrast image data acquisition unit 224 (processor) acquires left and right phase-contrast image data (second data, third data). The shape of the input data (image data size) is W (number of pixels in the width direction) x H (number of pixels in the height direction) x (3 channels of RGB + 1 channel of phase-contrast image).

[0062] The color image data acquisition unit 222 acquires color image data (first data) for all pixels of the image sensor 202 (a region including specific pixels of the image sensor). For pixel positions where data is missing due to the arrangement of phase-difference pixels, data is acquired by interpolation. Similarly, the phase-difference image data acquisition unit 224 acquires phase-difference image data (second data) for all pixels of the image sensor (a region including specific pixels of the image sensor). For pixel positions where data is missing due to the arrangement of phase-difference pixels, data is acquired by interpolation. Note that the number of pixels in an RGB image (color image) and a phase-difference image often differ. In this case, it is preferable that the processor 220 thins out or resizes the RGB image based on the phase-difference image to generate the above-mentioned RGB image + phase-difference image data (the same applies to each of the following embodiments). Specific methods of thinning and resizing will be described later.

[0063] [Areas for Acquiring Color Image Data and Phase-Contrast Image Data] In Example 1, the color image data acquisition unit 222 (processor) may acquire color image data for the entire area (all pixels) of the image sensor, or for a portion of the area. The color image data acquisition unit 222 can acquire color image data for an area in the image sensor 202 that includes a specific pixel for which distance-related information is to be calculated. The area from which data is acquired may be a single specific pixel (one pixel), or a one-dimensional or two-dimensional area that includes the specific pixel. Note that the above description also applies to the acquisition of left and right phase-contrast image data by the phase-contrast image data acquisition unit 224, but the area from which color image data is acquired and the area from which phase-contrast image data are acquired are assumed to be the same or identical.

[0064] [Acquisition of First Feature and Second Feature] In Example 1, the relationship information acquisition unit 226 (processor) can acquire, as the “second feature,” the deviation (amount and direction of deviation) between the signal value distribution (first data) in the above region of the color image C1 and the signal value distribution (second data) in the above region of the phase-contrast image P1L (phase-contrast image A). In addition, the relationship information acquisition unit 226 can acquire, as the “first feature,” the deviation (amount and direction of deviation) between the signal value distribution (first data) in the above region of the color image C1 and the signal value distribution (third data) in the above region of the phase-contrast image P1R (phase-contrast image B).

[0065] [Acquisition and Output of Information Indicating the Relationship Between the First to Third Data] The relationship information acquisition unit 226 (processor) acquires information indicating the relationship between the first data, the second data, and the third data based on the first feature and the second feature, as described below. The "information indicating the relationship" may be information regarding the distance from the imaging device 10 to the subject. In this case, the "information regarding the distance" may be the distance itself, or information corresponding to the distance (phase difference, parallax, etc.).

[0066] The input / output control unit 234 (processor) can output the "information indicating a relationship" as numerical values, characters, figures, symbols, etc. This "output" includes, but is not limited to, displaying on the monitor 280, recording in the recording unit 270, and outputting to the external device 300. Furthermore, when outputting the "information indicating a relationship," the relationship information acquisition unit 226 and the input / output control unit 234 may change color or brightness depending on distance, phase difference, parallax, etc. The relationship information acquisition unit 226 and the input / output control unit 234 may output the "information indicating a relationship," such as the "information regarding distance," as a two-dimensional map (a distance map, a phase difference map, a parallax map, etc.). The processor 220 may set the output mode of information in response to a user operation via the operation unit 260, or may set the output mode automatically without relying on a user operation.

[0067] In the first embodiment, the relationship information acquisition unit 226 (processor) performs stereo matching of the input data described above using a neural network NN1 (estimator) and outputs a two-dimensional distance map M1. The distance map M1 is an example of information regarding the distance from the image capture device 10 to the subject, and is information indicating the relationship between first data, second data, and third data. In the distance map M1, the shade of black and white (lightness) corresponds to the distance, with whiter areas (lighter areas) indicating closer distances and blacker areas (darker areas) indicating farther distances. The relationship information acquisition unit 226 and the input / output control unit 234 may output the distance map M1 using a different pattern of color or lightness (for example, a pattern in which closer points are red and farther points are blue, or vice versa).

[0068] The neural network NN1 is an estimator constructed using a machine learning algorithm and includes an input layer, intermediate layers (convolutional layer, pooling layer, batch normalization layer, etc.), and an output layer. The neural network NN1 performs a convolution operation using a filter to output a distance map M1. Note that the layer configuration is not limited to the configuration shown in FIG. 9 . The processor 220 can construct the estimator (neural network NN1) by providing the relationship between the left and right phase-contrast images and distance information as ground truth data (teaching data), as well as corresponding simulation results, as training data to a neural network such as a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), or an autoencoder for learning. The training data may be acquired by the input / output control unit 234 (processor) from an external device 300, or data recorded in the recording unit 270 may be used. Alternatively, a separately constructed estimator (such as layer configuration, filter size, and weight parameter values) may be imported into the imaging device 10 and used as the estimator.

[0069] In constructing the estimator, distance information (information corresponding to the distance from the imaging device to the subject) serving as ground truth data can be obtained by using actual measurements such as LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging). LiDAR is a technology that irradiates an object with laser light and measures the distance to the object and the shape of the object from the results of receiving the reflected light. LiDAR may be a Time of Flight (TOF) method using pulsed laser light, or a Frequency-Modulated Continuous Wave (FMCW) method using continuous wave laser light.

[0070] When using the neural network NN1 configured as described above, it is preferable to perform a process (error backpropagation) in the learning process, in which the result output by the output layer is compared with the "information on distance" as correct data to calculate the loss (error), and the weight parameters in the intermediate layer are updated from the output layer to the input layer so as to reduce the loss.

[0071] The processor 220 may train the neural network NN1 to output a disparity map or a phase difference map instead of or in addition to the distance map M1 (the same applies to other embodiments). The distance corresponds to the magnitude of the disparity, with a short distance resulting in a large disparity and a long distance resulting in a small disparity.

[0072] According to the first embodiment, by adding RGB images to the phase-contrast images as described above, it is possible to compensate for the loss of contrast due to the color combination and improve the stereo matching accuracy of the phase-contrast images. Although there is a positional misalignment between the RGB images and the phase-contrast images, in an image-plane phase-contrast sensor such as the image sensor 202, the actual pixel misalignment amount is within ± several pixels (depending on the pixel array and pixel pitch), so even an input that simply adds together the RGB images and the phase-contrast images is effective. Conversely, in cases where the parallax is large, such as in compound-eye stereo (stereo matching using multiple images acquired by a compound-eye optical system), the parallax between the RGB images and the phase-contrast images may be too large, preventing normal learning.

[0073] [Example 2] In addition to the lack of color information described above, distance estimation can be incorrect depending on conditions such as low contrast and repetitive patterns, and it is difficult to calculate the cost for such areas using simple stereo matching. Therefore, in Example 2, an RGB image is segmented in advance (an example of "recognizing target pixels") to generate images divided into areas by class, such as sky and wall, and this information is added to the phase contrast image before being input to the stereo matching AI, so that difficult areas such as sky can be assigned to infinity as a result of the stereo matching cost volume.

[0074] FIG. 10 is a diagram illustrating stereo matching in Example 2. In a first aspect, the input data for stereo matching is obtained by superimposing a segmentation image S1 on left and right phase-contrast images P1L and P1R. A color image data acquisition unit 222 (processor) acquires color image data (first data) and performs segmentation using a known method (e.g., a machine learning algorithm such as a neural network), and a phase-contrast image data acquisition unit 224 (processor) acquires left and right phase-contrast image data (second data, third data). The segmentation result is an example of a "recognition result of a target pixel in a color image." The shape of the input data (image data size) is W (number of pixels in the width direction) × H (number of pixels in the height direction) × (one channel of segmentation result + one channel of phase-contrast image).

[0075] The neural network NN2 (estimator) is an estimator constructed using a machine learning algorithm, and outputs a distance map M2 (or a disparity map) similar to the distance map M1. The neural network NN2 can be configured in the same manner as the neural network NN1 described above.

[0076] In this way, even if processed data such as segmentation is added to the left and right phase difference images instead of raw information of color image data (RGB image data), the stereo matching results can be improved.

[0077] Third Embodiment In a third embodiment, a case will be described in which a phase-contrast image to be used for stereo matching is determined according to a position on the imaging element.

[0078] Phase-difference AF measures the amount and direction of misalignment between the left and right images and performs focus adjustment. Figure 11 is a conceptual diagram showing the relationship between the focus position and left and right image misalignment. Parts (a) to (c) of Figure 11 respectively show the front-focused, just-focused, and back-focused states, and it can be seen that the direction of image misalignment is opposite for front-focused and back-focused. The amount of image misalignment corresponds to the amount of focus misalignment. Note that "just-focused" refers to a state in which the focus is focused on the target position and the focal point is located on the light-receiving surface of the image sensor, "front-focused" refers to a state in which the focal point is located in front of the light-receiving surface, and "back-focused" refers to a state in which the focal point is located behind the light-receiving surface. Note that the "jas" in "jas" is an abbreviation of "just" and the "pin" is an abbreviation of "pint," and "jas" can be expressed in English as "just-focused" or "in perfect focus."

[0079] Furthermore, the positional relationship between the blur of a subject (here, a point light source) captured by a sensor with phase-difference pixels and the phase-difference image is as shown in Figure 12. Parts (a) (top row) to (c) of Figure 12 are subject images at each focus position for a normal image, phase-difference A (left phase-difference image), and phase-difference B (right phase-difference image), respectively. As the focus moves away from the focus point, both the color image (the image generated from the signal values ​​of the normal pixels) and the phase-difference image (the image generated from the signal values ​​of the phase-difference pixels) become blurred, and as described above with reference to Figure 11, the phase-difference image shift direction reverses depending on whether the focus is front or back (it becomes symmetrical about the center position of the original image). Furthermore, in Figure 12, differences in the contour shape of the subject image are apparent due to the different number of pixels between the normal pixels and the phase-difference pixels.

[0080] In an image plane phase difference sensor such as the image sensor 202, the output difference (sensitivity difference) between left and right phase difference images becomes large at the edges of the screen due to the sensor characteristics, as shown in FIG. 13 . For this reason, when performing correlation calculations for phase difference AF, sensitivity ratio correction is performed so that the signals from the left and right phase difference images are at the same level for each AF area before the correlation calculations are performed. However, a difference in signal-to-noise ratio (SN ratio) occurs between the left and right phase difference images depending on the sensor characteristics. For this reason, the correlation performance between the left and right phase difference images tends to decrease at the periphery of the sensor.

[0081] Here, the output difference between the left and right phase-contrast images differs depending on the distance from the center of the sensor with respect to normal pixels, as shown in FIG. 13 . Therefore, if stereo matching is performed between a color image and one of the left and right phase-contrast images with a higher output (signal value) depending on the pixel position, the baseline length will be shorter than when stereo matching is performed between the left and right phase-contrast images, but it will be possible to match images with a high S / N ratio. FIG. 14 shows such stereo matching in Example 3. In Example 3, the relationship information acquisition unit 226 (processor) inputs color image data C1 and the phase-contrast image P2 with a higher output between the left and right phase-contrast images (first phase-contrast image data, second phase-contrast image data) determined depending on the pixel position into a neural network NN3 (estimator) to acquire a two-dimensional distance map M3 (parallax map; information related to distance, information indicating the relationship between the first to third data).

[0082] Here, "which of the left and right phase-contrast images has a higher output" can be determined based on the relationship shown in Fig. 13, and the relationship information acquisition unit 226 selects the phase-contrast image A or B, which has a higher output, depending on the area of ​​the image sensor 202, and inputs it to the neural network NN3. The neural network NN3 (estimator) is an estimator constructed using a machine learning algorithm, and outputs a distance map M3 (or a parallax map) similar to the distance maps M1 and M2 to an output device. The neural network NN3 can be configured in the same way as the neural networks NN1 and NN2 described above.

[0083] According to the third embodiment, stereo matching can be performed using a combination of images with a high S / N ratio, and therefore matching performance can be improved in scenes where the S / N ratio is an issue, such as low contrast or low brightness.

[0084] [Example 4] If the relationship information acquisition unit 226 (processor) compares the stereo matching results of the RGB image (color image) and the phase-contrast images A and B in Example 3, it becomes possible to determine the reliability of the depth map (distance map) taking into account the characteristics of the on-chip phase-contrast sensor (image sensor 202). For example, as shown in part (a) of Figure 15, if the matching results of "RGB image + phase-contrast image A" and "RGB image + phase-contrast image B" are nearly symmetrical (if the subject image shift direction is opposite and the shift amount is approximately the same), the accuracy (reliability) of the depth map is sufficiently high. On the other hand, as shown in part (b) of the same figure, if the matching results of "RGB image + phase-contrast image A" and "RGB image + phase-contrast image B" are "symmetrical in the direction of shift but asymmetric in amount," the reliability is moderate, although there is an influence of the signal-to-noise ratio. However, as shown in part (c) of the same figure, if the matching result of "RGB image + phase-contrast image A" and the matching result of "RGB image + phase-contrast image B" are shifted in the same direction, which is an optically impossible result, it can be said to be "unreliable (low)." The breakdown of symmetry as shown in parts (b) and (c) of Figure 15 can occur when there is a large difference in the signal-to-noise ratio between phase-contrast images A and B, or when there is a subject with a repeating pattern (similar matching points are nearby). For convenience, in Figure 15, stereo matching is referred to as "SM," "phase-contrast images A and B" are referred to as "phase-contrast A and B," and "RGB image" is referred to as "RGB."

[0085] 16 is a diagram showing how a final distance map is obtained based on the symmetry determination result. The relationship information acquisition unit 226 repeats the above-described symmetry determination for all pixels of the image sensor 202, thereby obtaining a reliability map M6 of the distance information. The relationship information acquisition unit 226 (processor) may obtain a final distance map M7 from the distance maps M4 and M5 obtained by matching the RGB image (color image) with the phase-contrast images A and B, respectively, and the reliability map M6 (combining maps according to the reliability at each pixel position). Alternatively, the final distance map may be obtained by integrating the reliability map M6 obtained in Example 4 with a distance map obtained by a method other than the fourth aspect (for example, a distance map obtained by the method according to Examples 1 to 3 or from actual measurements using LiDAR, etc.).

[0086] As described above, according to Example 4, stereo matching is performed using a combination of an RGB image and a phase-contrast image, and the matching result using phase-contrast image A is compared with the matching result using phase-contrast image B, thereby making it possible to determine the reliability taking into account the characteristics of the image sensor 202 (image-plane phase-contrast sensor), and also to obtain a distance map taking into account the reliability.

[0087] [Example 5] The reliability determination performed in Example 4 may be performed by an AI (determiner). In this case, in addition to a mode in which matching is performed using the combinations of "RGB image + phase-contrast image A" and "RGB image + phase-contrast image B," a mode in which the RGB image, phase-contrast image A, and phase-contrast image B are input to an AI (neural network or the like) constituting a determiner, as shown in Fig. 17, may also be considered.

[0088] [Focus Control Using Distance Map] The optical system driver 230 (processor) uses the distance map (phase difference map) generated by the above-described method to determine a focus position according to the data acquisition conditions of the first and second phase difference image data, and can perform focus control (phase difference AF) by driving the focus lens 104 of the interchangeable lens 100 to the focus position via the lens driver 110. The optical system driver 230 may set a focus area on the distance map and determine the focus position based on the distribution of phase difference amounts in the focus area. The position, number, and shape of the focus area are not particularly limited, and there may be one or more. Furthermore, the position and size of the focus area may be variable, and an area containing a specific subject (e.g., a person or other specified subject) may be set as the focus area.

[0089] [Variations of Pixel Arrangement] In the first embodiment, a phase-difference pixel is provided in a portion of the image sensor, and the phase-difference pixel is formed by shielding one side of the light-receiving portion. However, it is also possible to provide the phase-difference pixel function to a normal color pixel, or to form the phase-difference pixel without shielding. For example, as shown in FIG. 18 , the light-receiving portion of the color pixel 207 can be divided into two, color pixel 207-1 and color pixel 207-2, and the signal values ​​of these color pixels can be extracted separately to use the color pixels 207-1 and 207-2 as two phase-difference pixels. Furthermore, by adding the signal value of color pixel 207-1 and the signal value of color pixel 207-2, the color pixel 207 can also be used as a normal color pixel. All pixels of the image sensor may be configured as such "pixels that function as both color pixels and phase-difference pixels" (full-surface phase difference). Even with this pixel arrangement, color image data, phase-difference image data, distance information, and the like can be acquired in the same manner as in the above-described embodiment.

[0090] 18, a common microlens (not shown) is provided for the color pixels 207-1 and 207-2. In addition, although the light receiving section is divided in the left-right direction in the example of Fig. 18, the light receiving section may be divided in the up-down direction.

[0091] 5, the pitch of the phase difference pixels of the image sensor 202 is wider than the pitch of the RGB pixels (color pixels, normal pixels), but this is because an image plane phase difference sensor is generally configured as "RGB pixels + left phase difference pixels + right phase difference pixels," and therefore the number of phase difference pixels themselves is smaller than that of RGB pixels. This is also because the lines on which phase difference pixels exist are thinned out in the vertical direction, for example, to one out of every 12 lines.

[0092] In contrast to this, in the above-described modified example, one pixel is divided, so the situation is slightly different, and a division structure is formed in which all pixels correspond to left and right phase difference pixels as shown in Fig. 18. However, even when there is a phase difference in all pixels, if AF or the like is controlled using a wide range of output, such as using the output of all pixels, the calculation time becomes extremely long, so in practice, vertical thinning or the like may be performed.

[0093] Here, in order to determine the reliability of the phase difference pixels, if an RGB image plus a phase difference image is used, in a sensor with vertical thinning, the number of lines with phase difference is, for example, one in 12 lines, so it is necessary to thin out or resize the RGB pixels in accordance with the number of phase difference lines.

[0094] Even when such thinning or resizing is performed, it is preferable to change the processing depending on how the RGB pixels are used, as in the above-described Examples 1 to 4, as in the following methods 1 and 2. Note that thinning or resizing of the RGB pixels can be performed by the color image data acquisition unit 222 (processor) and / or the relationship information acquisition unit 226 (processor).

[0095] (Method 1) Method of Cutting Out the Vicinity of Phase Difference Pixel As in the above-described Examples 1, 3, and 4, when calculating the phase difference amount by directly associating the phase difference pixel with the RGB pixel, it is better to use pixels that are as close as possible to form a phase difference image, and therefore it is better to use only pixels near the phase difference pixel.

[0096] (Method 2) Method of Resizing After Image Processing On the other hand, as in Example 2, when RGB pixels are first subjected to processing such as segmentation and then associated with phase difference pixels, if processing such as segmentation is performed after vertical thinning as in the case of a phase difference image, the accuracy of boundary determination will decrease. Therefore, a procedure of "segmenting the image while keeping it at a large size, and then resizing or thinning will be performed in that order" is more suitable.

[0097] [Switching between thinning and resizing depending on conditions such as S / N ratio] Thinning uses data from the nearest pixel, which is thought to enable a more accurate calculation of the amount of image misalignment in stereo matching than resizing and including information from surrounding pixels. On the other hand, when the S / N ratio is insufficient due to low contrast or high noise conditions, the RGB information added to the phase contrast image is also less reliable. In such cases, it is expected that "resizing will provide better accuracy than thinning."

[0098] 19 is a diagram showing how processing is switched depending on whether the conditions for degradation of the S / N ratio are met. In the example shown in the figure, if the conditions for degradation of the S / N ratio are met (the determination result is YES), averaging (resizing) is performed. This is because, when the S / N ratio is insufficient (low), adding data vertically or performing averaging resizing is considered an effective way to increase the S / N ratio of the RGB image. On the other hand, if the conditions for degradation of the S / N ratio are not met (the determination result is NO), thinning is performed.

[0099] [Similar Example of Resizing] The input for segmentation may be an undecimated image, and the resizing effect may be achieved by using AI that simultaneously performs segmentation and resizing and outputs the resulting image. Figure 20 shows an example of AI that performs segmentation and vertical 1 / 12 resizing. Such AI can be constructed using a machine learning algorithm such as a neural network. In this case, images with different S / N ratios must be provided to the AI ​​as training data to simultaneously learn the segmentation effect and the noise reduction effect.

[0100] [Effects of Switching Thinning and Resizing Methods] As described above, in the case of a method of switching processing depending on conditions such as the S / N ratio, the S / N ratio of RGB data can be improved, and the reliability determination process can be performed stably. Furthermore, in the case of a similar example of resizing, by performing the resizing process using AI simultaneously with processing such as segmentation, the S / N ratio of RGB data can be improved, and the reliability determination process can be performed stably.

[0101] Although the embodiment and modifications of the present invention have been described above, the present invention is not limited to the above-described aspects and various modifications are possible.

[0102] REFERENCE SIGNS LIST 10 Imaging device 100 Interchangeable lens 102 Zoom lens 104 Focus lens 110 Lens driving unit 200 Imaging device body 201 Phase difference pixel 201A Opening 201B Mask 201C Mask 201D Opening 202 Imaging element 202A Mask 202B Color pixel 202G Color pixel 202R Color pixel 202X Phase difference pixel 202Y Phase difference pixel 203 Phase difference pixel 203A Opening 203B Mask 203C Mask 203D Opening 206 A / D converter 207 Color pixel 207-1 Color pixel 207-2 Color pixel 210 Data processing unit 220 Processor 222 Color image data acquisition unit 224 Phase difference image data acquisition unit 226 Relationship information acquisition unit 230 Optical system driving unit 234 Input / output control unit 260 Operation unit 270 Recording unit 280 Monitor 300 External device

Claims

1. A data processing device having a processor, wherein the processor acquires, from an imaging element of an imaging device, signal values of color pixels having color filters arranged in an area including specific pixels of the imaging element as first data, acquires, in the area, signal values of first phase difference pixels that are pixels in which one side of a light receiving portion is shielded as second data, acquires, in the area, signal values of second phase difference pixels that are pixels in which the side opposite to the one side of the light receiving portion is shielded as third data, and acquires information indicating the relationship between the first data, the second data, and the third data based on a first feature acquired based on the first data and the second data, and a second feature acquired based on the first data and the third data.

2. The data processing device according to claim 1, wherein the processor acquires the first feature based on color image data generated from the signal values of the color pixels and first phase difference image data generated from the signal values of the first phase difference pixels, and acquires the second feature based on the color image data and second phase difference image data generated from the signal values of the second phase difference pixels.

3. The data processing device according to claim 2, wherein the processor acquires the first feature based on a recognition result of a target pixel in a color image represented by the color image data, and acquires the second feature based on the recognition result and the second phase difference image data.

4. A data processing device as described in claim 2 or 3, wherein the processor determines, based on the position of the specific pixel on the imaging element, whether to use the first phase-difference image data or the second phase-difference image data to obtain information indicating the relationship at the position, and, based on the result of the determination, generates information indicating the relationship at the position using the color image data and one of the first phase-difference image data and the second phase-difference image data.

5. A data processing device according to claim 4, wherein the processor acquires information indicating the relationship at the position using the phase difference image data having a higher signal value at the position between the first phase difference image data and the second phase difference image data.

6. The data processing device according to any one of claims 1 to 3, wherein the processor acquires information relating to the distance from the imaging device to the subject as information indicating the relationship.

7. The data processing device according to claim 6, wherein the processor acquires information indicating the reliability of the distance as information relating to the distance.

8. The data processing device according to claim 6, wherein the processor obtains information about the distance using an estimator constructed using a machine learning algorithm.

9. The data processing device according to any one of claims 1 to 3, wherein the processor causes an output device to output information indicating the relationship as a two-dimensional map.

10. A data processing device according to claim 2 or 3, wherein the color filters comprise a red color filter that transmits light in the red wavelength band, a green color filter that transmits light in the green wavelength band, and a blue color filter that transmits light in the blue wavelength band, and the processor generates the color image data based on signal values in the red wavelength band, signal values in the green wavelength band, and signal values in the blue wavelength band.

11. A data processing device according to claim 10, wherein the color filters further comprise a white color filter that transmits light in a white wavelength band, and the processor generates the color image data based on signal values in the red wavelength band, signal values in the green wavelength band, signal values in the blue wavelength band, and light in the white wavelength band.

12. An imaging device main body comprising the data processing device of any one of claims 1 to 3 and the imaging element, wherein the imaging element comprises color pixels arranged two-dimensionally and having color filters disposed thereon, first phase-difference pixels which are two-dimensionally arranged pixels not having color filters disposed thereon and which are pixels having one side shielded, and second phase-difference pixels which are two-dimensionally arranged pixels not having color filters disposed thereon and which are pixels having the one side and the side opposite to the optical axis of the imaging element shielded, and wherein the processor acquires color image data generated from signal values of the color pixels as the first data, acquires first phase-difference image data generated from signal values of the first phase-difference pixels as the second data, and acquires second phase-difference image data generated from signal values of the second phase-difference pixels as the third data.

13. The imaging device body according to claim 12, wherein the arrangement pitch of the first phase difference pixels and the second phase difference pixels in the imaging element is wider than the arrangement pitch of the color pixels in the imaging element.

14. The imaging device body of claim 12, wherein the processor adjusts the size of the image represented by the color image data to the size of the image represented by the first phase-difference image data and / or the size of the image represented by the second phase-difference image data based on the relationship between the arrangement pitch of the color pixels and the arrangement pitch of the first phase-difference pixels and the second phase-difference pixels.

15. An imaging device comprising: an imaging device body according to claim 12; and a single optical system attachable to said imaging device body for forming an optical image of a subject on said imaging element, said imaging element being a single imaging element.

16. A method for operating a data processing device having a processor, wherein the processor acquires, from an imaging element of an imaging device, signal values of color pixels having color filters arranged in an area including specific pixels of the imaging element as first data, acquires, in the area, signal values of first phase difference pixels which are pixels having one side of their light receiving units shielded as second data, acquires, in the area, signal values of second phase difference pixels which are pixels having the side of their light receiving units opposite to the one side shielded as third data, and acquires information indicating the relationship between the first data, the second data, and the third data based on a first feature acquired based on the first data and the second data, and a second feature acquired based on the first data and the third data.

Images