Image data processing device, image data processing method, image data processing program, and imaging system

Abstract

Provided are an image data processing device, an image data processing method, an image data processing program, and an imaging system with which it is possible to generate high-quality multispectral images. The image data processing device processes image data obtained by an imaging device comprising: an optical system for separating incident light into a plurality of wavelengths and outputting the light of the separated wavelengths after polarizing the light in a specific direction; and an imaging element comprising a plurality of sets of pixels having mutually different types of polarizers. The image data processing device performs: a process for acquiring image data; a process for detecting, from the acquired image data, a pixel with a pixel value outside a predetermined range as an anomalous pixel; a process for, if an anomalous pixel is detected, correcting the pixel value of the anomalous pixel on the basis of the pixel value of a pixel around the anomalous pixel; and a process for, if an anomalous pixel is detected, generating an image of a wavelength separated from image data after the pixel value of the anomalous pixel is corrected.

Description

[Technical Field]

[0001] The present invention relates to an image data processing device, an image data processing method, an image data processing program, and an imaging system, and more particularly to an image data processing device, an image data processing method, an image data processing program, and an imaging system that generate multispectral images. [Background technology]

[0002] Patent Document 1 describes a technology for simultaneously capturing images corresponding to each optical system of the imaging lens by using an imaging lens equipped with multiple optical systems with different imaging characteristics and an imaging element in which each pixel has directivity with respect to the angle of incidence of light. In Patent Document 1, a predetermined interference removal process is performed on image data output from the imaging element to generate an image corresponding to each optical system. In addition, Patent Document 1 changes the content of the interference removal process depending on whether or not there are saturated pixels.

[0003] Patent Document 2 describes a technique for detecting defective pixels by processing image data captured by a so-called polarization imaging element. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 130581 [Patent Document 2] International Publication No. 2018 / 042815 Summary of the Invention

[0005] One embodiment of the technique of the present disclosure provides an image data processing device, an image data processing method, an image data processing program, and an imaging system that are capable of generating high-quality multispectral images. [Means for solving the problem]

[0006] (1) An image data processing device that processes image data captured by an imaging device that includes an optical system that splits incident light into multiple wavelengths and polarizes the split wavelength light in a specific direction before emitting it, and an imaging element that includes multiple sets of pixels with different types of polarizers, and that includes a processor that performs the following processes: acquiring image data; detecting pixels from the acquired image data whose pixel values ​​fall outside a predetermined range as abnormal pixels; correcting the pixel values ​​of the abnormal pixels based on the pixel values ​​of surrounding pixels when an abnormal pixel is detected; and generating an image of the split wavelengths from the image data after correcting the pixel values ​​of the abnormal pixels when an abnormal pixel is detected.

[0007] (2) The image data processing device of (1), wherein the processor performs a process to correct the pixel value of the abnormal pixel based on the pixel values ​​of surrounding pixels when the pixel value falls outside a predetermined range due to saturation.

[0008] (3) An image data processing device according to (1), wherein the processor, when a pixel value falls outside a predetermined range due to a malfunction, performs processing to generate an image of spectral wavelengths from image data excluding the abnormal pixel without correcting the pixel value of the abnormal pixel.

[0009] (4) An image data processing device according to any one of (1) to (3), wherein the process of detecting an abnormal pixel includes a process of detecting a set of pixels including an abnormal pixel, and a process of identifying the abnormal pixel from the detected set of pixels.

[0010] (5) The image data processing device of (4), wherein the process of detecting a set of pixels including an abnormal pixel detects the set of pixels including an abnormal pixel based on pixel values ​​of pixels that make up the set of pixels.

[0011] (6) The image data processing device of (5) detects a set of pixels including an abnormal pixel by calculating the sum of the pixel values ​​of the pixels that make up the set of pixels, or the sum of the values ​​obtained by multiplying the pixel values ​​of the pixels that make up the set of pixels by a specific coefficient, and detects a set of pixels whose calculated sum is equal to or greater than a first threshold as a set of pixels including an abnormal pixel.

[0012] (7) An image data processing device according to (5) or (6), in which the process of identifying abnormal pixels involves extracting pixels whose pixel values ​​are equal to or less than a second threshold value and / or pixels whose pixel values ​​are saturated from a set of pixels including the abnormal pixels, thereby identifying the abnormal pixels.

[0013] (8) An image data processing device according to (5) or (6), wherein the process of identifying an abnormal pixel identifies the abnormal pixel based on pixel values ​​of pixels surrounding a set of pixels including the abnormal pixel.

[0014] (9) The image data processing device of (8), wherein the process of identifying abnormal pixels includes a process of detecting a set of pixels including abnormal pixels from among sets of pixels surrounding a set of pixels including abnormal pixels, and a process of identifying abnormal pixels from within the set of pixels including abnormal pixels based on the detection result of the set of pixels including abnormal pixels.

[0015] (10) The image data processing device of (9), in which the range for detecting a set of pixels including abnormal pixels is switched according to the resolution of the optical system.

[0016] (11) The image data processing device of (8), wherein the process of identifying an abnormal pixel includes a process of estimating the pixel value of a pixel from the pixel values ​​of surrounding pixels, and a process of identifying a pixel whose difference from the estimated pixel value is equal to or greater than a third threshold as an abnormal pixel.

[0017] (12) The image data processing device of (11), wherein the process of estimating the pixel value of a pixel from the pixel values ​​of surrounding pixels estimates the pixel value from the pixel values ​​of surrounding pixels equipped with the same type of polarizer.

[0018] (13) An image data processing method for processing image data captured by an imaging device equipped with an optical system that splits incident light into multiple wavelengths and polarizes the split wavelength light in a specific direction before emitting it, and an imaging element that has multiple sets of pixels with different types of polarizers, the image data processing method including: a process for acquiring image data; a process for detecting, from the acquired image data, pixels whose pixel values ​​fall outside a predetermined range as abnormal pixels; a process for correcting the pixel values ​​of the abnormal pixels based on the pixel values ​​of surrounding pixels when an abnormal pixel is detected; and a process for generating an image of the split wavelengths from the image data after correcting the pixel values ​​of the abnormal pixels when an abnormal pixel is detected.

[0019] (14) An image data processing program for processing image data captured by an imaging device that includes an optical system that splits incident light into multiple wavelengths and polarizes the split wavelength light in a specific direction before emitting it, and an imaging element that includes multiple sets of pixels with different types of polarizers, the image data processing program causing a computer to realize the following functions: acquiring image data; detecting pixels from the acquired image data whose pixel values ​​fall outside a predetermined range as abnormal pixels; correcting the pixel values ​​of the abnormal pixels based on the pixel values ​​of surrounding pixels when an abnormal pixel is detected; and generating an image of the split wavelengths from the image data after correcting the pixel values ​​of the abnormal pixels when an abnormal pixel is detected.

[0020] (15) An imaging system comprising an imaging device having an optical system that separates incident light into multiple wavelengths and polarizes the separated wavelength light in a specific direction and outputs it, and an imaging element having multiple sets of pixels with different types of polarizers, and an image data processing device of any one of (1) to (12) that processes image data captured by the imaging device. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a multispectral camera system to which the present invention is applied. [Figure 2] An exploded view showing the schematic configuration of the filter unit [Figure 3] FIG. 1 is a diagram showing an example of the arrangement of pixels and polarizers in an image sensor; [Figure 4] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image data processing device. [Figure 5] Block diagram of functions realized by the image data processing device [Figure 6] Flowchart showing the procedure for detecting abnormal pixels [Figure 7] Flowchart showing the image data processing procedure [Figure 8] 1 is a flowchart showing the procedure for detecting and correcting abnormal pixels. [Figure 9] FIG. 1 is a diagram showing an outline of a conventional interference removal process. [Figure 10] FIG. 1 is a diagram showing an outline of a process for removing interference using the method of the first embodiment; [Figure 11] Block diagram of functions realized by the image data processing device [Figure 12] FIG. 10 is a diagram showing an outline of a process for removing interference using a second method. [Figure 13] 1 is a flowchart showing a procedure for processing image data by an image data processing device. [Figure 14] Conceptual diagram of how to identify abnormal pixels using a set of surrounding pixels [Figure 15] A diagram showing eight peripheral pixel sets [Figure 16] Conceptual diagram of identifying abnormal pixels by expanding the range of the reference pixel set [Figure 17] A diagram showing eight peripheral pixel sets [Figure 18] A diagram showing the surrounding 16 pixel sets DETAILED DESCRIPTION OF THE INVENTION

[0022] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] [First embodiment] [Multispectral camera system] FIG. 1 is a diagram showing a schematic configuration of a multispectral camera system to which the present invention is applied.

[0024] A multispectral camera system simultaneously captures images separated into multiple wavelengths, and the captured images are called multispectral images.

[0025] The multispectral camera system 1 shown in Figure 1 is a so-called polarization-based multispectral camera system, and shows an example of a system that captures images separated into three wavelengths. The polarization-based system is a multispectral camera system that uses polarized light.

[0026] 1, the multispectral camera system 1 of this embodiment is mainly composed of a multispectral camera 10 and an image data processing device 300. The multispectral camera system 1 is an example of an imaging system.

[0027] [Multispectral camera] The multispectral camera 10 of this embodiment is mainly composed of a lens device 100 and a camera body 200. The multispectral camera 10 is an example of an imaging device.

[0028] [Lens device] Lens device 100 splits incident light into multiple wavelengths, and polarizes the split wavelengths in specific directions before emitting the light. In this embodiment, the lens device 100 splits incident light into three wavelengths. Lens device 100 is an example of an optical system.

[0029] As shown in FIG. 1, the lens device 100 includes a plurality of lens groups 110A and 110B and a filter unit 120.

[0030] Each of the lens groups 110A and 110B is composed of at least one lens. For convenience, only two lens groups 110A and 110B are shown in FIG. 1. Hereinafter, as necessary, the two lens groups 110A and 110B will be distinguished by referring to the lens group 110A arranged in front of the filter unit 120 as the first lens group and the lens group 110B arranged in back of the filter unit 120 as the second lens group. Note that "front side" means "object side" and "rear side" means "image side."

[0031] The filter unit 120 is disposed in the optical path. More specifically, the filter unit 120 is disposed at or near the pupil position of the lens apparatus 100.

[0032] The vicinity of the pupil position refers to an area that satisfies the following formula:

[0033] |d|<φ / (2tanθ) Here, θ is the maximum chief ray angle at the pupil position (the chief ray angle is the angle with the optical axis), φ is the pupil diameter, and |d| is the distance from the pupil position.

[0034] FIG. 2 is a development view showing a schematic configuration of the filter unit.

[0035] The filter unit 120 is composed of a filter frame 122 having a plurality of windows (openings), and a plurality of filters (optical elements) attached to the respective windows of the filter frame 122.

[0036] 2, the filter frame 122 of this embodiment has a disk-like shape and three window portions 122A, 122B, and 122C. The three window portions 122A, 122B, and 122C are formed by circular openings and are arranged at equal intervals along the circumferential direction. Hereinafter, as necessary, the window portion designated by reference numeral 122A will be referred to as the first window portion, the window portion designated by reference numeral 122B as the second window portion, and the window portion designated by reference numeral 122C as the third window portion to distinguish between the window portions 122A, 122B, and 122C.

[0037] Two filters are attached to each of the three windows 122A, 122B, and 122C. The two filters are band-pass filters (BPFs) 123A, 123B, and 123C, and polarized light filters (PLFs) 124A, 124B, and 124C.

[0038] Bandpass filters 123A, 123B, and 123C that transmit different wavelength ranges of light are attached to the three window portions 122A, 122B, and 122C, respectively. The transmission wavelength ranges of the bandpass filters 123A, 123B, and 123C attached to the three window portions 122A, 122B, and 122C correspond to the wavelength ranges of the three images to be captured. In other words, they correspond to the wavelength ranges of the multispectral images to be captured.

[0039] A bandpass filter 123A that transmits light in a first wavelength band λ1 is attached to the first window portion 122A. Hereinafter, as necessary, the bandpass filter 123A attached to the first window portion 122A will be referred to as the first bandpass filter 123A to distinguish it from other bandpass filters.

[0040] A bandpass filter 123B that transmits light in the second wavelength band λ2 is attached to the second window portion 122B. Hereinafter, as necessary, the bandpass filter 123B attached to the second window portion 122B will be referred to as the second bandpass filter 123B to distinguish it from the other bandpass filters.

[0041] A bandpass filter 123C that transmits light in a third wavelength band λ3 is attached to the third window portion 122C. Hereinafter, as necessary, the bandpass filter 123C attached to the third window portion 122C will be referred to as the third bandpass filter 123C to distinguish it from the other bandpass filters.

[0042] It is preferable to use reflective bandpass filters for the bandpass filters 123A, 123B, and 123C in view of the high degree of freedom in spectral transmission characteristics.

[0043] Polarizing filters 124A, 124B, and 124C having different angles of the transmission axis (transmission axis orientations) are attached to the three windows 122A, 122B, and 122C, respectively.

[0044] A polarizing filter 124A whose transmission axis is set at a first angle α1 (first direction) is attached to the first window portion 122A. As an example, in the lens device 100 of the present embodiment, a polarizing filter 124A whose transmission axis is set at 0° is attached. Hereinafter, as necessary, the polarizing filter 124A attached to the first window portion 122A will be referred to as the first polarizing filter 124A to distinguish it from other polarizing filters.

[0045] A polarizing filter 124B having a transmission axis set at a second angle α2 (second direction) is attached to the second window portion 122B. As an example, in the lens device 100 of the present embodiment, a polarizing filter 124B having a transmission axis set at 60° is attached. Hereinafter, as necessary, the polarizing filter 124B attached to the second window portion 122B will be referred to as the second polarizing filter 124B to distinguish it from the other polarizing filters.

[0046] A polarizing filter 124C having a transmission axis set at a third angle α3 (third direction) is attached to the third window portion 122C. As an example, in the lens device 100 of the present embodiment, a polarizing filter 124A having a transmission axis set at 120° is attached. Hereinafter, as necessary, the polarizing filter 124C attached to the third window portion 122C will be referred to as the third polarizing filter 124C to distinguish it from the other polarizing filters.

[0047] The angle of the transmission axis is defined as 0° when it is parallel to the X-axis, and the counterclockwise direction when viewed from the object side (front side) is defined as the positive (+) direction. Therefore, a transmission axis of 60° means that it is tilted 60° counterclockwise with respect to the X-axis. Furthermore, a transmission axis of 120° means that it is tilted 120° counterclockwise with respect to the X-axis.

[0048] The X-axis is an axis set in a plane perpendicular to the optical axis Z. In the plane perpendicular to the optical axis Z, the axis perpendicular to the X-axis is defined as the Y-axis. As will be described later, the image sensor provided in the camera body 200 has its upper and lower sides of the light receiving surface arranged parallel to the X-axis. Furthermore, its left and right sides are arranged parallel to the Y-axis.

[0049] It is preferable to use absorption type polarizing filters 124A, 124B, and 124C in order to suppress ghosting.

[0050] With the above configuration, light incident on lens device 100 is split into three wavelengths as it passes through filter unit 120, and each wavelength is polarized into light with a specific vibration direction before it is emitted. Specifically, the light is split into light in a first wavelength range λ1 polarized in a first direction, light in a second wavelength range λ2 polarized in a second direction, and light in a third wavelength range λ3 polarized in a third direction.

[0051] [Camera body] As shown in FIG. 1, the camera body 200 has an image sensor 210. The image sensor 210 is disposed on the optical axis of the lens device 100 and receives light that has passed through the lens device 100. This image sensor 210 is configured as a so-called polarization image sensor. A polarization image sensor is an image sensor equipped with a polarizer, and a polarizer is provided for each pixel. The polarizer is provided, for example, between a microlens and a photodiode. Note that this type of polarization image sensor is well known, and therefore a detailed description thereof will be omitted (see, for example, International Publication No. 2020 / 071253, etc.).

[0052] The type of polarizer (angle of the transmission axis) mounted on the image sensor 210 is selected depending on the number of wavelengths to be captured. When capturing an image separated into three wavelengths, a polarization image sensor equipped with polarizers in at least three directions is used. In this embodiment, a polarization image sensor equipped with polarizers in four directions is used.

[0053] FIG. 3 is a diagram showing an example of the arrangement of pixels and polarizers in an imaging element.

[0054] As shown in the figure, four polarizers with different transmission axis angles are regularly arranged with respect to the pixels arranged in a matrix. The polarizer with a transmission axis angle of β1 is referred to as the first polarizer, the polarizer with a transmission axis angle of β2 is referred to as the second polarizer, the polarizer with a transmission axis angle of β3 is referred to as the third polarizer, and the polarizer with a transmission axis angle of β4 is referred to as the fourth polarizer. As an example, in this embodiment, the transmission axis angle β1 of the first polarizer is set to 0°, the transmission axis angle β2 of the second polarizer is set to 45°, the transmission axis angle β3 of the third polarizer is set to 90°, and the transmission axis angle β4 of the fourth polarizer is set to 135°.

[0055] The pixel P1 equipped with the first polarizer is the first pixel, the pixel P2 equipped with the second polarizer is the second pixel, the pixel P3 equipped with the third polarizer is the third pixel, and the pixel P4 equipped with the fourth polarizer is the fourth pixel. A 2x2 pixel array consisting of the first pixel P1, the second pixel P2, the third pixel P3, and the fourth pixel P4 is one pixel set SP, and this pixel set SP is repeatedly arranged along the X-axis and the Y-axis.

[0056] In this way, an image sensor equipped with a four-directional polarizer can capture a four-directional polarized image in one shot.

[0057] The image sensor 210 is configured, for example, as a CMOS (Complementary Metal Oxide Semiconductor) type equipped with a drive unit, an ADC (Analog to Digital Converter), a signal processing unit, and the like. In this case, the image sensor 210 is driven to operate by the built-in drive unit. The signal of each pixel is converted into a digital signal by the built-in ADC and output. Furthermore, the signal of each pixel is subjected to correlated double sampling processing, gain processing, correction processing, and the like by the built-in signal processing unit and then output. The signal processing may be performed after conversion into a digital signal, or may be performed before conversion into a digital signal.

[0058] The output Vout (signal value of each pixel) of the image sensor 210 is expressed by the following equation, for example.

[0059] Vout = (Vin - Vth) × Gain Here, Vin is the voltage generated by the incidence of light, Vth is the threshold voltage, and Gain is the gain.

[0060] In the case of 8-bit, the Vout range is 0 to 255. Therefore, if the incident light is too strong, the pixel will saturate. For example, in the case of 8-bit, even if the output voltage is 255 or higher, it will all be expressed as 255. Also, for a faulty pixel, the output of that pixel will be 0 or a value close to 0.

[0061] In addition to the image sensor 210, the camera body 200 is equipped with an output unit (not shown) that outputs data of an image captured by the image sensor 210, a camera control unit (not shown) that controls the overall operation of the camera body 200, and the like. The camera control unit is configured, for example, with a microprocessing unit (MPU) that includes a processor and memory. The microprocessing unit functions as the camera control unit by executing a predetermined control program.

[0062] The image data output from the camera body 200 is what is known as RAW image data. That is, it is unprocessed image data. This RAW image data is processed by the image data processing device 300, and an image is generated that is spectrally separated into multiple wavelengths.

[0063] [Image data processing device] The image data processing device 300 processes image data (RAW image data) output from the camera body 200 of the multispectral camera 10 to generate an image dispersed into a plurality of wavelengths. Specifically, the image data processing device 300 generates images of wavelengths corresponding to the transmission wavelength bands λ1, λ2, and λ3 of the bandpass filters 123A, 123B, and 123C attached to the respective window portions 122A, 122B, and 122C of the filter unit 120 built into the lens device 100.

[0064] FIG. 4 is a diagram illustrating an example of a hardware configuration of an image data processing device.

[0065] As shown in the figure, the image data processing device 300 includes a CPU (Central Processing Unit) 311, a ROM (Read Only Memory) 312, a RAM (Random Access Memory) 313, an auxiliary storage device 314, an input device 315, an output device 316, and an input / output interface (I / F) 317. Such an image data processing device 300 is configured, for example, by a general-purpose computer such as a personal computer.

[0066] The image data processing device 300 functions as an image data processing device when a CPU 311, which is a processor, executes a predetermined program (image data processing program). The program executed by the CPU 311 is stored in a ROM 312 or an auxiliary storage device 314.

[0067] The auxiliary storage device 314 constitutes a storage unit of the image data processing device 300. The auxiliary storage device 314 is constituted by, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like.

[0068] The input device 315 constitutes an operation unit of the image data processing device 300. The input device 315 is composed of, for example, a keyboard, a mouse, a touch panel, and the like.

[0069] The output device 316 constitutes a display unit of the image data processing device 300. The output device 316 is constituted by a display such as a liquid crystal display (Liquid Crystal Display) or an organic light emitting diode display (Organic EL Display).

[0070] The input / output interface 317 constitutes a connection unit of the image data processing device 300. The image data processing device 300 is connected to the camera body 200 of the multispectral camera 10 via the input / output interface 317.

[0071] FIG. 5 is a block diagram of functions realized by the image data processing device.

[0072] As shown in the figure, the image data processing device 300 realizes the functions of an image data acquisition unit 320, an abnormal pixel detection unit 321, a pixel value correction unit 322, an image generation unit 323, an output control unit 324, and a recording control unit 325. These functions are realized by the CPU 311 executing a predetermined program (image data processing program).

[0073] The image data acquisition unit 320 acquires image data obtained by capturing an image from the multispectral camera 10. As described above, the image data acquired from the multispectral camera 10 is RAW image data. The image data is acquired via the input / output interface 317.

[0074] The abnormal pixel detection unit 321 analyzes the acquired image data and performs processing to detect abnormal pixels. Here, an abnormal pixel is a pixel whose pixel value is outside a predetermined range, or a pixel with an inappropriate brightness value. Abnormal pixels include faulty pixels and saturated pixels. A faulty pixel is a pixel whose pixel value is outside a predetermined range due to a fault. A saturated pixel is a pixel whose pixel value is outside a predetermined range due to saturation. A faulty pixel has a pixel value of 0 or a value close to 0. On the other hand, a saturated pixel has a pixel value that is a saturated value. For example, in the case of 8 bits, the pixel value is 255.

[0075] If the captured image data contains abnormal pixels, it will be impossible to generate appropriate images for each wavelength in the subsequent image generation unit 323. For this reason, the image data processing device 300 of this embodiment detects and corrects abnormal pixels in the captured image data to generate images for each wavelength.

[0076] The detection of abnormal pixels is performed in units of pixel sets SP.

[0077] FIG. 6 is a flowchart showing the procedure for detecting an abnormal pixel.

[0078] First, a process is performed to determine whether or not the target pixel set SP contains any abnormal pixels (step S1). Next, based on the results of this determination process, it is determined whether or not the target pixel set SP contains any abnormal pixels (step S2). If it is determined that the target pixel set SP does not contain any abnormal pixels, the process of detecting abnormal pixels for that pixel set is terminated. On the other hand, if it is determined that the target pixel set SP contains any abnormal pixels, a process is performed to identify abnormal pixels within the target pixel set SP (step S3).

[0079] The process of determining whether or not the pixel set SP of interest includes an abnormal pixel is performed based on the signal values ​​(pixel values) of the four pixels P1, P2, P3, and P4 that make up the pixel set SP.

[0080] As described above, the image sensor 210 used in the multispectral camera 10 of this embodiment is a polarization image sensor. In a polarization image sensor, the outputs of the pixels in each pixel set have a certain relationship. That is, a certain relationship is established between the output signals based on the angle of the transmission axis of the polarizer mounted in each pixel. For example, as in the image sensor 210 of this embodiment, when one pixel set SP is composed of four pixels P1 to P4 and has polarizers at 0°, 45°, 90°, and 135°, respectively, the relationship of the following formula (1) is established. However, it is assumed that the same amount of light is incident on each of the pixels P1 to P4.

[0081] x1+x3=x2+x4 …(1) Here, x1 is the pixel value of the first pixel P1, i.e., the pixel value of the pixel whose polarizer has a transmission axis angle of 0°. x2 is the pixel value of the second pixel P2, i.e., the pixel value of the pixel whose polarizer has a transmission axis angle of 45°. x3 is the pixel value of the third pixel P3, i.e., the pixel value of the pixel whose polarizer has a transmission axis angle of 90°. x4 is the pixel value of the fourth pixel P4, i.e., the pixel value of the pixel whose polarizer has a transmission axis angle of 135°.

[0082] That is, it has polarizers at 0°, 45°, 90°, and 135°. Ta In the image sensor, in each pixel set, the sum of pixel values ​​of pixels having polarizers that are orthogonal to each other is equal.

[0083] If the pixel set SP contains abnormal pixels, the relationship in the above formula (1) is significantly disrupted. The image data processing device 300 of this embodiment uses the relationship shown in the above formula (1) to determine whether the target pixel set SP contains abnormal pixels. Specifically, E is calculated using the following formula (2), and if E is equal to or greater than a threshold value Th1, it is determined that the target pixel set SP contains abnormal pixels. E is referred to as an intensity value. The threshold value Th1 is an example of a first threshold value. E=(x1+x3)-(x2+x4) …(2)

[0084] When the target pixel set SP includes abnormal pixels, the process of identifying the abnormal pixels within the pixel set SP is performed as follows: That is, pixels whose pixel values ​​are equal to or less than a threshold value Th2 and pixels whose pixel values ​​are saturated are extracted from the target pixel set SP, and the extracted pixels are identified as abnormal pixels.

[0085] Here, the process of extracting pixels whose pixel values ​​are equal to or less than threshold value Th2 is a process of extracting faulty pixels. A faulty pixel has an output value of 0 or close to 0. Therefore, pixels whose pixel values ​​are 0 or close to 0 are extracted to identify faulty pixels. Therefore, threshold value Th2 is set to a value of 0 or close to 0. Threshold value Th2 is an example of a second threshold value.

[0086] On the other hand, the process of extracting pixels with saturated pixel values ​​is a process of extracting saturated pixels. Since saturated pixels have saturated pixel values, the saturated pixels are extracted and identified. If the output of the image sensor 210 is 8 bits, the saturated value is 255.

[0087] When an abnormal pixel is detected, the pixel value correction unit 322 performs a process of correcting the pixel value of the abnormal pixel. The pixel value correction unit 322 corrects the pixel value of the abnormal pixel based on the pixel values ​​of pixels surrounding the abnormal pixel. In this embodiment, the pixel value of the abnormal pixel is corrected based on the pixel values ​​of other pixels in the pixel set in which the abnormal pixel is detected. In other words, since the pixel values ​​of pixels in the same pixel set have the relationship of equation (1) above, the pixel value of the abnormal pixel is estimated and corrected using the relationship of equation (1) above.

[0088] For example, suppose the pixel value x1 of the first pixel P1 is 180, the pixel value x2 of the second pixel P2 is 158.0385, the pixel value x3 of the third pixel P3 is 240, and the pixel value x4 of the fourth pixel P4 is 255. In this case, the pixel value x4 of the fourth pixel P4 is a saturated value, so the fourth pixel P4 is an abnormal pixel. From the above equation (1), the pixel value x4 of the fourth pixel P4 is calculated as x4 = x1 + x3 - x2. Therefore, in this case, the pixel value x4 of the fourth pixel P4 is corrected to 261.9615, from x4 = 180 + 240 - 158.0385.

[0089] The image generation unit 323 performs predetermined signal processing on image data obtained by imaging to generate images of multiple wavelengths. If pixel value correction processing has been performed, the image data after correction is subjected to the predetermined signal processing to generate images of multiple wavelengths. The images of multiple wavelengths are images of wavelengths dispersed by the lens device 100 of the multispectral camera 10. Specifically, they are images of the transmission wavelength bands of the bandpass filters 123A, 123B, and 123C attached to the windows 122A, 122B, and 122C of the filter unit 120. In this embodiment, the image generation unit 323 generates an image of a first wavelength band λ1 (first image), an image of a second wavelength band λ2 (second image), and an image of a third wavelength band λ3 (third image). The image generation unit 323 performs interference removal processing (interference removal processing) on ​​the image data acquired by the image data acquisition unit 320 for each pixel set to generate images of the wavelength bands λ1, λ2, and λ3. This processing will be outlined below.

[0090] As described above, an image sensor (polarization image sensor) equipped with polarizers in four directions can capture polarized images in four directions in a single shot. These polarized images in four directions contain image components in the wavelength ranges λ1, λ2, and λ3 at a predetermined ratio (crosstalk). The crosstalk ratio is determined by the angles α1, α2, and α3 of the transmission axes of polarizing filters 124A, 124B, and 124C attached to windows 122A, 122B, and 122C of filter unit 120, respectively, and the angles of the transmission axes of the polarizers provided in pixels P1, P2, P3, and P4 of image sensor 210. Specifically, it can be calculated by the square of the cosine (cos) of the angle difference between the angles α1, α2, α3 of the transmission axes of the polarizing filters 124A, 124B, 124C attached to the respective windows 122A, 122B, 122C and the angles β1, β2, β3, β4 of the transmission axes of the polarizers provided in the respective pixels P1, P2, P3, P4. Therefore, for example, the ratio (interference rate) of light that has passed through the first window 122A (light that has passed through the first polarizing filter 124A) received by the first pixel P1 can be calculated by the cos 2 It is calculated as (|α1-β1|).

[0091] In this way, the interference rate is known, and by using this information on the known interference rate, images of each wavelength can be generated. Specifically, images of each wavelength are generated as follows.

[0092] In the image captured by the imaging element 210, the pixel value of the first pixel P1 is x1, the pixel value of the second pixel P2 is x2, the pixel value of the third pixel P3 is x3, and the pixel value of the fourth pixel P4 is x4.

[0093] Furthermore, the pixel value of the corresponding pixel in the generated first image is set to X1, the pixel value of the corresponding pixel in the second image to X2, and the pixel value of the corresponding pixel in the third image to X3.

[0094] If the proportion of light in the first wavelength band λ1 received by the first pixel P1 is b11, the proportion of light in the second wavelength band λ2 received by the first pixel P1 is b12, and the proportion of light in the third wavelength band λ3 received by the first pixel P1 is b13, then the following relationship holds between X1, X2, X3, and x1. b11*X1+b12*X2+b13*X3=x1…(3-1)

[0095] Furthermore, if the proportion of light in the first wavelength band λ1 received by the second pixel P2 is b21, the proportion of light in the second wavelength band λ2 received by the second pixel P2 is b22, and the proportion of light in the third wavelength band λ3 received by the second pixel P2 is b23, then the following relationship holds between X1, X2, X3, and x2. b21*X1+b22*X2+b23*X3=x2…(3-2)

[0096] Furthermore, if the proportion of light in the first wavelength band λ1 received by the third pixel P3 is b31, the proportion of light in the second wavelength band λ2 received by the third pixel P3 is b32, and the proportion of light in the third wavelength band λ3 received by the third pixel P3 is b33, then the following relationship holds between X1, X2, X3, and x3. b31*X1+b32*X2+b33*X3=x3…(3-3)

[0097] Furthermore, if the proportion of light in the first wavelength band λ1 received by the fourth pixel P4 is b41, the proportion of light in the second wavelength band λ2 received by the fourth pixel P4 is b42, and the proportion of light in the third wavelength band λ3 received by the fourth pixel P4 is b43, then the following relationship holds between X1, X2, X3, and x4. b41*X1+b42*X2+b43*X3=x4…(3-4)

[0098] By solving the simultaneous equations (3-1) to (3-4) above for X1, X2, and X3, the pixel values ​​X1, X2, and X3 of the corresponding pixels in the first, second, and third images can be obtained.

[0099] In this way, by using the information on the interference rate, it is possible to generate images for each wavelength from the image captured by the imaging element.

[0100] Here, the above simultaneous equations can be expressed as the following equation (4) using matrix B.

[0101]

number

[0102]

number

[0103] Inverse matrix B of matrix B -1 Let be matrix A.

[0104]

number

[0105] X1, X2, and X3 can be calculated by multiplying both sides of the above equation (4) by matrix A. That is, they can be calculated using the following equation (5). Matrix A is an interference cancellation matrix.

[0106]

number

[0107] The image data processing device 300 holds each element (a11, a12, ...) of the interference cancellation matrix A as a coefficient group. Information about the coefficient group is stored, for example, in the auxiliary storage device 314. The image generation unit 323 acquires the information about the coefficient group from the auxiliary storage device 314, performs interference cancellation processing, and generates an image for each wavelength.

[0108] The output control unit 324 controls the output of the images (first image, second image, and third image) of each wavelength generated by the image generation unit 323. In this embodiment, the output to the display, which is the output device 316, is controlled.

[0109] In response to an instruction from the user, the recording control unit 325 controls the recording of the images of each wavelength generated by the image generation unit 323. The generated images of each wavelength are recorded in the auxiliary storage device 314.

[0110] [Image data processing procedure] FIG. 7 is a flowchart showing the procedure for processing image data.

[0111] First, a process of acquiring image data from the multispectral camera 10 is performed (step S11). Next, a process of detecting and correcting abnormal pixels from the acquired image data is performed (step S12). Next, a process of generating images for each wavelength is performed (step S13).

[0112] FIG. 8 is a flowchart showing the procedure of the process of detecting and correcting an abnormal pixel.

[0113] As described above, the process of detecting abnormal pixels is performed for each pixel set. First, information on pixel values ​​x1 to x4 of each pixel P1 to P4 in the pixel set to be detected is acquired (step S21_1). Next, an intensity value E is calculated from the acquired pixel values ​​x1 to x4 of each pixel P1 to P4. (Step S21_2)That is, the value of E=(x1+x3)-(x2+x4) is calculated. Next, the calculated intensity value E is compared with a threshold value Th1 to determine whether the intensity value E is equal to or greater than the threshold value Th1 (step S21_3).

[0114] Here, when the intensity value E is equal to or greater than the threshold value Th1, the target pixel set includes an abnormal pixel, whereas when the intensity value E is less than the threshold value Th1, the target pixel set does not include an abnormal pixel.

[0115] If the intensity value E is equal to or greater than the threshold value Th1, a process for identifying abnormal pixels is performed (step S21_4). This process is performed by extracting pixels whose pixel values ​​are equal to or less than the threshold value Th2 and pixels whose pixel values ​​are saturated from the target pixel set.

[0116] On the other hand, if the intensity value E is less than the threshold value Th1, the abnormal pixel processing for the pixel set under consideration is terminated. After this, it is determined whether or not the detection of abnormal pixels has been completed for all pixel sets (step S21_6). If the detection of abnormal pixels has not been completed for all pixel sets, information on the pixel values ​​of the next pixel set is acquired (step S21_1), and the detection of abnormal pixels is performed in the same manner. When the detection of abnormal pixels has been completed for all pixel sets, the abnormal pixel detection and correction processing is terminated.

[0117] Once an abnormal pixel is identified in step S21_4, a process of correcting the pixel value of the identified abnormal pixel is performed (step S21_5). That is, a true pixel value is estimated from the pixel values ​​of other pixels in the pixel set, and the pixel value of the abnormal pixel is corrected by the estimated pixel value. The true pixel value is estimated using the relationship x1+x3=x2+x4. After the correction, it is determined whether or not detection of abnormal pixels has been completed for all pixel sets (step S21_6). If detection of abnormal pixels has not been completed for all pixel sets, information on the pixel values ​​of the next pixel set is obtained (step S21_1), and abnormal pixel detection is performed in the same procedure. When detection of abnormal pixels has been completed for all pixel sets, the process of detecting and correcting abnormal pixels is completed.

[0118] When the abnormal pixels have been corrected, in the process of generating images for each wavelength, images for each wavelength are generated based on the corrected image data.

[0119] As described above, according to the image data processing device 300 of this embodiment, when captured image data contains abnormal pixels, the pixel values ​​of the abnormal pixels are corrected, thereby making it possible to generate high-quality images. In other words, if interference removal processing is performed while the image contains abnormal pixels, the generated image may be corrupted, but the image data processing device 300 of this embodiment corrects the pixel values ​​of the abnormal pixels before performing interference removal processing, making it possible to generate high-quality images.

[0120] [Example] The effect of this method compared with conventional interference removal processing is shown below.

[0121] Fig. 9 is a diagram showing an outline of a process for removing interference using a conventional method, and Fig. 10 is a diagram showing an outline of a process for removing interference using the method of the above embodiment.

[0122] The conventional method is a method in which, even if an abnormal pixel is included, interference removal processing is performed directly without any processing.

[0123] 9 and 10 show an example where light of each wavelength is incident on a pixel set at the following intensities: light in the first wavelength band λ1 is 100, light in the second wavelength band λ2 is 100, and light in the third wavelength band λ3 is 220.

[0124] 9 and 10 show an example in which light is incident on each of the pixels P1 to P4 in the pixel set with the following intensities: the first pixel P1 is 180, the second pixel is 158.0385, the third pixel P3 is 240, and the fourth pixel P4 is 261.9615.

[0125] 9 and 10 show an example in which the output of the image sensor is 8 bits. In this case, the signal values ​​(pixel values) of each pixel actually output from the image sensor are as follows: the first pixel P1 is 180, the second pixel is 158.0385, the third pixel P3 is 240, and the fourth pixel P4 is 255. That is, the pixel value of the fourth pixel P4 is output as a saturated value (255).

[0126] As shown in FIG. 9, in the conventional method, even if an abnormal pixel is included, the interference removal process is performed as is, so the calculated intensity of each wavelength will be a value different from the actual intensity.

[0127] On the other hand, as shown in FIG. 10, according to the method of the above embodiment, if an abnormal pixel is included, correction is performed to remove interference, so that the correct value can be calculated for the intensity of each wavelength.

[0128] [Second embodiment] If the captured image data has redundancy, it is possible to generate an image for each wavelength even if abnormal pixels are removed. For example, when generating images for three wavelengths, three pixels are required to generate an image for each wavelength. That is, one pixel set needs to be composed of three pixels (pixels with polarizers in three directions). Therefore, if one pixel set is composed of four pixels (pixels with polarizers in four directions), there is redundancy, and an image for each wavelength can be generated even if one pixel is omitted. That is, interference can be removed. In this case, the interference removal matrix is ​​changed to generate an image for each wavelength. That is, the interference removal process is performed by changing the interference removal matrix.

[0129] Incidentally, a pixel that has become an abnormal pixel due to a fault (a faulty pixel) outputs an abnormal signal value every time. In other words, the same pixel becomes an abnormal pixel every time. If the same pixel becomes an abnormal pixel every time, the position of the abnormal pixel is known, so a method of generating images for each wavelength from image data excluding the abnormal pixel is preferable, as this simplifies the process. In other words, since an interference cancellation matrix can be prepared in advance, the process can be simplified by not having to perform correction processing.

[0130] On the other hand, pixels that have become abnormal due to saturation (saturated pixels) may not become abnormal by changing the scene, settings, etc. Therefore, for saturated pixels, the method of the first embodiment described above, that is, the method of performing correction and normal interference removal processing, is preferable.

[0131] In the image data processing device of this embodiment, when image data obtained by imaging includes abnormal pixels, the interference removal processing method is changed depending on the cause of the abnormality, and images of each wavelength are generated.

[0132] FIG. 11 is a block diagram of functions realized by the image data processing device.

[0133] As shown in the figure, the image data processing device 300 of this embodiment further realizes the function of a processing method determination unit 326 in addition to the functions realized by the image data processing device 300 of the first embodiment.

[0134] The processing method determination unit 326 determines the image processing method to be used by the image generation unit 323 based on the abnormal pixel detection result by the abnormal pixel detection unit 321. That is, the interference removal processing method is determined depending on the cause of the abnormality. Specifically, if the abnormality is due to saturation, interference removal processing is performed using a first method. On the other hand, if the abnormality is due to a malfunction, interference removal processing is performed using a second method. The first method is a method in which the pixel values ​​of the abnormal pixels are corrected to generate images for each wavelength. In this case, normal interference removal processing is performed. That is, interference removal processing is performed using a normal interference removal matrix. The second method is a method in which images for each wavelength are generated excluding the abnormal pixels. In this case, the interference removal matrix is ​​changed to perform interference removal processing.

[0135] The cause of the abnormality is determined based on the pixel value of the abnormal pixel. Specifically, if the pixel value is a saturated value, it is determined to be an abnormality due to saturation. For example, in the case of 8 bits, if the pixel value is 255, it is determined to be an abnormality due to saturation. Also, if the pixel value is equal to or less than threshold value Th2, it is determined to be an abnormality due to a malfunction. The processing method is determined for each pixel set.

[0136] The image generating unit 323 processes the image data in accordance with the processing method determined by the processing method determining unit 326, and generates an image for each wavelength. The processing is performed in units of pixel sets.

[0137] The processing by the first method is the same as the processing in the first embodiment, so here, the processing by the second method will be explained.

[0138] Consider a case where a first pixel P1 in a pixel set is determined to be abnormal due to a failure. In this case, in the second method, images for each wavelength are generated based on the pixel values ​​of the second pixel P2, the third pixel P3, and the fourth pixel P4.

[0139] The pixel value of the second pixel P2 is x2, the pixel value of the third pixel P3 is x3, and the pixel value of the fourth pixel P4 is x4. The pixel values ​​of the corresponding pixels in the generated images of the three wavelengths are X1, X2, and X3, respectively.

[0140] If the proportion of light in the first wavelength band λ1 received by the second pixel P2 is d21, the proportion of light in the second wavelength band λ2 received by the second pixel P2 is d22, and the proportion of light in the third wavelength band λ3 received by the second pixel P2 is d23, then the following relationship holds between X1, X2, X3, and x2. d21*X1+d22*X2+d23*X3=x2…(6-1)

[0141] Furthermore, if the proportion of light in the first wavelength band λ1 received by the third pixel P3 is d31, the proportion of light in the second wavelength band λ2 received by the third pixel P3 is d32, and the proportion of light in the third wavelength band λ3 received by the third pixel P3 is d33, then the following relationship holds between X1, X2, X3, and x3. d31*X1+d32*X2+d33*X3=x3…(6-2)

[0142] Furthermore, if the proportion of light in the first wavelength band λ1 received by the fourth pixel P4 is d41, the proportion of light in the second wavelength band λ2 received by the fourth pixel P4 is d42, and the proportion of light in the third wavelength band λ3 received by the fourth pixel P4 is d43, then the following relationship holds between X1, X2, X3, and x4. d41*X1+d42*X2+d43*X3=x4…(6-3)

[0143] By solving the simultaneous equations (6-1) to (6-3) above for X1, X2, and X3, the pixel values ​​X1, X2, and X3 of the corresponding pixels in the first, second, and third images can be obtained.

[0144] Here, the above simultaneous equations can be expressed as the following equation (7) using matrix D.

[0145]

number

[0146]

number

[0147] Inverse matrix D of matrix D -1 is defined as matrix C. Matrix C is an interference cancellation matrix.

[0148]

number

[0149] X1, X2, and X3 can be calculated by multiplying both sides of the above equation (7) by the interference cancellation matrix C. That is, they can be calculated using the following equation (8).

[0150]

number

[0151] In this way, the pixel values ​​X1, X2, and X3 of the corresponding pixels in the image for each wavelength can be determined from information about the pixel values ​​of the remaining pixels in the pixel set.

[0152] The above is the processing method when the first pixel P1 is determined to be an abnormal pixel. When a pixel other than the first pixel is determined to be an abnormal pixel, the pixel values ​​X1, X2, and X3 of the corresponding pixels are calculated in the same manner. 3 You can ask for it.

[0153] The auxiliary storage device 314 stores information on the interference cancellation matrix when performing interference cancellation processing excluding the first pixel P1, information on the interference cancellation matrix when performing interference cancellation processing excluding the second pixel P2, information on the interference cancellation matrix when performing interference cancellation processing excluding the third pixel P3, and information on the interference cancellation matrix when performing interference cancellation processing excluding the fourth pixel P4. Note that the information on the interference cancellation matrix is ​​information in which each element of the interference cancellation matrix is ​​a coefficient group.

[0154] The image generating unit 323 acquires information on the coefficient group from the auxiliary storage device 314, performs interference removal processing, and generates images of each wavelength.

[0155] FIG. 12 is a diagram showing an outline of the processing when interference removal processing is performed using the second method.

[0156] The figure shows an example in which light of each wavelength is incident on a pixel set with the following intensities: light in the first wavelength band λ1 is 100, light in the second wavelength band λ2 is 100, and light in the third wavelength band λ3 is 220. The figure also shows an example in which light is incident on each of the pixels P1 to P4 in the pixel set with the following intensities: first pixel P1 is 180, second pixel P2 is 180, and so on. P2 The value of the third pixel P3 is 158.0385, the value of the fourth pixel P4 is 240, and the value of the fourth pixel P4 is 0.1. In other words, this shows an example in which the fourth pixel P4 is a failed pixel.

[0157] As described above, in the second method, when an abnormal pixel is included, the interference removal process is performed on three pixels excluding the abnormal pixel. That is, the interference removal process is performed on three pixels, the first pixel P1, the second pixel P2, and the third pixel P3. This eliminates the influence of the abnormal pixel and reduces the interference of each wavelength. Strength The correct value can be calculated.

[0158] [Image data processing procedure] FIG. 13 is a flowchart showing the procedure for processing image data by the image data processing device.

[0159] First, image data is acquired from the multispectral camera 10 (step S31). Next, abnormal pixels are detected from the acquired image data (step S32). Next, the presence or absence of abnormal pixels is determined based on the abnormal pixel detection result (step S33).

[0160] If there are no abnormal pixels, normal interference removal processing is performed and images of each wavelength are generated (step S37).

[0161] On the other hand, if there are abnormal pixels, the processing method for image generation is determined based on the cause of the abnormality (step S34). If the cause of the abnormality is saturation, the first method is selected as the processing method for image generation. As described above, the first method is a method of correcting the pixel values ​​of the abnormal pixels to generate images for each wavelength. If the cause of the abnormality is a malfunction, the second method is selected as the processing method for image generation. As described above, the second method is a method of generating images for each wavelength excluding the abnormal pixels.

[0162] After the processing method for image generation is determined, it is determined whether the determined processing method is the first method or not (step S35).

[0163] If the image generation processing method is the first method, that is, if the abnormality is due to saturation, the pixel values ​​of the abnormal pixels are corrected (step S36). Thereafter, the normal interference removal process is performed as in the case where there are no abnormal pixels, and images of each wavelength are generated (step S37).

[0164] If the image generation processing method is the second method (if the determination in step S35 is "N"), interference removal processing is performed excluding abnormal pixels, and images of each wavelength are generated (step S 38 ) More specifically, for pixel sets that include abnormal pixels, interference removal processing is performed excluding the abnormal pixels. For pixel sets that do not include abnormal pixels, normal interference removal processing is performed. For pixel sets that include abnormal pixels, interference removal processing is performed by switching the interference removal matrix depending on the position of the abnormal pixel.

[0165] The above processing may be performed in units of pixel sets or in units of image data.

[0166] As described above, according to the image data processing device of this embodiment, the processing method for eliminating interference can be changed depending on whether or not an abnormality exists and its cause, so that high-quality images can be generated efficiently.

[0167] [Variations] [How to identify abnormal pixels] Here, another example of a method for identifying an abnormal pixel in a pixel set when the pixel set including the abnormal pixel is detected will be described.

[0168] (1) Identifying abnormal pixels using a set of surrounding pixels Here, a method for identifying an abnormal pixel by using a surrounding pixel set when a pixel set including the abnormal pixel is detected will be described.

[0169] FIG. 14 is a conceptual diagram of a method for identifying an abnormal pixel using a set of surrounding pixels.

[0170] The figure shows a schematic diagram of a pixel arrangement. Each hatched square represents a pixel. The numbers in the squares are used to distinguish between the pixels. The hatching in the squares indicates the transmission axis orientation of each pixel (the angle of the transmission axis of the polarizer). For example, the transmission axis orientation of pixel P11 is 0°, the transmission axis orientation of pixel P12 is 45°, the transmission axis orientation of pixel P22 is 90°, and the transmission axis orientation of pixel P21 is 135°.

[0171] In the figure, a pixel set consisting of pixels P33, P34, P44, and P43 (a pixel set surrounded by a frame indicated by a thick line) is assumed to be a pixel set to be detected. Also, pixel P33 is assumed to be an abnormal pixel.

[0172] When a pixel set including an abnormal pixel is detected, the intensity value E is first calculated for the eight surrounding pixel sets.

[0173] FIG. 15 is a diagram showing eight surrounding pixel sets. As shown in the figure, the eight surrounding pixel sets are: (a) pixel set SP1 consisting of pixel P22, pixel P23, pixel P33, and pixel P32; (b) pixel set SP2 consisting of pixel P23, pixel P24, pixel P34, and pixel P33; (c) pixel set SP3 consisting of pixel P24, pixel P25, pixel P35, and pixel P34; (d) pixel set SP4 consisting of pixel P34, pixel P35, pixel P45, and pixel P44; (e) pixel set SP5 consisting of pixel P44, pixel P45, pixel P55, and pixel P54; (f) pixel set SP6 consisting of pixel P43, pixel P44, pixel P54, and pixel P53; (g) pixel set SP7 consisting of pixel P42, pixel P43, pixel P53, and pixel P52; and (h) pixel set SP8 consisting of pixel P32, pixel P33, pixel P43, and pixel P44. For each set of pixels, an intensity value E is calculated.

[0174] Next, pixel sets whose intensity value E is equal to or greater than the threshold value Th1 are extracted from the eight surrounding pixel sets. As described above, in pixel sets that include abnormal pixels, the intensity value E is equal to or greater than the threshold value Th1. Therefore, by calculating the intensity value E, it is possible to detect pixel sets that include abnormal pixels from the eight surrounding pixel sets. In this example, the intensity value E is equal to or greater than the threshold value Th1 in the pixel set that includes pixel P33. Specifically, the intensity value E is equal to or greater than the threshold value Th1 in pixel sets SP1, SP2, and SP8.

[0175] Next, overlapping pixels are extracted from the extracted pixel sets. That is, overlapping pixels are extracted between pixel sets that contain abnormal pixels. In this example, overlapping pixels are extracted from pixel sets SP1, SP2, and SP8. The only overlapping pixel among pixel sets SP1, SP2, and SP8 is pixel P33. The extracted pixel (pixel P33) is identified as an abnormal pixel.

[0176] The cause of the abnormality is identified. TaThe determination is made based on the pixel value of the abnormal pixel. For example, if the pixel value of the identified abnormal pixel is close to 0, it is determined to be an abnormality due to a malfunction. Also, if the pixel value of the identified abnormal pixel is a saturated value, it is determined to be an abnormality due to saturation.

[0177] In this way, by calculating the intensity value E of a set of surrounding pixels, it is possible to identify an abnormal pixel from the information of the intensity value E.

[0178] The range of the pixel set to be referenced can be further expanded. For example, 16 surrounding pixel sets can be added to the range of the pixel set to be referenced. By expanding the range of the pixel set to be referenced, abnormal pixels can be identified with higher accuracy.

[0179] FIG. 16 is a conceptual diagram showing a case where abnormal pixels are identified by expanding the range of the pixel set to be referenced.

[0180] This figure shows an example of a case where an abnormal pixel is identified by further referring to a set of 16 surrounding pixels.

[0181] In the figure, a pixel set consisting of pixels P33, P34, P44, and P43 (a pixel set surrounded by a frame indicated by a thick line) is assumed to be a pixel set to be detected. Also, pixel P33 is assumed to be an abnormal pixel.

[0182] First, the intensity value E is calculated for eight pixel sets surrounding the pixel set to be detected.

[0183] FIG. 17 is a diagram showing eight peripheral pixel sets. As shown in the figure, the eight surrounding pixel sets are: (a) pixel set SP1 consisting of pixel P22, pixel P23, pixel P33, and pixel P32; (b) pixel set SP2 consisting of pixel P23, pixel P24, pixel P34, and pixel P33; (c) pixel set SP3 consisting of pixel P24, pixel P25, pixel P35, and pixel P34; (d) pixel set SP4 consisting of pixel P34, pixel P35, pixel P45, and pixel P44; (e) pixel set SP5 consisting of pixel P44, pixel P45, pixel P55, and pixel P54; (f) pixel set SP6 consisting of pixel P43, pixel P44, pixel P54, and pixel P53; (g) pixel set SP7 consisting of pixel P42, pixel P43, pixel P53, and pixel P52; and (h) pixel set SP8 consisting of pixel P32, pixel P33, pixel P43, and pixel P44. For each set of pixels, an intensity value E is calculated.

[0184] Next, abnormal pixels are identified based on the calculated intensity value E of the set of eight surrounding pixels. As described above, in this example, pixels P33 and P35 are abnormal pixels. In this case, pixel P34, which should be a normal pixel, is also determined to be an abnormal pixel. Therefore, when abnormal pixels are identified in two locations, a process is performed to identify truly abnormal pixels by further referencing the set of 16 surrounding pixels.

[0185] FIG. 18 is a diagram showing a set of 16 further surrounding pixels.

[0186] As shown in the figure, the 16 surrounding pixel sets are: (i) pixel set SP9 consisting of pixel P11, pixel P12, pixel P22, and pixel P21; (j) pixel set SP10 consisting of pixel P12, pixel P13, pixel P23, and pixel P22; (k) pixel set SP11 consisting of pixel P13, pixel P14, pixel P24, and pixel P23; (l) pixel set SP12 consisting of pixel P14, pixel P15, pixel P25, and pixel P24; (m) pixel set SP13 consisting of pixel P15, pixel P16, pixel P26, and pixel P25; (n) pixel set SP14 consisting of pixel P25, pixel P26, pixel P36, and pixel P35; (o) pixel set SP15 consisting of pixel P35, pixel P36, pixel P46, and pixel P45; (p) pixel set SP16 consisting of pixel P45, pixel P46, pixel P56, and pixel P55. (q) pixel set SP17 consisting of pixel P55, pixel P56, pixel P66, and pixel P65; (r) pixel set SP18 consisting of pixel P54, pixel P55, pixel P65, and pixel P64; (s) pixel set SP19 consisting of pixel P53, pixel P54, pixel P64, and pixel P63; (t) pixel set SP20 consisting of pixel P52, pixel P53, pixel P63, and pixel P62; (u) pixel set SP21 consisting of pixel P51, pixel P52, pixel P62, and pixel P61; (v) pixel set SP22 consisting of pixel P41, pixel P42, pixel P52, and pixel P51; (w) pixel set SP23 consisting of pixel P31, pixel P32, pixel P42, and pixel P41; and (x) pixel set SP24 consisting of pixel P21, pixel P22, pixel P32, and pixel P31.

[0187] First, the intensity value E of the set of 16 surrounding pixels is calculated. In this example, the intensity value E is calculated for pixel sets SP9 to SP24. This makes it possible to detect pixel sets containing abnormal pixels in the set of 16 surrounding pixels. In this example, no pixel set containing abnormal pixels is detected in the pixel set surrounding pixel P33. On the other hand, a pixel set containing abnormal pixels is detected in the pixel set surrounding pixel P35. This makes it possible to determine that pixel P33 is truly an abnormal pixel. On the other hand, it is still not possible to determine whether pixel P34 is truly an abnormal pixel.

[0188] Therefore, next, a process is performed to correct the value of pixel P33, which was determined to be a truly abnormal pixel. After correction, the intensity value E is calculated again for the pixel set to be detected. If the calculated intensity value E is less than threshold value Th1, the pixel set to be detected does not include an abnormal pixel, and therefore pixel P34 can be determined not to be an abnormal pixel.

[0189] In this way, by widening the range of the pixel set to be referred to, it is possible to detect abnormal pixels with high accuracy even when abnormal pixels exist in the periphery.

[0190] In addition to faulty and saturated pixels, it is also possible to detect pixels with abnormal output (pixels that output values ​​that differ from the intended output).

[0191] The range of the pixel set to be referred to is preferably set according to, for example, the resolution of the lens device, the required resolution, and the like.

[0192] Here, the resolution of a lens device refers to the resolution of the lens device in use under the imaging conditions. Since it is the resolution under the imaging conditions, for example, if the aperture value (F-number) changes, the resolution of the lens device also changes. The resolution of a lens device also changes depending on the object distance.

[0193] The required resolution refers to the resolution of the image sensor that can satisfactorily depict the size of the region of interest of a certain object captured on the image sensor when the object is imaged. The region of interest, for example, is the defect in an image captured for defect detection purposes, or the object itself in an image captured for object identification purposes. As an example, consider the case of detecting a 100 mm defect. If a multispectral camera is set at an imaging magnification of 0.01, the defect will appear 1 mm in size on the image sensor. The required resolution is the number of pixels (pixel size) that can satisfactorily depict this 1 mm size.

[0194] When the resolution of the lens device is low, it is preferable to widen the range of the pixel set to be referenced, and when the resolution is high, it is preferable to narrow the range of the pixel set to be referenced. Similarly, when the required resolution is low, it is preferable to widen the range of the pixel set to be referenced, and when the required resolution is high, it is preferable to narrow the range of the pixel set to be referenced.

[0195] For example, when the resolution of the lens device is low (when the resolution of the lens device is equal to or less than a threshold), the process of identifying abnormal pixels is performed by referring to not only the peripheral eight pixel sets but also the peripheral sixteen pixel sets.On the other hand, when the resolution of the lens device is high (when the resolution of the lens device exceeds a threshold), the process of identifying abnormal pixels is performed by only the peripheral eight pixel sets.

[0196] Furthermore, for example, when the required resolution is low (when the required resolution is equal to or less than the threshold), the process of identifying abnormal pixels is performed by referring to not only the surrounding eight pixel sets but also the surrounding sixteen pixel sets.On the other hand, when the required resolution is high (when the required resolution exceeds the threshold), the process of identifying abnormal pixels is performed by only the surrounding eight pixel sets.

[0197] As an example, when capturing an image of a subject, if the size of the region of interest of the subject captured on the image sensor is a, the range of the referenced pixel set can be expanded to the extent that the size of the entire referenced pixel set is less than or equal to a.

[0198] The range of pixel sets to be referenced can be set based on the number of abnormal pixels, as well as the resolution of the lens device and the required resolution. For example, only when the number of abnormal pixels is large (when the number of abnormal pixels is equal to or greater than a threshold), the range of pixel sets to be referenced is expanded and abnormal pixels are identified. As an example, when the number of abnormal pixels is large (when the number of abnormal pixels is equal to or greater than a threshold), abnormal pixels are identified by referring to 16 pixel sets in addition to the surrounding 8 pixel sets. On the other hand, when the number of abnormal pixels is small (when the number of abnormal pixels is less than the threshold), abnormal pixels are identified using only the surrounding 8 pixel sets.

[0199] (2) A method of identifying abnormal pixels by referencing the pixel values ​​of similar pixels in the surrounding area Here, a method for identifying an abnormal pixel by referring to the pixel values ​​of surrounding pixels of the same type, where the pixels of the same type are pixels equipped with the same polarizer, will be described.

[0200] In this method, a pixel whose pixel value deviates from a value (estimated value) estimated from the pixel values ​​of surrounding pixels of the same type is estimated to be an abnormal pixel.

[0201] The estimated value can be obtained using known techniques such as the bilinear method or the bicubic method. For example, in the example shown in FIG. 14, the estimated value for pixel P33 is calculated based on the pixel values ​​of pixels P31 and P35. Specifically, the average value of the pixel values ​​of pixels P31 and P35 is calculated. Similarly, the estimated value for pixel P34 is calculated based on the pixel values ​​of pixels P32 and P36. The estimated value for pixel P44 is calculated based on the pixel values ​​of pixels P42 and P46. The estimated value for pixel P43 is calculated based on the pixel values ​​of pixels P41 and P45.

[0202] That is, the estimated value is calculated based on the pixel values ​​of two pixels of the same type that are located on either side of the target pixel. Therefore, for example, for pixel P33, the estimated value can be calculated based on the pixel values ​​of pixels P13 and P53, or the pixel values ​​of pixels P11 and P55, or the pixel values ​​of pixels P15 and P51. The same applies to other pixels.

[0203] Pixel to be detected set An estimated value is calculated for each pixel in the image, and pixels whose pixel values ​​deviate from the estimated value are estimated to be abnormal pixels. Specifically, for each pixel, the difference between the pixel value and the estimated value is calculated, and the calculated difference is compared with a threshold value Th3. The difference is calculated as the absolute value of the difference between the pixel value and the estimated value. Pixels whose calculated difference is equal to or greater than the threshold value Th3 are estimated to be abnormal pixels. The threshold value Th3 is an example of a third threshold value.

[0204] In this way, by referring to the pixel values ​​of surrounding pixels of the same type, an abnormal pixel can be identified.

[0205] (3) Other methods Identifying abnormal pixels can be performed by combining multiple methods, including the method of the above embodiment (the method of identifying from pixel values). For example, a method of identifying abnormal pixels can be adopted that combines the method of identifying from pixel values ​​with the method of using a set of surrounding pixels. Alternatively, a method of identifying abnormal pixels can be adopted that combines the method of identifying from pixel values ​​with the method of referring to the pixel values ​​of surrounding pixels of the same type. In this way, by combining multiple methods to identify abnormal pixels, it becomes possible to detect abnormal pixels with high accuracy.

[0206] [Method for correcting pixel values ​​of abnormal pixels] In the above embodiment, when an abnormal pixel is detected, the pixel value of the abnormal pixel is estimated and corrected using the relationship in equation (1). However, the method for correcting the pixel value of the abnormal pixel is not limited to this. As in the case of detecting an abnormality, the pixel value of the abnormal pixel may be estimated and corrected using the pixel values ​​of surrounding pixels of the same type. For example, in the example shown in FIG. 14, if pixel P33 is an abnormal pixel, the pixel value of pixel P33 can be estimated and corrected based on the pixel values ​​of pixels P31 and P35. In this case, specifically, the average value of pixel values ​​of pixels P31 and P35 is calculated, and the pixel value is estimated and corrected.

[0207] [Other embodiments and modifications] [Pixels containing abnormal pixels set Modified example of the detection method As described above, when one pixel set SP is composed of four pixels P1 to P4, each with a polarizer angle of 0°, 45°, 90°, and 135°, the relationship in equation (1) above holds. That is, the relationship x1 + x3 = x2 + x4 holds. In the above embodiment, this relationship is used to calculate the intensity value E = (x1 + x3) - (x2 + x4), and if the intensity value E is equal to or greater than the threshold value Th1, it is determined that the target pixel set SP includes an abnormal pixel.

[0208] Even if the combination of four pixels that make up one pixel set SP is different from the combination in the above embodiment, it is possible to calculate the intensity value E using a similar method and determine whether or not an abnormal pixel is included. Below, we will explain examples where one pixel set is made up of four pixels and the transmission axis orientations of each pixel (the angles of the transmission axes of the installed polarizers) are 0°, 60°, 90°, and 120°, and where they are 0°, 45°, 60°, and 120°.

[0209] (1) When the transmission axis direction of each pixel is 0°, 60°, 90°, or 120° When one pixel set is made up of four pixels and the transmission axis directions of the pixels are 0°, 60°, 90°, and 120°, respectively, the following relationship holds.

[0210] I0-2×I 60 +3×I 90 -2×I 120 =0 Here, I0 is the pixel value of the pixel whose transmission axis direction is 0°. 60 is the pixel value of the pixel whose transmission axis direction is 60°. 90 is the pixel value of the pixel whose transmission axis direction is 90°. 120 is the transmission axis direction 120 is the pixel value of the pixel at .degree.

[0211] Therefore, when the transmission axis orientations of the four pixels are 0°, 60°, 90°, and 120°, the intensity value E is calculated using the following formula, and when the intensity value E is equal to or greater than the threshold value Th1, it is determined that the target pixel set includes an abnormal pixel. E=I0-2×I 60 +3×I 90 -2×I 120

[0212] (2) When the transmission axis direction of each pixel is 0°, 45°, 60°, or 120°, one pixel set is composed of four pixels, and the transmission axis direction of each pixel is 0°, 45 °, 60 When the polarizers are oriented at 120° and 150°, the following relationship holds:

[0213] I0-3×I 45 +(√3+1)×I 60 -(√3-1)×I 120 =0 Here, I0 is the pixel value of the pixel whose transmission axis direction is 0°. 45 is the pixel value of the pixel whose transmission axis direction is 45°. 60 is the pixel value of the pixel whose transmission axis direction is 60°. 120 is the pixel value of a pixel whose transmission axis direction is 120°.

[0214] Therefore, when the transmission axis orientation of each pixel is 0°, 45°, 60°, or 120°, the intensity value E is calculated using the following formula, and when the intensity value E is equal to or greater than the threshold value Th1, it is determined that the target pixel set includes an abnormal pixel. E=I0-3×I 45 +(√3+1)×I 60 -(√3-1)×I 120

[0215] In this way, the relational equation between pixel values ​​between each pixel is found based on the transmission axis orientation of each pixel, and a formula for calculating the intensity value E is set. The found relational equation between pixel values ​​between each pixel can also be used to correct the pixel values ​​of abnormal pixels.

[0216] [Modification of the imaging element] Although the above embodiment shows an example in which the imaging element is a so-called monochrome imaging element, the present invention can also be applied to color imaging elements. In a color polarization imaging element, color filters are arranged in units of pixel sets. That is, each pixel in the same pixel set is equipped with a color filter of the same color. The color filter arrangement may be a known arrangement such as a Bayer arrangement.

[0217] If the image sensor is color, the process of identifying abnormal pixels is performed on a color-by-color basis. For example, in a method of identifying abnormal pixels by referring to pixel values ​​of surrounding pixels of the same type, the pixel values ​​of pixels equipped with the same polarizer and the same color filter are referred to identify the abnormal pixels.

[0218] [Multispectral camera variation] A multispectral camera is configured with a lens device and a camera body according to the number of wavelengths to be simultaneously captured. For example, when capturing a two-wavelength multispectral image, the lens device is configured to split incident light into two wavelengths and polarize each split wavelength of light in a specific direction before emitting it. The imaging element of the camera body is a polarization imaging element equipped with polarizers in at least two directions.

[0219] It is preferable that the lens device has a filter unit that is detachable from the lens barrel and can be freely replaced, so that images of various wavelengths can be captured simply by replacing the filter unit.

[0220] Furthermore, it is preferable that the filter unit be configured so that the filters (bandpass filters and polarizing filters) attached to each window can be detached or replaced. This allows the number and combination of wavelengths to be separated to be freely changed. In this case, it is not necessary to use all of the windows. For example, if the filter frame has four windows, and an image of three wavelengths is to be captured, one window can be used with light shielding.

[0221] Furthermore, the bandpass filters and polarizing filters to be attached to each window may be configured to be attached individually to the window, or may be configured to be attached as an integrated (bonded) unit. When integrated, it is possible to have a configuration in which there is no air gap between the filters. When integrated, the filters can be bonded and integrated, for example, by optical contact.

[0222] The shape of each window provided in the filter unit is not particularly limited, and various shapes can be adopted, for example, a sector shape divided equally in the circumferential direction.

[0223] [Modification of image data processing device] In the multispectral camera system of the above embodiment, the multispectral camera and the image data processing device are configured as separate entities, but the camera body of the multispectral camera may also have the function of the image data processing device.

[0224] The functions of the image data processing device are realized by various processors. The various processors include CPUs and / or GPUs (Graphic Processing Units), which are general-purpose processors that execute programs and function as various processing units, programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations designed specifically to execute specific processes. The term "program" is synonymous with "software."

[0225] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types. For example, a single processing unit may be configured with multiple FPGAs, or a combination of a CPU and an FPGA. Alternatively, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by computers used as clients or servers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system on chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure. [Explanation of symbols]

[0226] 1. Multispectral camera system 10 Multispectral Camera 100 Lens device 110A lens group 110B lens group 120 Filter Unit 122 Filter Frame 122A Window (first window) 122B Window section (second window section) 122C Window section (third window section) 123A Bandpass Filter (1st Bandpass Filter) 123B Bandpass Filter (Second Bandpass Filter) 123C Bandpass Filter (3rd Bandpass Filter) 124A Polarizing Filter (1st Polarizing Filter) 124B Polarizing Filter (Second Polarizing Filter) 124C Polarizing Filter (Third Polarizing Filter) 200 camera body 210 Image sensor 300 Image data processing device 311 CPU 312 ROM 313 RAM 314 Auxiliary storage 315 Input Device 316 Output Device 317 Input / Output Interface 320 Image data acquisition unit 321 Abnormal pixel detection unit 322 Pixel value correction unit 323 Image Generation Unit 324 Output control section 325 Recording control section 326 Treatment Method Decision Department P1 pixel (first pixel) P2 pixel (second pixel) P3 pixel (third pixel) P4 pixel (4th pixel) P11~P16 pixels P21~P26 pixels P31~P36 pixels P41~P46 pixels P51~P56 pixels P61~P66 pixels SP pixel set SP1~SP24 pixel set Z optical axis S1~S3: Processing procedure for detecting abnormal pixels S11~S13 Image data processing procedure S21_1 to S21_6: Procedures for detecting and correcting abnormal pixels S31~S37 Image data processing procedure

Claims

1. An image data processing device that processes image data captured by an imaging device that includes an optical system that splits incident light into a plurality of wavelengths and polarizes the split light of the wavelengths in specific directions and outputs the polarized light, and an imaging element that includes a plurality of sets of pixels each having a different type of polarizer, a processor; The processor: A process of acquiring the image data; a process of detecting pixels whose pixel values ​​are outside a predetermined range from the acquired image data as abnormal pixels; a process of correcting the pixel value of the abnormal pixel based on pixel values ​​of surrounding pixels having a polarizer of a different type from that of the polarizer of the abnormal pixel when the abnormal pixel is detected; a process of generating an image of the wavelengths dispersed from the image data after correcting the pixel values ​​of the abnormal pixels when the abnormal pixels are detected; To do Image data processing device.

2. the process of correcting the pixel value of the abnormal pixel includes correcting the pixel value of the abnormal pixel based on pixel values ​​of other pixels in the set of pixels in which the abnormal pixel has been detected; The image data processing device according to claim 1 .

3. the process of correcting the pixel values ​​of the abnormal pixels includes correcting the pixel values ​​of the abnormal pixels based on a relational expression of pixel values ​​that holds between the pixels constituting the pixel set; The image data processing device according to claim 1 .

4. The relationship is determined based on the transmission axis orientations of the pixels constituting the pixel set. The image data processing device according to claim 3 .

5. the processor performs a process of correcting the pixel value of the abnormal pixel based on pixel values ​​of surrounding pixels when the pixel value falls outside the predetermined range due to saturation. The image data processing device according to any one of claims 1 to 4.

6. When a pixel value is outside the predetermined range due to a malfunction, the processor performs a process of generating an image of the wavelengths dispersed from the image data excluding the abnormal pixel without correcting the pixel value of the abnormal pixel. The image data processing device according to any one of claims 1 to 4.

7. The process of detecting the abnormal pixel includes: detecting a set of pixels including the abnormal pixel; identifying the anomalous pixel from the set of detected pixels; Including, The image data processing device according to any one of claims 1 to 6.

8. An image data processing device that processes image data captured by an imaging device that includes an optical system that splits incident light into a plurality of wavelengths and polarizes the split light of the wavelengths in specific directions and outputs the polarized light, and an imaging element that includes a plurality of sets of pixels each having a different type of polarizer, a processor; The processor: A process of acquiring the image data; a process of detecting pixels whose pixel values ​​are outside a predetermined range from the acquired image data as abnormal pixels; a process of correcting the pixel value of the abnormal pixel based on pixel values ​​of surrounding pixels when the abnormal pixel is detected; a process of generating an image of the wavelengths dispersed from the image data after correcting the pixel values ​​of the abnormal pixels when the abnormal pixels are detected; and the process of detecting the abnormal pixel includes a process of detecting a set of pixels including the abnormal pixel, and a process of identifying the abnormal pixel from the detected set of pixels. Image data processing device.

9. The process of detecting a set of pixels including an abnormal pixel includes: detecting a set of pixels including the abnormal pixel based on pixel values ​​of pixels constituting the set of pixels; 9. The image data processing device according to claim 7 or 8.

10. The process of detecting a set of pixels including an abnormal pixel includes: the sum of the pixel values ​​of the pixels that make up the set of pixels, or calculating a sum of pixel values ​​of the pixels constituting the set of pixels multiplied by a particular coefficient; A set of pixels whose calculated sum is equal to or greater than a first threshold is detected as a set of pixels including the abnormal pixel. The image data processing device according to claim 9 .

11. The process of identifying the abnormal pixel includes: extracting pixels whose pixel values ​​are equal to or less than a second threshold value and / or pixels whose pixel values ​​are saturated from the set of pixels including the abnormal pixels to identify the abnormal pixels; 11. The image data processing device according to claim 9 or 10.

12. The process of identifying the abnormal pixel includes: identifying the abnormal pixel based on pixel values ​​of pixels surrounding the set of pixels including the abnormal pixel; 11. The image data processing device according to claim 9 or 10.

13. The process of identifying the abnormal pixel includes: A process of detecting a set of pixels including the abnormal pixel from a set of pixels surrounding the set of pixels including the abnormal pixel; a process of identifying the abnormal pixel from the set of pixels including the abnormal pixel based on a result of detecting the set of pixels including the abnormal pixel; Including, The image data processing device according to claim 12.

14. a range for detecting the set of pixels including the abnormal pixel is switched according to a resolution of the optical system; The image data processing device according to claim 13 .

15. The process of identifying the abnormal pixel includes: A process of estimating a pixel value of a pixel from pixel values ​​of surrounding pixels; a process of identifying pixels whose difference from the estimated pixel value is equal to or greater than a third threshold as abnormal pixels; Including, The image data processing device according to claim 12.

16. The process of estimating the pixel value of a pixel from the pixel values ​​of surrounding pixels includes estimating the pixel value from the pixel values ​​of surrounding pixels having the same type of polarizer. The image data processing device according to claim 15.

17. An image data processing method for processing image data captured by an imaging device including an optical system that separates incident light into a plurality of wavelengths and polarizes the separated light of the wavelengths in specific directions and outputs the polarized light, and an imaging element that includes a plurality of sets of pixels each having a different type of polarizer, the method comprising: A process of acquiring the image data; a process of detecting pixels whose pixel values ​​are outside a predetermined range from the acquired image data as abnormal pixels; a process of correcting the pixel value of the abnormal pixel based on pixel values ​​of surrounding pixels having a polarizer of a different type from that of the polarizer of the abnormal pixel when the abnormal pixel is detected; a process of generating an image of the wavelengths dispersed from the image data after correcting the pixel values ​​of the abnormal pixels when the abnormal pixels are detected; Including, Image data processing method.

18. An image data processing method for processing image data captured by an imaging device including an optical system that separates incident light into a plurality of wavelengths and polarizes the separated light of the wavelengths in specific directions and outputs the polarized light, and an imaging element that includes a plurality of sets of pixels each having a different type of polarizer, the method comprising: A process of acquiring the image data; a process of detecting pixels whose pixel values ​​are outside a predetermined range from the acquired image data as abnormal pixels; a process of correcting the pixel value of the abnormal pixel based on pixel values ​​of surrounding pixels when the abnormal pixel is detected; a process of generating an image of the wavelengths dispersed from the image data after correcting the pixel values ​​of the abnormal pixels when the abnormal pixels are detected; the process of detecting the abnormal pixel includes a process of detecting a set of pixels including the abnormal pixel, and a process of identifying the abnormal pixel from the detected set of pixels. Image data processing method.

19. An image data processing program for processing image data captured by an imaging device including an optical system that separates incident light into a plurality of wavelengths and polarizes the separated light of the wavelengths in specific directions and outputs the polarized light, and an imaging element that includes a plurality of sets of pixels each having a different type of polarizer, a function of acquiring the image data; a function of detecting, as an abnormal pixel, a pixel whose pixel value is outside a predetermined range from the acquired image data; a function of correcting the pixel value of the abnormal pixel when the abnormal pixel is detected based on pixel values ​​of surrounding pixels that have a polarizer of a different type from that of the polarizer of the abnormal pixel; a function of generating an image of the wavelengths dispersed from the image data after correcting the pixel values ​​of the abnormal pixels when the abnormal pixels are detected; To realize this on a computer, Image data processing program.

20. An image data processing program for processing image data captured by an imaging device including an optical system that separates incident light into a plurality of wavelengths and polarizes the separated light of the wavelengths in specific directions and outputs the polarized light, and an imaging element that includes a plurality of sets of pixels each having a different type of polarizer, a function of acquiring the image data; a function of detecting, as an abnormal pixel, a pixel whose pixel value is outside a predetermined range from the acquired image data; a function of correcting the pixel value of the abnormal pixel based on pixel values ​​of surrounding pixels when the abnormal pixel is detected; a function of generating an image of the wavelengths dispersed from the image data after correcting the pixel values ​​of the abnormal pixels when the abnormal pixels are detected; The method is implemented by a computer, and the function of detecting the abnormal pixel includes a function of detecting a set of pixels including the abnormal pixel, and a function of identifying the abnormal pixel from the detected set of pixels. Image data processing program.

21. A non-transitory computer-readable recording medium on which the program according to claim 19 or 20 is recorded.

22. an imaging device including: an optical system that separates incident light into a plurality of wavelengths and outputs the separated light of the wavelengths polarized in specific directions; and an imaging element that includes a plurality of sets of pixels each having a different type of polarizer; an image data processing device according to any one of claims 1 to 16, which processes image data captured by the imaging device; An imaging system comprising:

Images