Image processing device, image processing method, and program

Abstract

This image processing device is to be used for an image outputted from an imaging device comprising an optical system. The optical system has a plurality of filters provided around an optical axis. This image processing device comprises a processor. The processor performs, on the image, a process for correcting image deviation due to positional deviation of an optical image caused by light diffraction by means of the plurality of filters.

Description

Image processing device, image processing method, and program

[0001] The technology of the present disclosure relates to an image processing device, an image processing method, and a program.

[0002] International Publication No. WO 2021 / 085368 discloses an imaging device including an imaging optical system, an imaging element, and a signal processing unit. The imaging optical system has a first pupil region that passes light of a first wavelength band and a second pupil region that passes light of a second wavelength band different from the first wavelength band. The imaging optical system reduces axial chromatic aberration of the imaging optical system due to the difference between the first and second wavelength bands based on a relationship between aberrations other than the axial chromatic aberration of the imaging optical system and the positions of the first and second pupil regions of the imaging optical system. The imaging element includes a first pixel that receives light passing through the first pupil region of the imaging optical system and a second pixel that receives light passing through the second pupil region. The signal processing unit processes a signal output from the imaging element and generates a first image in the first wavelength band and a second image in the second wavelength band based on the output signal of the first pixel and the output signal of the second pixel, respectively.

[0003] Japanese Patent Application Laid-Open Publication No. 2022-063720 discloses an image correction device including a band image acquisition means, a high-resolution image acquisition means, a position difference acquisition means, a correction band image creation means, and a correction band image output means. The band image acquisition means acquires multiple band images obtained by imaging an object. The high-resolution image acquisition means acquires high-resolution images obtained by imaging the object, the band images having a higher resolution than the band images. The position difference acquisition means designates one of the multiple band images as a reference band image and at least one of the remaining band images as a target band image, and acquires a position difference between the target band image and the reference band image. The correction band image creation means designates a pixel of the target band image as a target pixel, and for each target pixel, determines pixel values ​​of individual partial regions obtained by dividing the imaging region of the target pixel into multiple regions based on the pixel value of the target pixel and the pixel value relationship between multiple pixels of the high-resolution image corresponding to the target pixel, and creates a correction band image that retains pixel values ​​of light related to the target band image at pixel positions of the reference band image based on the pixel values ​​and position differences for each determined partial region. The corrected band image output means outputs the corrected band image.

[0004] International Publication No. 2021 / 153473 discloses a lens device including an imaging optical system and an optical element. The imaging optical system includes a lens that forms an optical image of a subject. The optical element includes a frame having a plurality of aperture regions, a plurality of optical filters arranged in at least one of the aperture regions, the plurality of optical filters including two or more optical filters that transmit light of at least a portion of different wavelength bands, and a plurality of polarizing filters arranged in at least one of the aperture regions, the plurality of polarizing filters having different polarization directions. The amount of light emitted from the imaging optical system is variable for each of the aperture regions.

[0005] Japanese Patent Laid-Open Publication No. 2014-045275 discloses an image processing device including an image acquisition unit, a phase-contrast image generation unit, and a resolution-enhancing processing unit. The image acquisition unit acquires captured images obtained by capturing a first subject image and a second subject image having parallax with respect to the first subject image. The phase-contrast image generation unit generates a first image corresponding to the first subject image and a second image corresponding to the second subject image based on the captured images. The resolution-enhancing processing unit performs resolution-enhancing processing on the captured images based on the first and second images.

[0006] As an example, one embodiment of the technology disclosed herein provides an image processing device, an image processing method, and a program that can obtain a multispectral image of higher image quality than when a multispectral image is generated based on a spectral image in which image misalignment occurs.

[0007] A first aspect of the technology of the present disclosure is an image processing device that is applied to an image output from an imaging device that has an optical system, the optical system having a plurality of filters arranged around an optical axis, and the image processing device having a processor that performs processing on the image to correct image shift due to positional shift of an optical image that occurs when the image is dispersed by the plurality of filters.

[0008] A second aspect of the technology of the present disclosure is an image processing device according to the first aspect, wherein the optical system has a plurality of apertures, each aperture is provided with a respective filter, the image shift includes image shift based on at least the characteristics of each aperture, and processing is performed based on the characteristics.

[0009] A third aspect of the technique of the present disclosure is the image processing device according to the second aspect, characterized in that the image processing device includes a center of gravity position of an opening.

[0010] A fourth aspect of the technique of the present disclosure is the image processing device according to the third aspect, wherein the center of gravity position is determined based on the position and / or shape of the opening.

[0011] A fifth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fourth aspects, in which the positional shift of the optical image includes at least the positional shift of the optical image caused by the characteristics of the optical system.

[0012] A sixth aspect of the technology of the present disclosure is an image processing device according to any one of the first to fifth aspects, in which the processing is performed on a partial area of ​​the image.

[0013] A seventh aspect of the technology of the present disclosure is an image processing device according to any one of the first to sixth aspects, wherein the image is a spectral image for generating a multispectral image.

[0014] An eighth aspect of the technology of the present disclosure is an image processing device according to the seventh aspect, wherein the image is an image generated by performing interference removal processing on imaging data obtained by imaging using an imaging device.

[0015] A ninth aspect of the technique of the present disclosure is the image processing device according to any one of the first to eighth aspects, wherein the plurality of filters have different wavelength bands.

[0016] A tenth aspect of the technology of the present disclosure is an image processing device according to any one of the first to ninth aspects, in which the multiple filters are arranged in a line around the optical axis.

[0017] An eleventh aspect of the technique of the present disclosure is the image processing device according to the ninth aspect, wherein the processing is performed based on a combination of wavelength bands.

[0018] A twelfth aspect of the technique of the present disclosure is an image processing device according to any one of the first to eleventh aspects, in which processing is performed based on design values ​​related to the optical system.

[0019] A thirteenth aspect of the technique of the present disclosure is the image processing device according to the ninth aspect, wherein the processing is performed based on a correction value for each wavelength band.

[0020] A fourteenth aspect of the technology of the present disclosure is an image processing device according to any one of the ninth, eleventh, and thirteenth aspects, in which the image includes wavelength band images corresponding to each wavelength band.

[0021] A fifteenth aspect of the technology of the present disclosure is an image processing device according to the thirteenth aspect, wherein the image includes wavelength band images corresponding to each wavelength band, and the correction value varies depending on the position of the wavelength band image.

[0022] A sixteenth aspect of the technology of the present disclosure is an image processing device according to the thirteenth aspect, wherein the image includes wavelength band images corresponding to each wavelength band, and the correction value is determined based on the direction and / or amount of positional shift of the wavelength band images relative to a reference position within the image.

[0023] A seventeenth aspect of the technique of the present disclosure is the image processing device according to the fourteenth aspect, wherein the wavelength band image is an image showing a characteristic portion of a subject.

[0024] An 18th aspect of the technology of the present disclosure is an image processing device according to the 16th aspect, wherein the wavelength band image is an image showing a characteristic part of the subject, and the reference position is a position corresponding to the characteristic part.

[0025] A nineteenth aspect of the technique of the present disclosure is the image processing device according to the sixteenth aspect, wherein the reference position is the position of any one of the plurality of wavelength band images.

[0026] A twentieth aspect of the technique of the present disclosure is the image processing device according to the seventeenth aspect, wherein the subject has a point and the characteristic portion is a point.

[0027] A twenty-first aspect of the technique of the present disclosure is the image processing device according to the seventeenth aspect, wherein the subject has a checkerboard pattern, and the characteristic portions are intersections included in the checkerboard pattern.

[0028] A twenty-second aspect according to the technique of the present disclosure is the image processing device according to the seventeenth aspect, wherein the subject is a calibration member.

[0029] A 23rd aspect of the technology of the present disclosure is an image processing device according to the 17th aspect, in which the subject has a plurality of characteristic parts, and the plurality of characteristic parts are arranged in a straight line.

[0030] A 24th aspect of the technology of the present disclosure is an image processing device according to the 17th aspect, in which the subject has a plurality of characteristic parts, and the plurality of characteristic parts are arranged at equal intervals.

[0031] A twenty-fifth aspect of the technique of the present disclosure is the image processing device according to the seventeenth aspect, wherein the subject includes an object for inspection.

[0032] A twenty-sixth aspect of the technique of the present disclosure is the image processing device according to the twenty-fifth aspect, wherein the subject includes a plurality of inspection objects.

[0033] A 27th aspect of the technology of the present disclosure is an image processing device according to any one of the first to 26th aspects, wherein the optical system has polarizing filters provided corresponding to each filter, the multiple polarizing filters have different polarization axes, the imaging device includes an image sensor having multiple pixel blocks, and each pixel block is provided with multiple types of polarizers having different polarization axes.

[0034] A 28th aspect of the technology of the present disclosure is an image processing method applied to an image output from an imaging device equipped with an optical system, wherein the optical system has a plurality of filters arranged around an optical axis, and the image processing method includes performing processing on the image to correct image shift due to positional shift of an optical image that occurs when the image is dispersed by the plurality of filters.

[0035] A 29th aspect of the technology of the present disclosure is a program for causing a computer to perform image processing on an image output from an imaging device equipped with an optical system, wherein the optical system has a plurality of filters arranged around an optical axis, and the image processing includes processing for correcting image shift due to positional shift of an optical image caused by dispersion of the image by the plurality of filters.

[0036] 1 is a perspective view showing an example of an imaging device; FIG. 2 is an exploded perspective view showing an example of a pupil division filter; FIG. 3 is a block diagram showing an example of a hardware configuration of an imaging device; FIG. 4 is an explanatory diagram showing an example of a configuration of a photoelectric conversion element; FIG. 5 is a block diagram showing an example of a manner in which a multispectral image is generated based on a plurality of spectral images; FIG. 6 is a front view showing an example of a multispectral image generated based on a plurality of spectral images having image shifts; FIG. 7 is a schematic diagram showing a first example of a relationship between an aperture formed in the pupil division filter and an optical image; FIG. 8 is a schematic diagram showing a second example of a relationship between an aperture formed in the pupil division filter and an optical image; FIG. 9 is a block diagram showing an example of a functional configuration for executing multispectral image generation processing; FIG. 10 is a block diagram showing an example of the operation of an output value acquisition unit and an interference removal processing unit; FIG. 11 is a block diagram showing an example of the operation of a correction processing unit; FIG. 12 is a block diagram showing an example of the operation of a multispectral image generation unit; FIG. 13 is a block diagram showing an example of an example of an hardware configuration of an imaging device and an example of a functional configuration for executing correction value derivation processing; FIG. 14 is a block diagram showing an example of an optical image formed on a light receiving surface when a dot chart is captured by the imaging device; FIG. 15 is a block diagram showing an example of the operation of a spectral image acquisition unit; FIG. 16 is a block diagram showing an example of the operation of a correction value derivation unit; FIG. 17 is a flowchart showing an example of the flow of multispectral image generation processing; 10 is a block diagram showing an example of the operation of a correction value calculation unit according to a second modified example. FIG. 11 is a block diagram showing an example of an optical image formed on a light receiving surface when a grid chart is imaged by an imaging device. FIG. 12 is a block diagram showing an example of the operation of a spectral image acquisition unit according to a third modified example. FIG. 13 is a block diagram showing an example of the operation of a correction value derivation unit according to the third modified example. FIG. 14 is a block diagram showing an example of the operation of a correction value derivation unit according to a fourth modified example. FIG. 15 is a block diagram showing an example of the operation of a correction value derivation unit according to a fifth modified example. FIG. 16 is a block diagram showing an example of the operation of a correction value derivation unit according to a seventh modified example. FIG. 17 is a block diagram showing an example of the operation of a correction value derivation unit according to an eighth modified example. FIG. 18 is a block diagram showing an example of the operation of a correction value derivation unit according to a ninth modified example.

[0037] Hereinafter, an example of an embodiment of an image processing device, an image processing method, and a program according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0038] First, the terms used in the following description will be explained.

[0039] LED is an abbreviation for "light emitting diode". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". I / F is an abbreviation for "Interface". RAM is an abbreviation for "Random Access Memory". CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro Luminescence". TPU is an abbreviation for "Tensor processing unit". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". ASIC is an abbreviation for "Application Specific Integrated Circuit." FPGA is an abbreviation for "Field-Programmable Gate Array." PLD is an abbreviation for "Programmable Logic Device." SoC is an abbreviation for "System-on-a-Chip." IC is an abbreviation for "Integrated Circuit."

[0040] In the description of this specification, "center" refers to "center" in the sense of including, in addition to the perfect "center," an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, "same" refers to "same" in the sense of including, in addition to the perfect "same," an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, "orthogonal" refers to "orthogonal" in the sense of including, in addition to the perfect "orthogonal," an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, "vertical" refers to "vertical" in the sense of including, in addition to the perfect vertical, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, a "straight line" refers to a straight line that includes, in addition to a perfect straight line, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, "equally spaced" refers to, in addition to a perfectly equal interval, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure.

[0041] 1 , an image capture device 10 includes a lens device 12 and an image capture device body 14. The image capture device 10 is an example of an "image capture device" and an "image processing device" according to the techniques of the present disclosure. The lens device 12 has a pupil division filter 16 that splits incident light into multiple wavelength bands. The image capture device 10 is a multispectral camera that captures light split into multiple wavelength bands by the pupil division filter 16, thereby generating and outputting a multispectral image 74.

[0042] In this embodiment, as an example, a multispectral image generated based on light dispersed into three wavelength bands will be described as the multispectral image 74. Three wavelength bands is merely an example, and four or more wavelength bands may be used. In other words, the imaging device 10 may be a multispectral camera capable of capturing an image of a subject with a higher wavelength resolution than a multispectral camera capable of capturing an image of light dispersed into three wavelength bands.

[0043] Furthermore, the multispectral image 74 may include an image obtained by capturing light in the visible light band, and may also include an image in which light in a wavelength band that cannot be perceived by the human eye (e.g., the near-infrared band and / or the ultraviolet band) is visualized.

[0044] The multispectral image 74 can be used for measuring, inspecting, analyzing, and evaluating a subject as an object to be observed in various fields such as medicine, agriculture, and industry.

[0045] As an example, as shown in FIG. 2, the pupil division filter 16 has a frame 18, spectral filters 20A to 20C, and polarizing filters 22A to 22C.

[0046] The frame 18 has openings 24A to 24C. The openings 24A to 24C have the same shape. As an example, the shape of each of the openings 24A to 24C is a fan shape. However, the shape of each of the openings 24A to 24C may be a shape other than a fan shape (for example, a square shape or a circle). The openings 24A to 24C are formed in a line at equal intervals around the optical axis OA. The center of gravity G of each of the openings 24A to 24C is located at a position off the optical axis OA. Each center of gravity G is the center of the geometric shape of each of the openings 24A to 24C. Hereinafter, when it is not necessary to distinguish between the openings 24A to 24C, each of the openings 24A to 24C will be referred to as an "opening 24." The opening 24 is an example of an "opening" according to the technology of the present disclosure.

[0047] The spectral filters 20A to 20C are provided in the openings 24A to 24C, respectively, and are arranged at equal intervals around the optical axis OA. Each of the spectral filters 20A to 20C is a bandpass filter that transmits light in a specific wavelength band. The spectral filters 20A to 20C have different wavelength bands. Specifically, the spectral filter 20A transmits light in a first wavelength band λ １ and the spectral filter 20B has a second wavelength band λ ２ and the spectral filter 20C has a third wavelength band λ ３ It has.

[0048] Hereinafter, when there is no need to distinguish between the spectral filters 20A to 20C, each of the spectral filters 20A to 20C will be referred to as a "spectral filter 20." The spectral filter 20 is an example of a "filter" according to the technology of the present disclosure. １ , second wavelength band λ ２ , and the third wavelength band λ ３ When there is no need to distinguish between the first wavelength band λ １ , second wavelength band λ ２ , and the third wavelength band λ ３ are referred to as "wavelength band λ."

[0049] Polarizing filters 22A to 22C are provided corresponding to spectral filters 20A to 20C, respectively. Specifically, polarizing filter 22A is provided in opening 24A and overlaps spectral filter 20A. Polarizing filter 22B is provided in opening 24B and overlaps spectral filter 20B. Polarizing filter 22C is provided in opening 24C and overlaps spectral filter 20C.

[0050] Each of the polarizing filters 22A to 22C is an optical filter that transmits light that vibrates in a specific direction. The polarizing filters 22A to 22C have a polarization axis OA with a different polarization angle. Specifically, the polarizing filter 22A has a first polarization angle α １ and the polarizing filter 22B has a second polarization angle α ２ and the polarizing filter 22C has a third polarization angle α ３The polarization axis may be referred to as a transmission axis. For example, the first polarization angle α １ is set to 0°, and the second polarization angle α ２ is set to 45°, and the third polarization angle α ３ is set to 90°.

[0051] Hereinafter, when there is no need to distinguish between the polarizing filters 22A to 22C, each of the polarizing filters 22A to 22C will be referred to as a "polarizing filter 22." The polarizing filter 22 is an example of the "polarizing filter 22" according to the technology of the present disclosure. In addition, the first polarization angle α １ , the second polarization angle α ２ , and the third polarization angle α ３ When it is not necessary to distinguish between the first polarization angle α １ , the second polarization angle α ２ , and the third polarization angle α ３ are referred to as "polarization angles α."

[0052] 2, the number of the openings 24 is three, corresponding to the number of the wavelength bands λ, but the number of the openings 24 may be greater than the number of the wavelength bands λ (i.e., the number of the spectral filters 20). Furthermore, unused openings 24 of the openings 24 may be blocked by a shielding member (not shown). Furthermore, although the spectral filters 20 have different wavelength bands λ in the example shown in FIG. 2, the spectral filters 20 may include spectral filters 20 having the same wavelength band λ.

[0053] 3, the lens device 12 includes an optical system 26, and the imaging device body 14 includes an image sensor 28. The optical system 26 is an example of an "optical system" according to the technology of the present disclosure, and the image sensor 28 is an example of an "image sensor" according to the technology of the present disclosure.

[0054] The optical system 26 has a pupil division filter 16, a first lens 30, and a second lens 32. The first lens 30, the pupil division filter 16, and the second lens 32 are arranged in this order along the optical axis OA of the lens device 12, from the subject 4 side to the image sensor 28 side. In the following description, the subject 4 side may be referred to as the "object side," and the image sensor 28 side may be referred to as the "image side."

[0055] The first lens 30 causes light obtained by light emitted from the light source 2 being reflected by the subject 4 (hereinafter referred to as "subject light") to be incident on the pupil division filter 16. The subject light is an example of "light" according to the technology of the present disclosure. The second lens 32 forms an image of the subject light that has passed through the pupil division filter 16 on the light receiving surface 34A of the photoelectric conversion element 34 provided in the image sensor 28.

[0056] The light source 2 is, for example, an LED light source, a laser light source, an incandescent light bulb, or the like. The light emitted from the light source 2 is unpolarized. The light source 2 may be provided in the imaging device body 14 and / or the lens device 12. The light emitted from the light source may also be natural light.

[0057] The pupil division filter 16 is disposed at the pupil position of the optical system 26. The pupil position refers to the diaphragm surface that limits the brightness of the optical system 26. The pupil position here includes nearby positions, and nearby positions refer to the range from the entrance pupil to the exit pupil. The configuration of the pupil division filter 16 is as described using Figure 2. For convenience, Figure 3 shows a state in which a plurality of spectral filters 20 and a plurality of polarizing filters 22 are linearly arranged along a direction perpendicular to the optical axis OA.

[0058] The image sensor 28 includes a photoelectric conversion element 34 and a signal processing circuit 36. The image sensor 28 is, for example, a CMOS image sensor. In the present embodiment, a CMOS image sensor is used as the image sensor 28, but the technology of the present disclosure is not limited thereto. For example, the technology of the present disclosure can be applied even if the image sensor 28 is another type of image sensor, such as a CCD image sensor.

[0059] As an example, Fig. 3 shows a schematic configuration of the photoelectric conversion element 34. As an example, Fig. 4 specifically shows the configuration of a portion of the photoelectric conversion element 34. The photoelectric conversion element 34 has a pixel layer 38, a polarizing filter layer 40, and a spectral filter layer 42. Note that the configuration of the photoelectric conversion element 34 shown in Fig. 3 is just an example, and the technology of the present disclosure is applicable even if the photoelectric conversion element 34 does not have the spectral filter layer 42.

[0060] The pixel layer 38 has a plurality of pixels 44. The plurality of pixels 44 are arranged in a matrix and form the light receiving surface 34A of the photoelectric conversion element 34. Each pixel 44 is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light.

[0061] Hereinafter, the pixels 44 provided in the photoelectric conversion element 34 will be referred to as "physical pixels 44" to distinguish them from the pixels that form the multispectral image 74. Furthermore, the pixels that form the multispectral image 74 will be referred to as "image pixels."

[0062] The photoelectric conversion element 34 outputs the electrical signals output from the plurality of physical pixels 44 as imaging data to the signal processing circuit 36. The signal processing circuit 36 ​​digitizes the analog imaging data input from the photoelectric conversion element 34. The imaging data is image data representing a captured image 70.

[0063] The plurality of physical pixels 44 form a plurality of pixel blocks 46. Each pixel block 46 is formed by a total of four physical pixels 44, two vertically and two horizontally. For convenience, in FIG. 3 , the four physical pixels 44 forming each pixel block 46 are shown as being linearly arranged in a direction perpendicular to the optical axis OA, but the four physical pixels 44 are arranged adjacent to each other in the vertical and horizontal directions of the photoelectric conversion element 34 (see FIG. 4 ). The physical pixels 44 are an example of a "pixel" according to the technology of the present disclosure, and the pixel blocks 46 are an example of a "pixel block" according to the technology of the present disclosure.

[0064] The polarizing filter layer 40 has multiple types of polarizers 48A to 48D. Each of the polarizers 48A to 48D is an optical filter that transmits light vibrating in a specific direction. The polarizers 48A to 48D have polarization axes OA with different polarization angles θ. Specifically, the polarizer 48A has a first polarization angle θ １ and the polarizer 48B has a second polarization angle θ ２ and the polarizer 48C has a third polarization angle θ ３ and the polarizer 48D has a fourth polarization angle θ ４ As an example, the first polarization angle θ １ is set to 0°, and the second polarization angle θ ２ is set to 45°, and the third polarization angle θ ３ is set to 90°, and the fourth polarization angle θ ４ is set to 135°.

[0065] Hereinafter, when there is no need to distinguish between the polarizers 48A to 48D, the polarizers 48A to 48D will be referred to as "polarizers 48." The polarizer 48 is an example of a "polarizer" according to the technology of the present disclosure. In addition, the first polarization angle θ １ , second polarization angle θ ２ , third polarization angle θ ３ , and the fourth polarization angle θ ４ When there is no need to distinguish between the first polarization angle θ １ , second polarization angle θ ２ , third polarization angle θ ３ , and the fourth polarization angle θ ４ are referred to as "polarization angle θ."

[0066] The spectral filter layer 42 has a B filter 50A, a G filter 50B, and an R filter 50C. The B filter 50A is a blue bandpass filter that transmits the most light in the blue wavelength band among light in a plurality of wavelength bands. The G filter 50B is a green bandpass filter that transmits the most light in the green wavelength band among light in a plurality of wavelength bands. The R filter 50C is a red bandpass filter that transmits the most light in the red wavelength band among light in a plurality of wavelength bands. The B filter 50A, G filter 50B, and R filter 50C are assigned to each pixel block 46.

[0067] For convenience, Fig. 3 shows the B filter 50A, G filter 50B, and R filter 50C arranged in a line in a direction perpendicular to the optical axis OA, but as an example, as shown in Fig. 4, the B filter 50A, G filter 50B, and R filter 50C are arranged in a matrix in a predetermined pattern arrangement. In the example shown in Fig. 4, the B filter 50A, G filter 50B, and R filter 50C are arranged in a matrix in a Bayer array, which is an example of a predetermined pattern arrangement. Note that the predetermined pattern arrangement may be an RGB stripe array, an R / G checkerboard array, an X-Trans (registered trademark) array, a honeycomb array, or the like, in addition to the Bayer array.

[0068] Hereinafter, when it is not necessary to distinguish between the B filter 50A, the G filter 50B, and the R filter 50C, they will each be referred to as "filters 50."

[0069] 3, the imaging device body 14 includes, in addition to the image sensor 28, a control driver 52, an input / output I / F 54, a computer 56, and a display 58. The input / output I / F 54 is connected to the signal processing circuit 36, the control driver 52, the computer 56, and the display 58.

[0070] The computer 56 has a processor 60, a storage 62, and a RAM 64. The processor 60 controls the entire imaging device 10. The processor 60 is, for example, an arithmetic processing device including a CPU and a GPU, and the GPU operates under the control of the CPU and is responsible for executing image-related processing. Here, an arithmetic processing device including a CPU and a GPU is given as an example of the processor 60, but this is merely one example, and the processor 60 may be one or more CPUs that integrate a GPU function, or one or more CPUs that do not integrate a GPU function.

[0071] The processor 60, the storage 62, and the RAM 64 are connected via a bus 66, which is connected to the input / output I / F 54. The processor 60 is an example of a "processor" according to the technology of the present disclosure. The computer 56 is an example of a "computer" according to the technology of the present disclosure.

[0072] The storage 62 is a non-transitory storage medium that stores various parameters and programs. For example, the storage 62 is a flash memory (e.g., an EEPROM). However, this is merely an example, and an HDD or the like may also be used as the storage 62 in addition to the flash memory. The RAM 64 temporarily stores various information and is used as a work memory. Examples of the RAM 64 include a DRAM and / or an SRAM.

[0073] The processor 60 reads out a necessary program from the storage 62 and executes the read program on the RAM 64. The processor 60 controls the control driver 52 and the signal processing circuit 36 ​​in accordance with the program executed on the RAM 64. The control driver 52 controls the photoelectric conversion element 34 under the control of the processor 60.

[0074] The display 58 is, for example, a liquid crystal display or an EL display, and displays various images including the multispectral image 74. Note that the display 58 may be provided in an external device (not shown) communicatively connected to the imaging device 10.

[0075] 5, in the imaging device 10, spectral images 72A to 72C corresponding to the respective wavelength bands λ are generated based on the captured image 70, and a multispectral image 74 is generated based on the spectral images 72A to 72C. １ The spectral image 72B corresponds to the second wavelength band λ ２ The spectral image 72C corresponds to the third wavelength band λ ３ is a spectral image corresponding to

[0076] The specific details of the process of generating the spectral images 72A to 72C based on the captured image 70 and the process of generating the multispectral image 74 based on the spectral images 72A to 72C will be described later. Hereinafter, when there is no need to distinguish between the spectral images 72A to 72C, each of the spectral images 72A to 72C will be referred to as a "spectral image 72." The spectral image 72 is an example of a "spectral image" and an "image" according to the techniques of the present disclosure.

[0077] 6 shows an example of a multispectral image 74. The multispectral image 74 includes a plurality of spectral images 72. Each spectral image 72 includes an image of an object having the shape of the letter "F."

[0078] In the example shown in FIG. 6 , image shifts (hereinafter referred to as "image shifts") occur in the respective spectral images 72. The image shifts differ for each spectral image 72. One factor that causes the image shifts to differ for each spectral image 72 is that the spectral filters 20 (see FIG. 2 ) corresponding to each spectral image 72 are provided in apertures 24 that are formed at different positions with respect to the optical axis OA, resulting in parallax between the multiple apertures 24. When parallax occurs between the multiple apertures 24, a positional shift due to parallax occurs in the optical images formed on the light receiving surface 34A for each wavelength band λ. The positional shifts of the optical images have different magnitudes and directions for each wavelength band λ.

[0079] One factor that causes image shifts that differ for each spectral image 72 is that the parallax generated in the optical system 26 differs for each wavelength band λ due to the multiple openings 24 being formed at different positions. When image shifts occur in each spectral image 72, the image quality of the multispectral image 74 deteriorates compared to when no image shifts occur.

[0080] Furthermore, the positional deviation of the optical image is affected not only by the position of the opening 24 but also by the shape of the opening 24. Here, Fig. 7 shows an example of how light emitted from an object point 76 is imaged on the light-receiving surface 34A when the shape of the opening 24 is sector-shaped, and Fig. 8 shows an example of how light emitted from an object point 76 is imaged on the light-receiving surface 34A when the shape of the opening 24 is rectangular.

[0081] 7 and 8 , as an example, the optical system 26 generally has aberration, so that light does not converge to a single point on the light-receiving surface 34A but spreads out in a manner that corresponds to the shape of the aperture 24. Therefore, an optical image 78 formed on the light-receiving surface 34A has a shape that corresponds to the shape of the aperture 24. If the positional deviation of the optical image 78 is defined by the center of gravity 78A of the shape of the optical image 78 (i.e., the center of the geometric shape), the positional deviation of the optical image 78 differs depending on the shape of the optical image 78. In this way, the positional deviation of the optical image 78 is affected not only by the position of the aperture 24 but also by the shape of the aperture 24.

[0082] That is, the positional deviation of the optical image 78 varies depending on the position G of the center of gravity of the opening 24, which is determined by the position and shape of the opening 24. For example, the position G of the center of gravity of the opening 24 is determined by its relative position with respect to the optical axis OA. Furthermore, the positional deviation of the optical image 78 also varies depending on characteristics of the optical system 26 other than the position G of the center of gravity of the opening 24. Examples of characteristics of the optical system 26 other than the position G of the center of gravity of the opening 24 include the position of the pupil division filter 16.

[0083] In addition to the parallax that differs for each wavelength band λ, other factors that cause image shifts include distortion that occurs in the optical system 26 and / or trapezoidal distortion that occurs when an object plane that is not perpendicular to the optical axis OA is captured.

[0084] In this embodiment, the processor 60 performs the multispectral image generation process described below to obtain a multispectral image 74 of higher image quality than when the multispectral image 74 is generated based on a spectral image 72 in which image shift occurs.

[0085] As an example, as shown in FIG. 9 , a multispectral image generation program 80 is stored in the storage 62. The multispectral image generation program 80 is an example of a "program" according to the technology of the present disclosure. The processor 60 reads the multispectral image generation program 80 from the storage 62 and executes the read multispectral image generation program 80 on the RAM 64. The processor 60 executes a multispectral image generation process for generating a multispectral image 74 in accordance with the multispectral image generation program 80 executed on the RAM 64.

[0086] The multispectral image generation process is realized by the processor 60 operating as an output value acquisition unit 82 , an interference removal processing unit 84 , a correction processing unit 86 , and a multispectral image generation unit 88 in accordance with a multispectral image generation program 80 .

[0087] 10 , when imaging data output from the image sensor 28 is input to the processor 60, the output value acquisition unit 82 acquires an output value Y of each physical pixel 44 based on the imaging data. The output value Y of each physical pixel 44 corresponds to the luminance value of each pixel included in the captured image 70 represented by the imaging data.

[0088] Here, the output value Y of each physical pixel 44 is a value including interference (i.e., crosstalk). １ , second wavelength band λ ２ , and the third wavelength band λ ３ Since light of each wavelength band λ is incident, the output value Y is １ The value according to the amount of light in the second wavelength band λ ２ and the third wavelength band λ ３ The value is a mixture of values ​​according to the amount of light.

[0089] To obtain the multispectral image 74, the processor 60 needs to perform a process of separating and extracting values ​​corresponding to each wavelength band λ from the output value Y for each physical pixel 44, that is, an interference removal process that removes interference, on the output value Y. Therefore, in this embodiment, to obtain the multispectral image 74 (see FIG. 7 ), the interference removal processing unit 84 performs the interference removal process on the output value Y of each physical pixel 44 acquired by the output value acquisition unit 82.

[0090] Here, the interference removal process will be described. The output value Y of each physical pixel 44 includes, for red, green, and blue, the luminance values ​​for each polarization angle θ as components of the output value Y. The output value Y of each physical pixel 44 is expressed by equation (1).

[0091] However, Y θ１＿Ｒ is the red output value Y, and the polarization angle is the first polarization angle θ １ The luminance value of the component Y θ２＿Ｒ is the red output value Y, and the polarization angle is the second polarization angle θ ２ The luminance value of the component Y θ３＿Ｒ is the red output value Y, and the polarization angle is the third polarization angle θ ３ The luminance value of the component Y θ４＿Ｒ is the output value Y of red, and the polarization angle is the fourth polarization angle θ ４ is the luminance value of the component.

[0092] Also, Y θ１＿Ｇ is the green output value Y, and the polarization angle is the first polarization angle θ １ The luminance value of the component Y θ２＿Ｇ is the green output value Y, and the polarization angle is the second polarization angle θ ２ The luminance value of the component Y θ３＿Ｇ is the green output value Y, and the polarization angle is the third polarization angle θ ３ The luminance value of the component Y θ４＿Ｇ is the output value Y of green, and the polarization angle is the fourth polarization angle θ ４ is the luminance value of the component.

[0093] Also, Y θ１＿Ｂ is the blue output value Y, and the polarization angle is the first polarization angle θ １The luminance value of the component Y θ２＿Ｂ is the blue output value Y, and the polarization angle is the second polarization angle θ ２ The luminance value of the component Y θ３＿Ｂ is the blue output value Y, and the polarization angle is the third polarization angle θ ３ The luminance value of the component Y θ４＿Ｂ is the output value Y of blue, and the polarization angle is the fourth polarization angle θ ４ is the luminance value of the component.

[0094] The pixel value X of each image pixel forming the multispectral image 74 is determined by the first polarization angle α １ A first waveband λ having １ The brightness value X of the polarized light (hereinafter referred to as "first wavelength band polarized light") λ１ and the second polarization angle α ２ A second waveband λ having ２ The brightness value X of the polarized light (hereinafter referred to as "second wavelength band polarized light") λ２ and the third polarization angle α ３ A third waveband λ having ３ The brightness value X of the polarized light (hereinafter referred to as the "third wavelength band polarized light") λ３ and as components of the pixel value X. The pixel value X of each image pixel is expressed by equation (2).

[0095] The output value Y of each physical pixel 44 is expressed by equation (3).

[0096] In equation (3), A is an interference matrix. The interference matrix A (not shown) is a matrix that indicates the characteristics of interference. The interference matrix A is determined in advance based on a plurality of known values, such as the spectrum of the subject light, the spectral transmittance of the first lens 30, the spectral transmittance of the second lens 32, the spectral transmittances of the plurality of spectral filters 20, and the spectral sensitivity of the image sensor 28.

[0097] The interference cancellation matrix, which is the generalized inverse matrix of the interference matrix A, is defined as A ＋ In this case, the pixel value X of each image pixel is expressed by equation (4).

[0098] Interference cancellation matrix A ＋Similarly to the interference matrix A, the interference cancellation matrix A is a matrix defined based on the spectrum of the subject light, the spectral transmittance of the first lens 30, the spectral transmittance of the second lens 32, the spectral transmittances of the plurality of spectral filters 20, the spectral sensitivity of the image sensor 28, etc. ＋ is stored in advance in the storage 62.

[0099] The interference cancellation processor 84 uses the interference cancellation matrix A stored in the storage 62 ＋ and the output value Y of each physical pixel 44 acquired by the output value acquisition unit 82, and ＋ and the output value Y of each physical pixel 44, the pixel value X of each image pixel is output according to equation (4).

[0100] Here, as described above, the pixel value X of each image pixel is the luminance value X of the first wavelength band polarized light. λ１ and the brightness value X of the second wavelength band polarized light. λ２ and the brightness value X of the third wavelength band polarized light. λ３ and are included as components of the pixel value X.

[0101] The spectral image 72A of the captured image 70 is a first wavelength band λ １ The brightness value of the light X λ１ (i.e., the brightness values ​​X λ１ The spectral image 72B of the captured image 70 is an image based on the second wavelength band λ ２ The brightness value of the light X λ２ (i.e., the brightness values ​​X λ２ The spectral image 72C of the captured image 70 is an image based on the third wavelength band λ ３ The brightness value of the light X λ３ (i.e., the brightness values ​​X λ３ (Image based on ).

[0102] In this way, the interference removal processing is performed by the interference removal processing unit 84, and the captured image 70 is converted into a first wavelength band polarized light luminance value X λ１ and a spectral image 72A corresponding to the second wavelength band polarized light intensity value X λ２ and a spectral image 72B corresponding to the third wavelength band polarized light intensity value X λ３and a spectral image 72C corresponding to the wavelength band λ of the spectral filter 20. That is, the captured image 70 is separated into spectral images 72 for each wavelength band λ of the spectral filters 20. As described above, image shift occurs in each spectral image 72 (see FIG. 6 ). The image shift differs for each spectral image 72.

[0103] As an example, as shown in FIG. 11 , the correction processing unit 86 performs a process (hereinafter referred to as the “correction process”) to correct image misalignment for each spectral image 72. The correction process is an example of the “image processing” and “calibration process” according to the techniques of the present disclosure. Correction value groups 90A to 90C are pre-stored in the storage 62. The correction value group 90A includes a plurality of correction values ​​92A for correcting the image misalignment of the spectral image 72A. The correction value group 90B includes a plurality of correction values ​​92B for correcting the image misalignment of the spectral image 72B. The correction value group 90C includes a plurality of correction values ​​92C for correcting the image misalignment of the spectral image 72C. The process of deriving the correction values ​​92A to 92C (hereinafter referred to as the “correction value derivation process”) will be described in detail later.

[0104] Hereinafter, when there is no need to distinguish between the correction values ​​92A to 92C, each of the correction values ​​92A to 92C will be referred to as a "correction value 92." Each correction value 92 may be determined for each image pixel included in the spectral image 72, or may be determined for each image region of the spectral image 72. By performing a correction process based on each correction value 92 on each spectral image 72, the image misalignment of each spectral image 72 is corrected.

[0105] 12 , the multispectral image generating unit 88 generates a multispectral image 74 by combining a plurality of spectral images 72 whose image shifts have been corrected by the correction processing unit 86. For example, multispectral image data representing the multispectral image 74 is output to the display 58. The display 58 displays the multispectral image 74 based on the multispectral image data. Note that the multispectral image data may be output to a device (not shown) other than the display 58.

[0106] 13 shows an example of a processing device 100 for deriving the correction values ​​92 corresponding to each of the above-described spectral images 72. The processing device 100 includes a computer 102. The computer 102 includes a processor 104, a storage 106, and a RAM 108. The processor 104, the storage 106, and the RAM 108 are realized by hardware similar to the above-described processor 60, the storage 62, and the RAM 64 (see FIG. 2).

[0107] The storage 106 stores a correction value derivation program 110. The processor 104 reads the correction value derivation program 110 from the storage 106 and executes the read correction value derivation program 110 on the RAM 108. The processor 104 executes a correction value derivation process for deriving each correction value 92 in accordance with the correction value derivation program 110 executed on the RAM 108.

[0108] The correction value derivation process is realized by the processor 104 operating as the spectral image acquisition unit 112 and the correction value derivation unit 114 in accordance with the correction value derivation program 110. Note that although the processing device 100 is described here as a device separate from the imaging device 10, the processing device 100 may also be the imaging device 10. In other words, the correction value derivation process for deriving the correction value 92 may be performed by the imaging device 10.

[0109] 14 to 16 show examples of how the correction value 92 is derived. In the examples shown in FIGS. 14 to 16, a dot chart 120 is used as the subject for deriving the correction value 92. The dot chart 120 has a plurality of dots 122. Each dot 122 is a circular point. The size of the dots 122 may be set arbitrarily. As an example, the plurality of dots 122 are arranged side by side on a pair of diagonal lines (not shown) of the dot chart 120. The dot chart 120 is an example of an "subject" and a "calibration member" according to the technology of the present disclosure. The dots 122 are an example of a "feature portion" and a "point" according to the technology of the present disclosure.

[0110] 14 shows examples of optical images 130A to 130C, optical images 132A to 132C, and optical images 134A to 134C formed on the light receiving surface 34A when the dot chart 120 is imaged by the imaging device 10. The optical images 130A to 130C are optical images corresponding to a first dot 122A located in the upper right among the plurality of dots 122, the optical images 132A to 132C are optical images corresponding to a second dot 122B located in the lower right among the plurality of dots 122, and the optical images 134A to 134C are optical images corresponding to a third dot 122C located in the center among the plurality of dots 122.

[0111] The optical images 130A, 132A, and 134A are optical images in a first wavelength band λ １ The optical images 130B, 132B, and 134B are optical images corresponding to the second wavelength band λ ２ The optical images 130C, 132C, and 134C are optical images corresponding to the third waveband λ ３ The first reference position 136A is a position corresponding to the first dot 122A on the light receiving surface 34A, the second reference position 136B is a position corresponding to the second dot 122B on the light receiving surface 34A, and the third reference position 136C is a position corresponding to the third dot 122C on the light receiving surface 34A.

[0112] The optical images 130A to 130C are misaligned with respect to the first reference position 136A. The direction and amount of misalignment of the optical images 130A to 130C are different from one another. Similarly, the optical images 132A to 132C are misaligned with respect to the second reference position 136B, and the optical images 134A to 134C are misaligned with respect to the third reference position 136C. The direction and amount of misalignment of the optical images 132A to 132C are different from one another, and the direction and amount of misalignment of the optical images 134A to 134C are different from one another.

[0113] Furthermore, for example, the first dot 122A and the second dot 122B are arranged at positions symmetrical in the vertical direction of the dot chart 120, but the direction and amount of positional deviation of the optical images 130A-130C and the optical images 132A-132C differ from one another depending on the position and shape of the opening 24 (see FIG. 2). Therefore, the optical images 130A-130C and the optical images 132A-132C are not symmetrical in the vertical direction of the light-receiving surface 34A, but are asymmetrical. Furthermore, positional deviation also occurs at the center of the light-receiving surface 34A, as with the optical images 134A-134C corresponding to the third dot 122C.

[0114] 15 as an example, the spectral image acquisition unit 112 acquires spectral images 72A to 72C obtained by capturing an image of the dot chart 120. For example, the spectral image 72A includes a wavelength band image 140A corresponding to the optical image 130A, the spectral image 72B includes a wavelength band image 140B corresponding to the optical image 130B, and the spectral image 72C includes a wavelength band image 140C corresponding to the optical image 130C.

[0115] The reference position 142 is a position corresponding to the first dot 122A in each spectral image 72. The wavelength band images 140A to 140C are shifted relative to the reference position 142. The wavelength band images 140A to 140C are shifted in position in different directions and by different amounts. Hereinafter, when there is no need to distinguish between the wavelength band images 140A to 140C, the wavelength band images 140A to 140C will be referred to as "wavelength band images 140." Furthermore, hereinafter, wavelength band images (not shown) corresponding to dots 122 other than the first dot 122A may also be referred to as "wavelength band images 140." The wavelength band image 140 is an example of a "wavelength band image" according to the technology of the present disclosure.

[0116] 16 as an example, the correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value derivation unit 114 acquires each of the spectral images 72A to 72C acquired by the spectral image acquisition unit 112. The correction value derivation unit 114 also identifies the position of the wavelength band image 140A in the spectral image 72A and the position of the reference position 142 in the spectral image 72A based on the position of the first dot 122A in the dot chart 120 (see FIG. 15) and design values ​​related to the optical system 26 (see FIG. 2), etc.

[0117] Similarly, the correction value derivation unit 114 specifies the position of the wavelength band image 140B in the spectral image 72B and the position of the reference position 142 in the spectral image 72B based on the position of the first dot 122A in the dot chart 120 and design values, etc., related to the optical system 26. The correction value derivation unit 114 also specifies the position of the wavelength band image 140C in the spectral image 72C and the position of the reference position 142 in the spectral image 72C based on the position of the first dot 122A in the dot chart 120 and design values, etc., related to the optical system 26. The position of the first dot 122A in the dot chart 120 and the design values, etc., related to the optical system 26 are both known values.

[0118] The design values ​​related to the optical system 26 may include values ​​related to the specifications assigned to the image capture device 10 as a product, or may include values ​​related to the specifications. The design values ​​may also include values ​​related to the characteristics of the optical system 26. The characteristics of the optical system 26 may include, for example, the arrangement position of the pupil division filter 16 and / or the characteristics of the opening 24 (see FIG. 2 ). The characteristics of the opening 24 may include the center of gravity position G of the opening 24 (see FIGS. 7 and 8 ). The center of gravity position G of the opening 24 may be a position determined based on, for example, the position and / or the shape of the opening 24.

[0119] The design values ​​for the optical system 26 may also include values ​​relating to the combination of wavelength bands λ of the spectral filters 20 (see FIG. 2 ). The values ​​relating to the combination of wavelength bands λ may be values ​​indicating the wavelength bands of the spectral filters 20 themselves, or may be values ​​indicating the relationship between the wavelength bands of the spectral filters 20.

[0120] Then, for example, the correction value derivation unit 114 derives the direction and amount of positional deviation of the wavelength band image 140A relative to the reference position 142 based on the spectral image 72A, and derives a correction value 92A for an image pixel corresponding to the wavelength band image 140A among the multiple image pixels included in the spectral image 72A based on the derived direction and amount. The correction value 92A is a correction value that corrects the positional deviation of the wavelength band image 140A. For example, the correction value 92A includes a value indicating a direction opposite to the direction of positional deviation of the wavelength band image 140A relative to the reference position 142 (i.e., the direction of the reference position 142 relative to the wavelength band image 140A), and a value indicating the derived amount.

[0121] Similarly, the correction value derivation unit 114 derives the direction and amount of positional deviation of the wavelength band image 140B relative to the reference position 142 based on the spectral image 72B, and derives a correction value 92B for an image pixel corresponding to the wavelength band image 140B among the multiple image pixels included in the spectral image 72B based on the derived direction and amount. The correction value 92B is a correction value that corrects the positional deviation of the wavelength band image 140B. Furthermore, the correction value derivation unit 114 derives the direction and amount of positional deviation of the wavelength band image 140C relative to the reference position 142 based on the spectral image 72C, and derives a correction value 92C for an image pixel corresponding to the wavelength band image 140C among the multiple image pixels included in the spectral image 72C based on the derived direction and amount. The correction value 92C is a correction value that corrects the positional deviation of the wavelength band image 140C. Each correction value 92 differs depending on the position of the wavelength band image 140.

[0122] As an example, Figures 15 and 16 show an example in which a correction value 92 for an image pixel corresponding to a wavelength band image 140 is derived based on the wavelength band image 140 corresponding to the first dot 122A, but the direction and amount of positional shift are also derived for wavelength band images 140 corresponding to dots 122 other than the first dot 122A, and a correction value 92 for an image pixel corresponding to the wavelength band image 140 is derived based on the derived direction and amount.

[0123] Furthermore, for image pixels (hereinafter referred to as "non-corresponding image pixels") other than image pixels (hereinafter referred to as "corresponding image pixels") corresponding to the wavelength band image 140, the correction value 92 may be derived based on the correction value 92 of the corresponding image pixel. Furthermore, the correction value 92 of the non-corresponding image pixel may be derived based on the position of the non-corresponding image pixel relative to the corresponding image pixel.

[0124] Each correction value 92 may include a correction value for correcting image misalignment due to distortion and / or trapezoidal distortion. The correction value 92 may be derived based on only one of the direction and amount of misalignment. The correction value 92 may also be derived based on values ​​obtained through experiments. While the example given here shows the correction values ​​92 being derived by the processing device 100, the correction values ​​92 may also be derived experimentally by developers of the imaging device 10, etc. The correction values ​​92 derived in the above manner are stored in the storage 62 of the imaging device 10.

[0125] Next, the operation of the imaging device 10 according to this embodiment will be described. An example of the flow of a multispectral image generation process executed by the imaging device 10 is shown in FIG.

[0126] 17 , first, in step ST10, the output value acquisition unit 82 acquires the output value Y of each physical pixel 44 based on the imaging data output from the image sensor 28 (see FIG. 10 ). After the processing of step ST10 is executed, the multispectral image generation processing proceeds to step ST12.

[0127] In step ST12, the interference cancellation processor 84 uses the interference cancellation matrix A stored in the storage 62 ＋ and the output value Y of each physical pixel 44 acquired in step ST10 are acquired, and the acquired interference cancellation matrix A ＋ and the output value Y of each physical pixel 44, the pixel value X of each image pixel is output (see FIG. 10). By performing the interference removal process in step ST12, the captured image 70 is converted into a luminance value X of the first wavelength band polarized light. λ１ and a spectral image 72A corresponding to the second wavelength band polarized light intensity value X λ２and a spectral image 72B corresponding to the third wavelength band polarized light intensity value X λ３ After the process of step ST12 is executed, the multispectral image generation process proceeds to step ST14.

[0128] In step ST14, the correction processing unit 86 performs a correction process to correct image misalignment for each of the spectral images 72A to 72C (see FIG. 11). After the process of step ST14 is performed, the multispectral image generation process proceeds to step ST16.

[0129] In step ST16, the multispectral image generating unit 88 generates a multispectral image 74 by combining the spectral images 72A to 72C that have been corrected in step ST14 (see FIG. 12). After the processing of step ST16 is performed, the multispectral image generating process proceeds to step ST18.

[0130] In step ST18, the processor 60 determines whether a condition for terminating the multispectral image generation process (i.e., a termination condition) is met. An example of the termination condition is a condition in which the user issues an instruction to terminate the multispectral image generation process to the imaging device 10. If the termination condition is not met in step ST18, the determination is negative, and the multispectral image generation process proceeds to step ST10. If the termination condition is met in step ST18, the determination is positive, and the multispectral image generation process ends. Note that the image processing method described above as the operation of the imaging device 10 is an example of an "image processing method" according to the technology of the present disclosure.

[0131] Next, a description will be given of the operation of the processing device 100 according to this embodiment. An example of the flow of correction value derivation processing executed by the processing device 100 is shown in FIG.

[0132] 18, first, in step ST20, the spectral image acquisition unit 112 acquires the spectral images 72A to 72C obtained by the imaging device 10 (see FIG. 15). After the processing of step ST20 is performed, the correction value derivation processing proceeds to step ST22.

[0133] In step ST22, the correction value derivation unit 114 derives the direction and amount of positional shift of the wavelength band images 140A to 140C based on each of the spectral images 72A to 72C, and derives a correction value 92 for the image pixels included in each of the spectral images 72A to 72C based on the derived direction and amount (see FIG. 16 ). The derived correction value 92 is stored in the storage 62 of the imaging device 10. After the processing of step ST22 is executed, the correction value derivation processing proceeds to step ST24.

[0134] In step ST24, the processor 104 determines whether a condition for terminating the correction value derivation process (i.e., a termination condition) is met. An example of the termination condition is that correction values ​​92 have been derived for multiple image pixels included in each of the spectral images 72A to 72C. If the termination condition is not met in step ST24, the determination is negative, and the correction value derivation process proceeds to step ST22. If the termination condition is met in step ST24, the determination is positive, and the correction value derivation process ends.

[0135] Next, the effects of this embodiment will be described.

[0136] In this embodiment, the optical system 26 has a plurality of spectral filters 20 arranged around the optical axis OA (see FIGS. 2 and 3 ). The processor 60 performs a correction process on each spectral image 72 to correct image shifts due to positional shifts of the optical images caused by dispersion by the plurality of spectral filters 20 (see FIG. 11 ). Therefore, a multispectral image 74 with higher image quality can be obtained compared to when the multispectral image 74 is generated based on a spectral image 72 with image shifts.

[0137] The correction process is performed on the multiple spectral images 72 for generating the multispectral image 74. Therefore, image misalignment occurring in the multiple spectral images 72 can be corrected at a stage before the multispectral image 74 is generated.

[0138] The correction process is performed based on, for example, design values ​​related to the optical system 26. Therefore, image misalignment according to the design values ​​related to the optical system 26 can be corrected.

[0139] The design values ​​for the optical system 26 include, for example, the characteristics of the optical system 26. Therefore, by performing correction processing based on the characteristics of the optical system 26, it is possible to correct image misalignment according to the characteristics of the optical system 26.

[0140] The characteristics of the optical system 26 include, for example, the position of the pupil division filter 16. Therefore, by performing correction processing based on the position of the pupil division filter 16, it is possible to correct image deviation according to the position of the pupil division filter 16.

[0141] The characteristics of the optical system 26 include, for example, the characteristics of the aperture 24. Therefore, by performing correction processing based on the characteristics of the aperture 24, it is possible to correct image misalignment according to the characteristics of the aperture 24.

[0142] The characteristics of the opening 24 include, for example, the center of gravity position G of the opening 24. Therefore, by performing a correction process based on the center of gravity position G of the opening 24, it is possible to correct the image misalignment according to the center of gravity position G of the opening 24.

[0143] The center of gravity position G of the opening 24 is a position that is determined based on, for example, the position and / or the shape of the opening 24. Therefore, by performing a correction process based on the position and / or the shape of the opening 24, it is possible to correct image misalignment according to the position and / or the shape of the opening 24.

[0144] The design values ​​for the optical system 26 include, for example, values ​​related to the combination of the wavelength bands λ of the spectral filters 20. Therefore, by performing correction processing based on the values ​​related to the combination of the wavelength bands λ of the spectral filters 20, it is possible to correct image misalignment according to the combination of the wavelength bands λ of the spectral filters 20.

[0145] The correction process is performed based on the correction value 92 for each wavelength band λ. Therefore, even if the image misalignment differs between the spectral images 72 corresponding to each wavelength band λ, the image misalignment can be corrected for each spectral image 72 corresponding to each wavelength band λ.

[0146] Each spectral image 72 includes a wavelength band image 140. The positional shift of the wavelength band image 140 differs for each spectral image 72. Therefore, the image shift of each spectral image 72 can be corrected by correcting the positional shift of the wavelength band image 140 for each spectral image 72 through the correction process.

[0147] The correction value 92 used in the correction process differs depending on the position of the wavelength band image 140. Therefore, by using the correction value 92 according to the position of the wavelength band image 140 in the correction process, it is possible to correct the positional deviation that differs for each wavelength band image 140.

[0148] The correction value 92 is determined based on the direction and / or amount of positional shift of the wavelength band image 140 relative to the reference position 142 in the spectral image 72. Therefore, by using the correction value 92 corresponding to the direction and / or amount of positional shift of the wavelength band image 140 in the correction process, it is possible to correct the positional shift based on the direction and / or amount of positional shift of the wavelength band image 140.

[0149] The correction value derivation process for deriving the correction value 92 uses a dot chart 120 having dots 122. Therefore, in the correction value derivation process, the position of the wavelength band image 140 can be identified using known values ​​such as the positions of the dots 122 in the dot chart 120.

[0150] The dot chart 120 has dots 122 as characteristic portions. Therefore, the position of the wavelength band image 140 can be more easily identified than when a calibration member including characteristic portions with shapes more complex than the dots 122 is used.

[0151] In the correction value derivation process, a position corresponding to the dot 122 in the spectral image 72 is set as the reference position 142. Therefore, in each spectral image 72, the correction value 92 can be derived based on the positional deviation of each wavelength band image 140 relative to the reference position 142.

[0152] In the imaging device 10, an interference removal process is performed on the imaging data. Therefore, even if the output value Y of each physical pixel 44 corresponding to the imaging data is a value that includes interference (i.e., crosstalk), the interference removal process makes it possible to separate and extract values ​​corresponding to each wavelength band λ from the output value Y. This makes it possible to obtain spectral images 72 corresponding to each wavelength band λ from the captured image 70.

[0153] The spectral filters 20 have different wavelength bands λ, so that spectral images 72 corresponding to the respective wavelength bands λ can be obtained.

[0154] The plurality of spectral filters 20 are arranged in a line around the optical axis OA. Therefore, the number of pupil divisions can be ensured in a smaller space than when the plurality of spectral filters 20 are arranged concentrically with the optical axis OA, for example.

[0155] Next, a modification of this embodiment will be described.

[0156] 19 as an example, the correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value derivation unit 114 selects one wavelength band image 140 from among the plurality of wavelength band images 140, and sets the selected wavelength band image 140 as the reference position 142. Here, a case where wavelength band image 140C is set as the reference position 142 will be described as an example.

[0157] The correction value derivation unit 114 derives the correction value 92 in the same manner as in the above embodiment, except that the wavelength band image 140C is set as the reference position 142. For example, the correction value derivation unit 114 derives the direction and amount of positional deviation of the wavelength band image 140A with respect to the reference position 142 based on the spectral image 72A, and derives a correction value 92A for an image pixel corresponding to the wavelength band image 140A based on the derived direction and amount.

[0158] Similarly, the correction value derivation unit 114 derives the direction and amount of positional deviation of the wavelength band image 140B relative to the reference position 142 based on the spectral image 72B, and derives a correction value 92B for the image pixel corresponding to the wavelength band image 140A based on the derived direction and amount. Note that the correction value derivation unit 114 sets the correction value 92C for the image pixel corresponding to the wavelength band image 140C set as the reference position 142 to 0.

[0159] In the first modified example, the position of any one of the plurality of wavelength band images 140 is set as the reference position 142. Therefore, the correction value 92 can be derived based on the positional deviation of the remaining wavelength band images 140 relative to the position of the wavelength band image 140 set as the reference position 142.

[0160] It should be noted that the correction values ​​92 do not necessarily need to include a correction value for correcting image misalignment due to distortion and / or trapezoidal distortion, which allows the amount of correction by the correction values ​​92 to be reduced compared to when the correction values ​​92 include a correction value for correcting image misalignment due to distortion and / or trapezoidal distortion.

[0161] 20 shows an example of a second modified example in which the correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value derivation unit 114 selects one of the multiple spectral images 72. Here, the case where the spectral image 72C is selected will be described as an example. The correction value derivation unit 114 also performs image processing on the spectral image 72C to extract a wavelength band image 140C corresponding to the first dot 122A from the spectral image 72C.

[0162] Then, the correction value derivation unit 114 sets the extracted wavelength band image 140C as the reference position 142. Here, a case where the wavelength band image 140C corresponding to the first dot 122A in the spectral image 72C is set as the reference position 142 will be described as an example.

[0163] The correction value derivation unit 114 also performs image processing on the spectral image 72A to extract a wavelength band image 140A corresponding to the first dot 122A from the spectral image 72A. Similarly, the correction value derivation unit 114 performs image processing on the spectral image 72B to extract a wavelength band image 140B corresponding to the first dot 122A from the spectral image 72B.

[0164] The correction value derivation unit 114 also identifies the position of the wavelength band image 140A within the spectral image 72A by image processing. Similarly, the correction value derivation unit 114 also identifies the position of the wavelength band image 140B within the spectral image 72B by image processing. The correction value derivation unit 114 also identifies the position of the wavelength band image 140C within the spectral image 72C by image processing, and sets the identified position of the wavelength band image 140C as the position of the reference position 142.

[0165] Then, similarly to the first modified example, the correction value derivation unit 114 derives a correction value 92A based on the direction and amount of positional deviation of the wavelength band image 140A relative to the reference position 142. The correction value derivation unit 114 also derives a correction value 92B based on the direction and amount of positional deviation of the wavelength band image 140B relative to the reference position 142. Note that, in the second modified example as well, the correction value 92C is set to 0.

[0166] In the second modified example, the position of the wavelength band image 140 within the spectral image 72 and the position of the reference position 142 within the spectral image 72 are identified by image processing. Therefore, for example, the position of the wavelength band image 140 within the spectral image 72 and the position of the reference position 142 within the spectral image 72 can be identified without using the positions of the dots 122 within the dot chart 120 or design values ​​related to the optical system 26.

[0167] It should be noted that the correction values ​​92 do not necessarily need to include a correction value for correcting image misalignment due to distortion and / or trapezoidal distortion, which allows the amount of correction by the correction values ​​92 to be reduced compared to when the correction values ​​92 include a correction value for correcting image misalignment due to distortion and / or trapezoidal distortion.

[0168] As an example, a third modified example shown in FIGS. 21 to 23 is an example in which a grid chart 150 is used instead of the dot chart 120 of the second modified example. The grid chart 150 has a grid pattern. The grid pattern may be a checkered pattern in which the colors of adjacent regions 152 surrounded by the grid are different, or may be a pattern composed only of grid lines. In the third modified example, a checkered pattern is used as an example of a grid pattern. The colors of adjacent regions 152 may be any combination.

[0169] When a grid chart 150 is used, the intersections 154 included in the grid pattern correspond to characteristic portions. The multiple regions 152 are arranged linearly in the vertical and horizontal directions of the grid chart 150, and the multiple intersections 154 are also arranged linearly in the vertical and horizontal directions of the grid chart 150. As an example, each region 152 is square in shape, and the multiple intersections 154 are arranged at equal intervals in the vertical and horizontal directions of the grid chart 150. The grid chart 150 is an example of a "subject" and a "calibration member" according to the technology of the present disclosure. The intersections 154 are an example of a "characteristic portion" and an "intersection" according to the technology of the present disclosure.

[0170] 21 shows examples of optical images 160A-160C, optical images 162A-162C, and optical images 164A-164C formed on the light receiving surface 34A when the grid chart 150 is imaged by the imaging device 10. Optical images 160A-160C are optical images corresponding to a first intersection 154A located in the second row from the top and second column from the right among the multiple intersections 154, optical images 162A-162C are optical images corresponding to a second intersection 154B located in the second row from the bottom and second column from the right among the multiple intersections 154, and optical images 164A-164C are optical images corresponding to a third intersection 154C located in the center of the multiple intersections 154.

[0171] The optical images 160A, 162A, and 164A are optical images in a first wavelength band λ １ The optical images 160B, 162B, and 164B are optical images corresponding to the second wavelength band λ ２ The optical images 160C, 162C, and 164C are optical images corresponding to the third wavelength band λ ３ The first reference position 166A is a position corresponding to the first intersection 154A within the light-receiving surface 34A, the second reference position 166B is a position corresponding to the second intersection 154B within the light-receiving surface 34A, and the third reference position 166C is a position corresponding to the third intersection 154C within the light-receiving surface 34A.

[0172] The optical images 160A to 160C are misaligned with respect to the first reference position 166A. The direction and amount of misalignment of the optical images 160A to 160C are different from one another. Similarly, the optical images 162A to 162C are misaligned with respect to the second reference position 166B, and the optical images 164A to 164C are misaligned with respect to the third reference position 166C. The direction and amount of misalignment of the optical images 162A to 162C are different from one another, and the direction and amount of misalignment of the optical images 164A to 164C are different from one another.

[0173] 22, the spectral image acquisition unit 112 acquires spectral images 72A to 72C obtained by capturing an image of the grid chart 150. For example, the spectral image 72A includes a wavelength band image 170A corresponding to the optical image 160A, the spectral image 72B includes a wavelength band image 170B corresponding to the optical image 160B, and the spectral image 72C includes a wavelength band image 170C corresponding to the optical image 160C. The wavelength band images 170A to 170C are misaligned in different directions and by different amounts. Hereinafter, when it is not necessary to distinguish between the wavelength band images 170A to 170C, the wavelength band images 170A to 170C will be referred to as "wavelength band images 170."

[0174] 23 as an example, the correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value derivation unit 114 selects one of the multiple spectral images 72. Here, the case where the spectral image 72C is selected will be described as an example. The correction value derivation unit 114 also performs image processing on the spectral image 72C to extract a wavelength band image 170C from the spectral image 72C.

[0175] Then, the correction value derivation unit 114 sets the extracted wavelength band image 170C as the reference position 172. Here, a case where the wavelength band image 170C corresponding to the first intersection point 154A in the spectral image 72C is set as the reference position 172 will be described as an example.

[0176] The correction value derivation unit 114 also performs image processing on the spectral image 72A to extract a wavelength band image 170A corresponding to the first intersection 154A from the spectral image 72A. Similarly, the correction value derivation unit 114 performs image processing on the spectral image 72B to extract a wavelength band image 170B corresponding to the first intersection 154A from the spectral image 72B.

[0177] The correction value derivation unit 114 also identifies the position of the wavelength band image 170A within the spectral image 72A by image processing. Similarly, the correction value derivation unit 114 also identifies the position of the wavelength band image 170B within the spectral image 72B by image processing. The correction value derivation unit 114 also identifies the position of the wavelength band image 170C within the spectral image 72C by image processing, and sets the identified position of the wavelength band image 170C as the position of the reference position 172.

[0178] Then, similarly to the second modified example, the correction value derivation unit 114 derives a correction value 92A based on the direction and amount of positional deviation of the wavelength band image 170A relative to the reference position 172. The correction value derivation unit 114 also derives a correction value 92B based on the direction and amount of positional deviation of the wavelength band image 170B relative to the reference position 172. Note that, in the third modified example as well, the correction value 92C is set to 0.

[0179] In the third modification, the grid chart 150 has a grid pattern as a characteristic portion, which makes it easier to identify the position of the wavelength band image 170 compared to when a calibration member having a characteristic portion with a more complicated shape than a grid pattern is used.

[0180] It should be noted that the correction values ​​92 do not necessarily need to include a correction value for correcting image misalignment due to distortion and / or trapezoidal distortion, which allows the amount of correction by the correction values ​​92 to be reduced compared to when the correction values ​​92 include a correction value for correcting image misalignment due to distortion and / or trapezoidal distortion.

[0181] 24 , the grid chart 150 described in the third modification is used. As in the third modification, the correction value derivation unit 114 derives the direction and amount of positional deviation of the wavelength band image 170 relative to the reference position 172 based on the spectral image 72, and derives a correction value 92 for an image pixel corresponding to the wavelength band image 170 based on the derived direction and amount.

[0182] However, in the fourth modified example, the correction value derivation unit 114 derives each correction value 92 based on, for example, the intersections 154 of each row of the grid chart 150 being aligned in a straight line in the horizontal direction of the grid chart 150. For example, the wavelength band images 170 corresponding to the intersections 154 of each row of the grid chart 150 have different amounts of positional shift in the vertical direction of the spectral image 72, but the correction value 92 derived is such that the positions of the wavelength band images 170 corresponding to the intersections 154 of each row of the grid chart 150 are aligned in the vertical direction of the spectral image 72. As an example, FIG. 24 shows an aspect in which each correction value 92 is derived based on the intersections 154 of the second row from the top of the grid chart 150 being aligned in a straight line in the horizontal direction of the grid chart 150.

[0183] In the fourth modification, for example, each correction value 92 is derived based on the fact that the intersections 154 of each row of the grid chart 150 are aligned in a straight line in the horizontal direction of the grid chart 150. Therefore, for example, it is possible to obtain a correction value 92 that aligns the positions of the wavelength band images 170 corresponding to the intersections 154 of each row of the grid chart 150 in the vertical direction of the spectral image 72. This makes it possible to suppress vertical distortion of the spectral image 72 compared to, for example, a case in which a correction value 92 is derived that shifts the positions of the wavelength band images 170 corresponding to the intersections 154 of each row of the grid chart 150 in the vertical direction of the spectral image 72.

[0184] 25 shows an example of a fifth modification in which the correction value derivation unit 114 may derive each correction value 92 based on the intersections 154 of each column of the grid chart 150 being aligned in a straight line in the vertical direction of the grid chart 150. For example, the wavelength band images 170 corresponding to the intersections 154 of each column of the grid chart 150 have different amounts of positional shift in the horizontal direction of the spectral image 72. However, for example, a correction value may be derived as the correction value 92 such that the positions of the wavelength band images 170 corresponding to the intersections 154 of each column of the grid chart 150 are aligned in the horizontal direction of the spectral image 72. As an example, FIG. 25 shows an example in which the correction values ​​92 are derived based on the intersections 154 in the second column from the right of the grid chart 150 being aligned in a straight line in the vertical direction of the grid chart 150.

[0185] In the fifth modification, for example, it is possible to obtain the correction value 92 such that the positions of the wavelength band images 170 corresponding to the intersections 154 of each column of the grid chart 150 are aligned in the horizontal direction of the spectral image 72. This makes it possible to suppress horizontal distortion of the spectral image 72, compared to the case where, for example, a correction value 92 is derived such that the positions of the wavelength band images 170 corresponding to the intersections 154 of each column of the grid chart 150 are shifted in the horizontal direction of the spectral image 72.

[0186] 26 shows an example of a sixth modified example, in which the correction value derivation unit 114 may derive each correction value 92 based on, for example, the intersections 154 of each row of the grid chart 150 being aligned in a straight line in the horizontal direction of the grid chart 150 and the intersections 154 of each column of the grid chart 150 being aligned in a straight line in the vertical direction of the grid chart 150. As an example, FIG. 26 shows an aspect in which each correction value 92 is derived based on the intersections 154 of the second row from the top of the grid chart 150 being aligned in a straight line in the horizontal direction of the grid chart 150 and the intersections 154 of the second column from the right of the grid chart 150 being aligned in a straight line in the vertical direction of the grid chart 150.

[0187] In the sixth modification, for example, it is possible to obtain the correction value 92 such that the positions of the wavelength band images 170 corresponding to the intersections 154 of each row of the grid chart 150 are aligned in the vertical direction of the spectral image 72 and the positions of the wavelength band images 170 corresponding to the intersections 154 of each column of the grid chart 150 are aligned in the horizontal direction of the spectral image 72. This makes it possible to suppress distortion in the vertical direction and / or horizontal direction of the spectral image 72 compared to the case where, for example, a correction value is derived as the correction value 92 such that the positions of the wavelength band images 170 corresponding to the intersections 154 of each row of the grid chart 150 are shifted in the vertical direction of the spectral image 72 and / or a correction value is derived such that the positions of the wavelength band images 170 corresponding to the intersections 154 of each column of the grid chart 150 are shifted in the horizontal direction of the spectral image 72.

[0188] In the sixth modification, a correction value for correcting image misalignment due to trapezoidal distortion may be included in each correction value 92. In this way, trapezoidal distortion of the spectral image 72 can be suppressed more effectively than when the correction value 92 does not include a correction value for correcting image misalignment due to trapezoidal distortion.

[0189] Furthermore, in the fourth to sixth modified examples, the correction value derivation unit 114 may derive each correction value 92 based on, for example, the fact that the intersections 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150. In this way, horizontal distortion of the spectral image 72 can be suppressed compared to when each correction value 92 is derived regardless of whether the intersections 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150.

[0190] Furthermore, in the fourth to sixth modified examples, the correction value derivation unit 114 may derive each correction value 92 based on, for example, the fact that the intersections 154 of each column of the grid chart 150 are aligned at equal intervals in the vertical direction of the grid chart 150. In this way, vertical distortion of the spectral image 72 can be suppressed compared to when each correction value 92 is derived regardless of whether the intersections 154 of each column of the grid chart 150 are aligned at equal intervals in the vertical direction of the grid chart 150.

[0191] Furthermore, in the fourth to sixth modifications, the correction value derivation unit 114 may derive each correction value 92 based on, for example, that the intersections 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150 and that the intersections 154 of each column of the grid chart 150 are arranged at equal intervals in the vertical direction of the grid chart 150. In this way, vertical and / or horizontal distortion of the spectral image 72 can be suppressed compared to when each correction value 92 is derived regardless of whether the intersections 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150 and / or when each correction value 92 is derived regardless of whether the intersections 154 of each column of the grid chart 150 are arranged at equal intervals in the vertical direction of the grid chart 150.

[0192] Furthermore, when each correction value 92 is derived based on the fact that the intersections 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150 and that the intersections 154 of each column of the grid chart 150 are arranged at equal intervals in the vertical direction of the grid chart 150, each correction value 92 may include a correction value that corrects image misalignment due to trapezoidal distortion. In this way, trapezoidal distortion of the spectral image 72 can be suppressed compared to when the correction values ​​92 do not include a correction value that corrects image misalignment due to trapezoidal distortion.

[0193] Furthermore, for example, when each correction value 92 is derived based on the fact that the intersections 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150 and / or the intersections 154 of each column of the grid chart 150 are arranged at equal intervals in the vertical direction of the grid chart 150, the correction value 92 may be derived in the following manner.

[0194] That is, a correction value may be derived as the correction value 92 that aligns the spacing between the wavelength band images 170 corresponding to the intersections 154 located in an area other than the central area of ​​the grid chart 150 with the spacing between the wavelength band images 170 corresponding to the intersections 154 located in the central area of ​​the grid chart 150. In this way, vertical and / or horizontal distortion of the spectral image 72 can be suppressed compared to when the correction value 92 is derived regardless of the spacing between the wavelength band images 170 corresponding to the intersections 154 located in the central area of ​​the grid chart 150.

[0195] As an example, a seventh modified example shown in FIG. 27 is an example in which, in contrast to the second modified example, instead of the dot chart 120, inspection objects 180A-180D are used as the subject 4. As an example, each of the inspection objects 180A-180D is a test specimen. Each of the inspection objects 180A-180D may be of any type. The inspection objects 180A-180D are different types of objects. Hereinafter, when it is not necessary to distinguish between the inspection objects 180A-180D, each of the inspection objects 180A-180D will be referred to as an "inspection object 180." Each inspection object 180 has a characteristic portion 182. Hereinafter, an example in which there are four inspection objects 180 is given, but the number of the inspection objects 180 may be any number. The inspection object 180 is an example of a "subject" and an "inspection object" according to the technology of the present disclosure.

[0196] 27 shows an example of optical images 190A to 190C formed on the light receiving surface 34A when the inspection objects 180A to 180D are imaged by the imaging device 10. As an example, the optical images 190A to 190C are optical images corresponding to the characteristic portion 182 (hereinafter referred to as the "first characteristic portion 182A") of the first inspection object 180A. The optical image 190A is an optical image corresponding to the characteristic portion 182 of the first inspection object 180A (hereinafter referred to as the "first characteristic portion 182A"). １ and the optical image 190B corresponds to the second wavelength band λ ２ The optical image 190C is an optical image corresponding to the third waveband λ ３ The reference position 192 is a position on the light receiving surface 34A corresponding to the first feature portion 182A. The optical images 190A to 190C are shifted in position with respect to the reference position 192. The optical images 190A to 190C are shifted in position in different directions and by different amounts.

[0197] 27, for example, spectral image 72A includes wavelength band image 200A corresponding to optical image 190A, spectral image 72B includes wavelength band image 200B corresponding to optical image 190B, and spectral image 72C includes wavelength band image 200C corresponding to optical image 190C. The wavelength band images 200A to 200C are misaligned in different directions and amounts. Hereinafter, when there is no need to distinguish between the wavelength band images 200A to 200C, the wavelength band images 200A to 200C will be referred to as "wavelength band images 200."

[0198] The correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value derivation unit 114 selects one of the multiple spectral images 72. Here, a case where the spectral image 72C is selected will be described as an example. The correction value derivation unit 114 also performs image processing on the spectral image 72C to extract a wavelength band image 200C from the spectral image 72C.

[0199] Then, the correction value derivation unit 114 sets the extracted wavelength band image 200C as the reference position 202. Here, a case where the wavelength band image 200C corresponding to the first characteristic portion 182A in the spectral image 72C is set as the reference position 202 will be described as an example.

[0200] Furthermore, the correction value derivation unit 114 performs image processing on the spectral image 72A to extract a wavelength band image 200A corresponding to the first characteristic portion 182A from the spectral image 72A. Similarly, the correction value derivation unit 114 performs image processing on the spectral image 72B to extract a wavelength band image 200B corresponding to the first characteristic portion 182A from the spectral image 72B.

[0201] The correction value derivation unit 114 also identifies the position of the wavelength band image 200A within the spectral image 72A by image processing. Similarly, the correction value derivation unit 114 also identifies the position of the wavelength band image 200B within the spectral image 72B by image processing. The correction value derivation unit 114 also identifies the position of the wavelength band image 200C within the spectral image 72C by image processing, and sets the identified position of the wavelength band image 200C as the position of the reference position 202.

[0202] Then, similarly to the second modified example, the correction value derivation unit 114 derives the correction value 92A based on the direction and amount of positional deviation of the wavelength band image 200A relative to the reference position 202. The correction value derivation unit 114 also derives the correction value 92B based on the direction and amount of positional deviation of the wavelength band image 200B relative to the reference position 202. Note that, in the seventh modified example as well, the correction value 92C is set to 0.

[0203] In the seventh modification, a plurality of test objects 180 are used as the subject 4 when deriving the correction value 92. Therefore, the correction value 92 can be derived without using a calibration member (i.e., a member dedicated to deriving the correction value 92).

[0204] The correction value derivation process according to the seventh modified example may be incorporated into the correction process in the imaging device 10. Then, the correction value 92 derived by the correction value derivation process according to the seventh modified example may be used in the correction process.

[0205] 28 as an example, in contrast to the seventh modification, a plurality of inspection objects 180A-180E of the same type are used as the subject 4. Hereinafter, when there is no need to distinguish between the inspection objects 180A-180E, each of the inspection objects 180A-180E will be referred to as an "inspection object 180." Each inspection object 180 has a first characteristic portion 184 and a second characteristic portion 186.

[0206] The correction value derivation unit 114, for example, sets the wavelength band image 200 corresponding to the first feature portion 184 as the reference position 202, derives the direction and amount of positional shift of the wavelength band image 200 relative to the reference position 202 based on the spectral image 72, and derives a correction value 92 for the image pixel corresponding to the wavelength band image 200 based on the derived direction and amount.

[0207] However, in the eighth modified example, the correction value derivation unit 114 derives each correction value 92 based on, for example, the fact that the position of the first feature portion 184 relative to the second feature portion 186 is the same for each of the test objects 180. For example, the wavelength band images 200 corresponding to the first feature portion 184 of each of the test objects 180 have positional deviations in different directions and / or amounts relative to the wavelength band images 200 corresponding to the second feature portion 186 of each of the test objects 180, but a correction value is derived as the correction value 92 such that the direction and / or amount of positional deviation of the wavelength band images 200 corresponding to the first feature portion 184 of each of the test objects 180 are the same.

[0208] In the eighth modification, for example, each correction value 92 is derived based on the fact that the position of the first feature portion 184 relative to the second feature portion 186 is the same for each test object 180. Therefore, for example, it is possible to obtain a correction value 92 that aligns the direction and / or amount of positional shift of the wavelength band images 200 corresponding to the first feature portion 184 of each test object 180. This makes it possible to suppress vertical and / or horizontal distortion of the spectral image 72, compared to, for example, a case in which the correction value 92 is derived regardless of the position of the first feature portion 184 relative to the second feature portion 186 being the same for each test object 180.

[0209] For example, when each correction value 92 is derived based on the fact that the position of the first feature portion 184 relative to the second feature portion 186 is the same for each inspection object 180, the correction value 92 may be derived in the following manner.

[0210] That is, a correction value may be derived as the correction value 92 that adjusts the spacing between the wavelength band images 200 corresponding to the first feature portion 184 and the second feature portion 186 of the inspection objects 180A-180D located in a region other than the central region of the imaging target region 210 by the imaging device 10 to the spacing between the wavelength band images 200 corresponding to the first feature portion 184 and the second feature portion 186 of the inspection object 180E located in the central region of the imaging target region 210. In this way, vertical and / or horizontal distortion of the spectral image 72 can be suppressed compared to when the correction value 92 is derived regardless of the spacing between the wavelength band images 200 corresponding to the first feature portion 184 and the second feature portion 186 of the inspection object 180E located in the central region.

[0211] The correction value derivation process according to the eighth modified example may be incorporated into the correction process in the imaging device 10. Then, the correction value 92 derived by the correction value derivation process according to the eighth modified example may be used in the correction process.

[0212] 29 as an example, in contrast to the eighth modification, two subjects 220A and 220B are used as subjects 4. As an example, an image capture target area 210 captured by imaging device 10 includes subject area 210A where subject 220A is located, subject area 210B where subject 220B is located, and empty areas 210C and 210D where subject 220A and subject 220B are not located.

[0213] The correction value derivation unit 114 divides each spectral image 72 obtained by imaging the imaging target area 210 by the imaging device 10 into areas 212A to 212D. Area 212A corresponds to the subject area 210A, area 212B corresponds to the subject area 210B, area 212C corresponds to the empty area 210C, and area 212D corresponds to the empty area 210D. In each spectral image 72, area 212A includes a wavelength band image 230A corresponding to the subject 220A, and area 212B includes a wavelength band image 230B corresponding to the subject 220B.

[0214] The correction value derivation unit 114 may divide each spectral image 72 into regions 212A to 212D based on, for example, an instruction received by the imaging device 10 from a user or the like, or may divide each spectral image 72 into multiple regions 212A to 212D based on the presence or absence of a wavelength band image by performing image processing on each spectral image 72. Here, each spectral image 72 is divided into four regions 212A to 212D corresponding to two subjects 220A and 220B, but the number of subjects included in the imaging target region 210 and the number of regions into which each spectral image 72 is divided may be any number. The correction value derivation unit 114 then derives the correction value 92 for the regions 212A and 212B, but does not derive the correction value 92 for the regions 212C and 212D.

[0215] In the ninth modification, the correction value 92 is derived for the region 212A and the region 212B, but the correction value 92 is not derived for the region 212C and the region 212D. Therefore, the load on the processor 104 of the processing device 100 can be reduced compared to when the correction value 92 is derived also for the region 212C and the region 212D.

[0216] Furthermore, when the correction value 92 derived according to the ninth modification is used, the image capturing device 10 performs correction processing on a partial region of the spectral image 72. Therefore, the load on the processor 60 of the image capturing device 10 can be reduced compared to when correction processing is performed on the entire region of the spectral image 72.

[0217] In the above embodiment, the image capturing device 10 performs the correction process on the spectral image 72. However, the spectral image 72 may be input from the image capturing device 10 to an external device, and the external device may perform the correction process on the spectral image 72. In this case, the external device is an example of an "image processing device" related to the technology of the present disclosure.

[0218] In addition, in the above embodiment, the processor 60 is exemplified for the imaging device 10, but instead of the processor 60, or together with the processor 60, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0219] In addition, in the above embodiment, the processor 104 is exemplified for the processing device 100, but instead of or together with the processor 104, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0220] Furthermore, in the above embodiment, the imaging device 10 has been described with reference to an example in which the multispectral image generation program 80 is stored in the storage 62. However, the technology of the present disclosure is not limited to this. For example, the multispectral image generation program 80 may be stored in a portable, non-transitory, computer-readable storage medium (hereinafter simply referred to as a "non-transitory storage medium") such as an SSD or a USB memory. The multispectral image generation program 80 stored in the non-transitory storage medium may be installed in the computer 56 of the imaging device 10.

[0221] Alternatively, the multispectral image generation program 80 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the multispectral image generation program 80 may be downloaded and installed on the computer 56 of the imaging device 10 in response to a request from the imaging device 10.

[0222] Furthermore, it is not necessary to store the entire multispectral image generation program 80 in a storage device such as another computer or server device connected to the imaging device 10, or in the storage 62; only a portion of the multispectral image generation program 80 may be stored therein.

[0223] In the above embodiment, the processing device 100 has been described with reference to an example in which the correction value derivation program 110 is stored in the storage 106. However, the technology of the present disclosure is not limited to this. For example, the correction value derivation program 110 may be stored in a non-transitory storage medium. The correction value derivation program 110 stored in the non-transitory storage medium may be installed in the computer 102 of the processing device 100.

[0224] In addition, the correction value derivation program 110 may be stored in a storage device such as another computer or server device connected to the processing device 100 via a network, and the correction value derivation program 110 may be downloaded in response to a request from the processing device 100 and installed in the computer 102 of the processing device 100.

[0225] Furthermore, it is not necessary to store the entire correction value derivation program 110 in a storage device such as another computer or server device connected to the processing device 100, or in the storage 106; only a portion of the correction value derivation program 110 may be stored.

[0226] Furthermore, although the imaging device 10 has a built-in computer 56 , the technology of the present disclosure is not limited to this. For example, the computer 56 may be provided outside the imaging device 10 .

[0227] Furthermore, although the processing device 100 has a built-in computer 102 , the technology of the present disclosure is not limited to this. For example, the computer 102 may be provided outside the processing device 100 .

[0228] In addition, in the above embodiment, the imaging device 10 is exemplified as a computer 56 including a processor 60, a storage 62, and a RAM 64, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 56. Furthermore, instead of the computer 56, a combination of a hardware configuration and a software configuration may be used.

[0229] In the above embodiment, the processing device 100 is exemplified as a computer 102 including a processor 104, a storage 106, and a RAM 108, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 102. Furthermore, a combination of a hardware configuration and a software configuration may be used instead of the computer 102.

[0230] Furthermore, the following various processors can be used as hardware resources for executing the various processes described in the above embodiments. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing various processes by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, and ASICs, which are processors with a circuit configuration designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute various processes.

[0231] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0232] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes various processes. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute various processes, on a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.

[0233] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The above-described gaze detection process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0234] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0235] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0236] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0237] The following additional notes are provided regarding the above-described embodiments.

[0238] (Supplementary Note 1) A component for deriving a correction value used in a calibration process, wherein the calibration process is performed on an image output from an imaging device equipped with an optical system, the optical system having a plurality of filters arranged around an optical axis and having different wavelength bands, the correction value is a correction value for correcting image shift due to positional shift of an optical image caused by dispersion of the image by the plurality of filters, the image including wavelength band images corresponding to each of the wavelength bands, and the component including a characteristic portion corresponding to a reference position when the correction value is derived based on the direction and / or amount of positional shift of the wavelength band images relative to a reference position in the image. (Supplementary Note 2) An apparatus for deriving correction values ​​used in a calibration process, wherein the calibration process is a calibration process for an image output from an imaging device equipped with an optical system, the optical system having a plurality of filters arranged around an optical axis and having different wavelength bands, the correction values ​​are correction values ​​for correcting image shift due to positional shift of an optical image caused by dispersion of the image by the plurality of filters, the image including wavelength band images corresponding to each of the wavelength bands, the apparatus including a processor, and the processor deriving the correction values ​​based on a direction and / or amount of positional shift of the wavelength band images relative to a reference position in the image. (Supplementary Note 3) A method for deriving a correction value used in a calibration process, wherein the calibration process is performed on an image output from an imaging device equipped with an optical system, the optical system having a plurality of filters arranged around an optical axis and having different wavelength bands, the correction value is a correction value for correcting image shift due to positional shift of an optical image caused by dispersion of the image by the plurality of filters, the image including wavelength band images corresponding to each of the wavelength bands, and the method comprises deriving the correction value based on a direction and / or amount of positional shift of the wavelength band images relative to a reference position in the image.(Supplementary Note 4) A program for causing a computer to execute a process for deriving a correction value to be used in a calibration process, wherein the calibration process is a calibration process for an image output from an imaging device equipped with an optical system, the optical system having a plurality of filters arranged around an optical axis and having different wavelength bands, the correction value is a correction value for correcting image shift due to positional shift of an optical image caused by dispersion of the image by the plurality of filters, the image includes wavelength band images corresponding to each of the wavelength bands, and the process includes deriving the correction value based on the direction and / or amount of positional shift of the wavelength band images relative to a reference position in the image.

Claims

1. An image processing device that is applied to an image output from an imaging device having an optical system, The optical system includes a plurality of filters arranged around an optical axis, The image processing device includes a processor. The processor performs a process for correcting image shift caused by a position shift of an optical image caused by dispersion by the plurality of filters on the image. Image processing device.

2. The optical system has a plurality of apertures, Each of the openings is provided with a respective filter, The image shift includes an image shift based on at least a characteristic of each of the apertures, The processing is performed based on the characteristics. The image processing device according to claim 1 .

3. The characteristics include a center of gravity position of the opening. The image processing device according to claim 2 .

4. The center of gravity position is a position determined based on the position and / or shape of the opening. The image processing device according to claim 3 .

5. The positional deviation of the optical image includes at least a positional deviation of the optical image caused by the characteristics of the optical system. The image processing device according to claim 1 .

6. The processing is performed on a partial area of ​​the image. The image processing device according to claim 1 .

7. The images are spectral images for generating a multispectral image. The image processing device according to claim 1 .

8. The image is an image generated by performing interference removal processing on imaging data obtained by imaging using the imaging device. The image processing device according to claim 7.

9. The plurality of filters have different wavelength bands. The image processing device according to claim 1 .

10. The plurality of filters are arranged side by side around the optical axis. The image processing device according to claim 1 .

11. The processing is performed based on the combination of the wavelength bands. The image processing device according to claim 9 .

12. The processing is performed based on design values ​​related to the optical system. The image processing device according to claim 1 .

13. The processing is performed based on a correction value for each of the wavelength bands. The image processing device according to claim 9 .

14. The image includes a waveband image corresponding to each of the wavebands. The image processing device according to claim 9 .

15. the images include waveband images corresponding to each of the wavebands; The correction value varies depending on the position of the waveband image. The image processing device according to claim 13.

16. the images include waveband images corresponding to each of the wavebands; The correction value is determined based on a direction and / or an amount of positional shift of the wavelength band image relative to a reference position in the image. The image processing device according to claim 13.

17. The wavelength band image is an image showing a characteristic portion of a subject. The image processing device according to claim 14.

18. the wavelength band image is an image showing a characteristic portion of a subject, The reference position is a position corresponding to the characteristic portion. The image processing device according to claim 16.

19. The reference position is the position of any one of the plurality of waveband images. The image processing device according to claim 16.

20. the object has a point; The characteristic portion is the point. The image processing device according to claim 17.

21. the subject has a lattice pattern, The characteristic portions are intersections included in the lattice pattern. The image processing device according to claim 17.

22. The subject is a calibration member. The image processing device according to claim 17.

23. the subject has a plurality of the characteristic portions, The plurality of characteristic portions are arranged in a straight line. The image processing device according to claim 17.

24. the subject has a plurality of the characteristic portions, The plurality of feature portions are arranged at equal intervals. The image processing device according to claim 17.

25. The subject includes an object to be inspected. The image processing device according to claim 17.

26. The subject includes a plurality of the inspection objects. The image processing device according to claim 25.

27. the optical system includes a polarizing filter provided corresponding to each of the filters, The plurality of polarizing filters have different polarization axes, The imaging device includes an image sensor having a plurality of pixel blocks; Each of the pixel blocks is provided with a plurality of types of polarizers having different polarization axes. The image processing device according to claim 1 .

28. An image processing method applied to an image output from an imaging device having an optical system, comprising: The optical system includes a plurality of filters arranged around an optical axis, The image processing method includes performing a process for correcting image shift caused by a position shift of an optical image caused by dispersion by the plurality of filters on the image. Image processing methods.

29. A program for causing a computer to execute image processing on an image output from an imaging device having an optical system, The optical system includes a plurality of filters arranged around an optical axis, The image processing includes a process of correcting image shift due to a position shift of an optical image caused by dispersing the image by the plurality of filters. program.