Information processing method, information processing device, information processing program, and information processing system

Abstract

An embodiment of the present invention provides an information processing method, an information processing device, an information processing program, and an information processing system, by which it is possible to acquire a high image quality multi-spectrum image. In an information processing method according to one embodiment of the present invention, a processor executes: a first parameter acquisition step of acquiring a first crosstalk elimination parameter used for elimination of crosstalk between a plurality of image signals; an information acquisition step in which first imaging is performed such that first image signals that are a plurality of image signals from a plurality of beams of light are acquired as information indicating wavelength characteristics of a subject; and a second parameter acquisition step in which, with reference to the information acquired by the first imaging, a second crosstalk elimination parameter is acquired, the second crosstalk elimination parameter being used to eliminate crosstalk among second image signals that are a plurality of image signals from second imaging.

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing device, an information processing program, and an information processing system for processing multispectral images. [Background technology]

[0002] Regarding techniques for capturing multispectral images, for example, Patent Documents 1 and 2 describe suppressing the influence of ghosts. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO15 / 004886 publication [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-36024 Summary of the Invention

[0004] One embodiment of the technique of the present disclosure provides an information processing method, an information processing device, an information processing program, and an information processing system for acquiring a multispectral image with good image quality. [Means for solving the problem]

[0005] An information processing method according to a first aspect of the present invention is an information processing method for acquiring interference removal parameters for a split-pupil multispectral camera, the information processing method being executed by an information processing device having a processor, the split-pupil multispectral camera comprising a plurality of aperture regions arranged at or near the pupil, a plurality of optical filters arranged in the plurality of aperture regions and transmitting a plurality of light beams having at least some different wavelength bands, and an image sensor outputting a plurality of image signals corresponding to the plurality of light beams, the processor including: a first parameter acquisition step of acquiring a first interference removal parameter used to remove interference from the plurality of image signals; The image capturing apparatus executes an information acquisition step of acquiring first image signals, which are multiple image signals for multiple light beams, as information indicating the wavelength characteristics of the subject, and a second parameter acquisition step of acquiring, by referring to the information acquired by the first shooting, a second interference elimination parameter used to eliminate interference from second image signals, which are multiple image signals from the second shooting, wherein in the second parameter acquisition step, the processor acquires a second interference elimination parameter that makes the difference between the information acquired by the first shooting and the second image signal from which interference has been eliminated using the second interference elimination parameter smaller than the difference between the information acquired by the first shooting and the second image signal from which interference has been eliminated using the first interference elimination parameter.

[0006] The information processing method according to the second aspect is the first aspect, wherein the first shooting is preliminary shooting, the second shooting is main shooting, and the focus position of the first shooting is the same as the focus position of the second shooting.

[0007] The information processing method according to the third aspect is the first or second aspect, in which, in the information acquisition step, the processor acquires wavelength intensities of the subject in a state in which light is not transmitted through some of the multiple opening areas.

[0008] The information processing method according to the fourth aspect is any one of the first to third aspects, in which, in the information acquisition step, the processor acquires as information a first image signal output by photographing a subject with known wavelength characteristics while blocking some of the multiple opening areas and leaving the remaining opening areas open.

[0009] In the information processing method of the fifth aspect, in the fourth aspect, the processor acquires a first image signal as information in an information acquisition process while physically blocking a portion of the opening area by placing a light-blocking member that does not transmit light over the portion of the opening area.

[0010] The information processing method of the sixth aspect is the same as that of the fourth aspect, in which the pupil-splitting multispectral camera includes a plurality of first polarizing elements arranged in a plurality of aperture regions that transmit light of different polarization angles, and in the information acquisition process, the processor acquires a first image signal as information while optically shielding some of the aperture regions by arranging a second polarizing element having a polarization angle perpendicular to the first polarizing element arranged in the some of the aperture regions.

[0011] In the information processing method of the seventh aspect, in the fourth aspect, the processor acquires a first image signal in an information acquisition step by photographing a subject with known wavelength characteristics in a state in which an optical filter that transmits one of a plurality of light beams and does not transmit the remaining light beams is positioned closer to the subject than the plurality of optical filters.

[0012] An information processing method according to an eighth aspect is any one of the first to third aspects, wherein in the information acquiring step, the processor acquires information about the subject having unknown wavelength characteristics as first information by capturing an image in a state where the plurality of optical filters are not arranged in the plurality of aperture regions and an optical filter that transmits one of the plurality of light beams and does not transmit the remaining light beams is arranged on the subject side of the plurality of optical filters, and acquires information about the subject having unknown wavelength characteristics as first information by capturing an image in a state where the plurality of optical filters are .... multiple By photographing the subject with the optical filter positioned closer to the subject, information about the subject whose wavelength characteristics are unknown is acquired as second information, and in the second parameter acquisition step, a parameter for correcting the second information is acquired as a second interference removal parameter based on the first information.

[0013] An information processing method according to a ninth aspect is any one of the first to eighth aspects, wherein the pupil-splitting multispectral camera comprises a plurality of first polarizing elements arranged in a plurality of aperture regions that transmit light of different polarization angles, and a plurality of second polarizing elements arranged on the image sensor that transmit light of polarization angles corresponding to the polarization angles of the plurality of first polarizing elements, and the processor acquires, as information, a plurality of image signals corresponding to the polarization angles of the plurality of first polarizing elements in the information acquisition process.

[0014] An information processing device according to a tenth aspect of the present invention is an information processing device for acquiring interference removal parameters for a pupil-splitting multispectral camera having a plurality of aperture regions arranged at or near the pupil, a plurality of optical filters arranged in the plurality of aperture regions that transmit a plurality of light beams having at least some different wavelength bands, and an image sensor that outputs a plurality of image signals corresponding to the plurality of light beams, and the information processing device includes a processor, wherein the processor executes a first parameter acquisition process for acquiring a first interference removal parameter used to remove interference from the plurality of image signals, an information acquisition process for acquiring a plurality of image signals for the plurality of light beams by a first image capture as information indicating the wavelength characteristics of the subject, and a second parameter acquisition process for acquiring a second interference removal parameter used to remove interference from second image signals, which are a plurality of image signals by a second image capture, by referring to the information acquired by the first image capture, and in the second parameter acquisition process, the processor acquires a second interference removal parameter that makes the difference between the information acquired by the first image capture and the second image signal from which interference has been removed using the second interference removal parameter smaller than the difference between the information acquired by the first image capture and the second image signal from which interference has been removed using the first interference removal parameter.

[0015] An information processing program according to an eleventh aspect of the present invention is an information processing program that causes an information processing device having a processor to execute an information processing method for acquiring interference removal parameters for a split-pupil multispectral camera that includes: a plurality of aperture regions arranged at or near a pupil position; a plurality of optical filters arranged in the plurality of aperture regions and that transmit a plurality of light beams having at least some different wavelength bands; and an image sensor that outputs a plurality of image signals corresponding to the plurality of light beams, the information processing program including: a first parameter acquisition step of acquiring a first interference removal parameter used for removing interference from the plurality of image signals; The image processing device executes an information acquisition step of acquiring a plurality of image signals for a plurality of light beams as information indicating the wavelength characteristics of the subject, and a second parameter acquisition step of acquiring a second interference removal parameter used to remove interference from a second image signal, which is a plurality of image signals from a second photograph, by referring to the information acquired by the first photograph. In the second parameter acquisition step, the processor is caused to acquire a second interference removal parameter that makes the difference between the information acquired by the first photograph and the second image signal from which interference has been removed using the second interference removal parameter smaller than the difference between the information acquired by the first photograph and the second image signal from which interference has been removed using the first interference removal parameter.

[0016] An information processing system according to a twelfth aspect of the present invention comprises a pupil-splitting multispectral camera having a plurality of aperture regions arranged at or near the pupil, a plurality of optical filters arranged in the plurality of aperture regions that transmit a plurality of light beams having at least some different wavelength bands, and an image sensor that outputs a plurality of image signals corresponding to the plurality of light beams, and an information processing device according to the tenth aspect.

[0017] The information processing system according to the thirteenth aspect is the twelfth aspect, in which the processor performs interference removal on a plurality of image signals using interference removal parameters, and outputs the plurality of image signals in an interference-removed state to an output device. [Brief explanation of the drawings]

[0018] [Figure 1]FIG. 1 is a diagram showing a schematic configuration of an imaging system 10 according to the first embodiment. [Figure 2] FIG. 2 is a perspective view showing the configuration of the lens device. [Figure 3] FIG. 3 is a cross-sectional view showing the configuration of the lens device. [Figure 4] FIG. 4 is an external view showing an example of a filter unit. [Figure 5] FIG. 5 is an external view showing another example of the filter unit. [Figure 6] FIG. 6 is a diagram showing how a filter set is arranged in a frame. [Figure 7] FIG. 7 is a diagram showing the polarization direction of a polarizing filter. [Figure 8] FIG. 8 is a diagram showing the configuration of an image sensor. [Figure 9] FIG. 9 is a diagram illustrating the functional configuration of the processor. [Figure 10] FIG. 10 is a diagram showing how a light-shielding member shields some of the opening areas and leaves other opening areas open. [Figure 11] FIG. 11 is a diagram showing a state of preliminary imaging for obtaining interference removal parameters. [Figure 12] FIG. 12 is a diagram showing the actual photographing in the first mode. [Figure 13] FIG. 13 is a diagram showing how optical blocking is performed by changing the wavelength band of illumination light. [Figure 14] FIG. 14 is a diagram showing a state in which a diffusion plate 99A is placed in front of a light source 99. As shown in FIG. [Figure 15] FIG. 15 is a diagram showing the state of preliminary imaging in the third mode. [Figure 16] FIG. 16 is a diagram showing the state of actual photography in the third mode. [Figure 17] FIG. 17 is a diagram showing a state in which a color filter is arranged closer to the subject side than the filter unit. [Figure 18] FIG. 18 shows a state in which a color filter is attached to a separate frame and inserted closer to the subject side than the filter unit. [Figure 19] FIG. 19 is a diagram showing a state in which a color filter is attached closer to the subject than the first lens. [Figure 20] FIG. 20 is a diagram showing a state in which a color filter is arranged closer to the subject side than the frame body. DETAILED DESCRIPTION OF THE INVENTION

[0019] <Interference removal for multispectral cameras> Among imaging devices that capture multispectral images, those that use polarizing elements are known. In such imaging devices, wavelength information mixed at each polarization pixel (e.g., polarization direction: 0°, 45°, 90°, 135°) is acquired, and interference removal (calculation using an inverse matrix) is performed based on the mixing ratio to generate images for each wavelength. However, if interference removal is performed using a theoretically calculated interference removal matrix, it is not possible to generate an accurate multispectral image due to differences (e.g., changes in polarization degree due to refraction, differences in ghosting and / or flare) between the development environment and the actual environment (the environment in which actual shooting and image processing are performed using the imaging device). Note that, while interference removal is performed for each pixel as an example in this embodiment, the present invention can be applied to a wide range of situations, such as removing interference only from pixels or regions that are significantly affected by interference.

[0020] Under these circumstances, the inventors of the present invention have conducted extensive research and come up with an idea for an information processing method, an information processing device, an information processing program, and an information processing system that can acquire multispectral images with high image quality. Hereinafter, several embodiments of the present invention will be described with reference to the accompanying drawings.

[0021] [First embodiment] [Schematic configuration of imaging system] FIG. 1 is a diagram showing a schematic configuration of an imaging system 10 (imaging system, imaging device, information processing system, information processing device) according to a first embodiment. The imaging system 10 is composed of a lens device 100 (a split-pupil multispectral camera), an imaging device main body 200 (a split-pupil multispectral camera), a display device 300 (such as a liquid crystal display), a storage device 310 (such as a magneto-optical recording device or semiconductor memory), and an operation device 320 (such as a keyboard, mouse, buttons, switches, or dials), and is capable of capturing an image of a light source 99 (subject) and acquiring a multispectral image. The configuration of each of these components will be described in detail below. The imaging system 10 is capable of calculating interference removal parameters based on the acquired multispectral image and performing interference removal using the calculated interference removal parameters.

[0022] [Lens device configuration] Fig. 2 is a perspective view showing the configuration of lens device 100, and Fig. 3 is a cross-sectional view showing the configuration of lens device 100. As shown in these figures, lens device 100 has an optical system including a first lens 110 and a second lens 120 arranged in a lens barrel 102, and these lenses move forward and backward in the direction of optical axis L by rotating a first lever 104 and a second lever 106, respectively, to adjust the focal length and / or image magnification. Note that first lens 110 and second lens 120 may be lens groups consisting of multiple lenses.

[0023] In addition, a slit 108 is formed in the lens barrel 102 at or near the pupil position of the lens device 100, and a filter unit 134 is inserted into this slit 108 to Light The first lens 110 and the second lens 120 are arranged so as to coincide with the optical axis L of the optical system.

[0024] [Filter unit configuration] 4 is an external view showing an example of filter unit 134. Filter unit 134 includes frame 135, which has four opening regions (opening regions 135A to 135D; multiple opening regions) formed therein. The center of gravity of opening regions 135A to 135D is center of gravity 135G. Filter sets 137 (filter sets 137A to 137D; multiple optical filters, multiple first filters) can be arranged in opening regions 135A to 135D. The configuration of filter set 137 will be described later.

[0025] 5 is an external view showing another example of the frame (filter unit). As shown in the figure, some of the apertures are opened depending on the number of images to be acquired. region 5(a) , one of the opening areas (here, opening area 135D) may be shielded by light-shielding member 135E, or a frame 133 having three opening areas 133A to 133C may be used, as shown in part (b) of the same figure.

[0026] [Filter set configuration] 6 is a diagram showing how filter sets are arranged in a frame. In the example shown in the figure, filter sets 137 (filter sets 137A to 137D) include color filters and polarizing filters, and are arranged in aperture regions 135A to 135D, respectively. Specifically, filter set 137A includes color filter 138A and polarizing filter 139A, filter set 137B includes color filter 138B and polarizing filter 139B, filter set 137C includes color filter 138C and polarizing filter 139C, and filter set 137D includes color filter 138D and polarizing filter 139D.

[0027] It is preferable that color filters 138A to 138D are optical filters that transmit light having at least some different wavelength bands, and polarizing filters 139A to 139D (first polarizing members) are polarizing filters that transmit light having different polarization angles.

[0028] 7 is a diagram showing the polarization directions of polarizing filters 139A to 139D. As shown in the figure, the polarization directions of polarizing filters 139A to 139D are 0°, 45°, 90°, and 135°, respectively. In the present invention, the first polarizing member may be a filter that polarizes using a polarizing film, or a filter that polarizes using a wire grid or multiple slits.

[0029] [Image sensor configuration] The image sensor 138 is a CMOS (Complementary Metal-Oxide Semiconductor) type image sensor (imaging element) that outputs a plurality of image signals corresponding to the plurality of light beams transmitted by the color filters 138A to 138D. As shown in FIG. 8, the image sensor 138 is a monochrome imaging element having a pixel array layer 211, a polarizing filter element array layer 213, and a microlens array layer 215. The layers are arranged in this order from the image plane side to the object side: pixel array layer 211, polarizing filter element array layer 213, and microlens array layer 215. Note that the image sensor 138 is not limited to a CMOS type, and may be an XY address type or a CCD (Charge Coupled Device) type image sensor.

[0030] The pixel array layer 211 is configured by arranging a large number of photodiodes 211A (multiple pixel groups) two-dimensionally. One photodiode 211A constitutes one pixel. The photodiodes 211A are regularly arranged along the horizontal direction (x direction) and vertical direction (y direction).

[0031] The polarization filter element array layer 213 is configured by two-dimensionally arranging four types of polarization filter elements 214A, 214B, 214C, and 214D, each with a different polarization direction (the polarization direction of the light to be transmitted). The polarization directions of the polarization filter elements 214A, 214B, 214C, and 214D can be, for example, 0°, 45°, 90°, and 135°. Furthermore, these polarization directions can correspond to the polarization directions of the polarization filters 139A to 139D in the filter unit 134 described above (see FIG. 7). The image sensor 138 includes a plurality of image groups that receive any of the light transmitted through the plurality of aperture areas by the polarization filter elements 214A to 214D. These polarization filter elements 214A to 214D generate a plurality of image groups. D are arranged at the same intervals as the photodiodes 211A and are provided for each pixel.

[0032] The microlens array layer 215 includes microlenses 216 arranged for each pixel.

[0033] The image sensor 138 includes an analog amplifier, an A / D converter (Analog-to-Digital Converter), and an image sensor driver, none of which are shown.

[0034] [Processor Configuration] 9 is a diagram showing the configuration of the processor 230. As shown in the diagram, the processor 230 includes an imaging control unit 232, an image acquisition unit 234, a parameter acquisition unit 236, an interference removal unit 238, a display control unit 240, and a recording control unit 242, and executes an information acquisition step (information acquisition processing) for acquiring a plurality of image signals, a parameter acquisition step (parameter acquisition processing) for acquiring interference removal parameters, an interference removal step (interference removal processing) for removing interference, etc., as will be described in detail below.

[0035] The functions of the processor 230 described above can be realized using various processors. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors described above also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing. The various processors described above also include a PLD (Programmable Logic Device), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing. Furthermore, the various processors described above also include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processing.

[0036] Note that each function of processor 230 may be realized by one processor or by multiple processors. Also, one processor may correspond to multiple functions. Furthermore, each function of processor 230 may be realized by a circuit, or some of the functions may be realized by a circuit and the rest may be realized by a processor.

[0037] When the processor or electrical circuit described above executes software (programs), processor-readable code for the software to be executed and data required for executing the software are stored in a non-transitory recording medium such as flash memory 244, and the processor references the software and data. The software stored in the non-transitory recording medium includes an adjustment program for executing the adjustment method according to this embodiment. Instead of flash memory 244, the code and data may be recorded in a non-transitory recording medium using various types of magneto-optical recording devices, semiconductor memory, or the like. Here, "semiconductor memory" includes flash memory, ROM (Read Only Memory), and EEPROM (Electronically Erasable and Programmable ROM). When processing using software, RAM 246, for example, is used as a temporary storage area.

[0038] [Get interference cancellation parameters] The acquisition of interference removal parameters (execution of an information processing method and an information processing program) executed by the imaging system 10 (information processing device) configured as described above will be described below. Note that the following describes a case in which images are acquired in three wavelength bands using three aperture regions 135A to 135C, and interference removal parameters are acquired based on these images (in the following embodiment, aperture region 135D is always shielded and is not used to acquire interference removal parameters).

[0039] The parameter acquisition unit 236 (processor) acquires, separately from acquiring interference removal parameters (second interference removal parameters) in the following manners, interference removal parameters (first interference removal parameters) to be used for removing interference from a plurality of image signals obtained by capturing images in a state where noise in a real environment, such as a development environment, is not taken into consideration (first parameter acquisition step, first parameter acquisition process).

[0040] [First aspect] In the first mode, some (two) of the three aperture regions 135A-135C (plural aperture regions) are physically blocked and the remaining (one) aperture region is left open, and an image of a subject with known wavelength characteristics is captured (preliminary capture, first capture). Images are obtained by repeatedly capturing images with different apertures being opened, and interference removal parameters are obtained based on these images. Note that the focus position in the preliminary capture (first capture) is the same as the focus position in the main capture (actual capture; second capture). Here, "equivalent" focus positions includes not only cases where the focus positions are exactly the same, but also cases where there is a deviation to an extent that the impact on interference removal is tolerable.

[0041] FIG. 10 shows a partial opening of the light blocking member 131. region and shield the remaining openings region 10A is a diagram showing a state in which the opening region 135A is opened. Specifically, as shown in part (a) of Fig. 10, a light-shielding member 131A corresponding to the opening region 135A is arranged on the subject side of the frame body 135 to shield the opening region other than the opening region 135A, a light-shielding member 131B corresponding to the opening region 135B is arranged on the subject side of the frame body 135 to shield the opening region other than the opening region 135B, and a light-shielding member 131C corresponding to the opening region 135C is arranged on the subject side of the frame body 135 to shield the opening region other than the opening region 135C. Part (b) of the same figure shows a state in which the opening region 135A is opened by the light-shielding member 131A.

[0042] It is preferable to use a material as the light-shielding member 131 that does not transmit light (light in the wavelength band used to acquire an image) at all or substantially (the influence of transmission is within an acceptable range in terms of the accuracy of interference removal).

[0043] Furthermore, filter sets 137A to 137C (color filters and polarizing filters) are arranged on the imaging device main body 200 side of frame 135 (see FIG. 6). Specifically, a color filter 138A of a transmitted light wavelength band λ1 and a polarizing filter 139A with a polarization angle of 0° are arranged in aperture region 135A, a color filter 138B of a transmitted light wavelength band λ2 and a polarizing filter 139B with a polarization angle of 45° are arranged in aperture region 135B, and a color filter 138C of a transmitted light wavelength band λ3 and a polarizing filter 139C with a polarization angle of 90° are arranged in aperture region 135C.

[0044] Fig. 11 is a diagram showing a preliminary image (first image) taken for acquiring interference removal parameters in the first mode. Part (a) of Fig. 11 shows a state in which the aperture region 135A that transmits light in the wavelength band λ1 (polarization angle 0 deg) is opened, and the remaining aperture regions are shielded by the light-shielding member 131A.

[0045] The imaging control unit 232 (processor) controls the reading of image signals from the image sensor 210 (image sensor) in response to an imaging instruction operation on the operation device 320 (such as a shutter button), and acquires the image signal output by the image capture as information indicating the wavelength characteristics of the light source 99 (subject) (information acquisition step, information acquisition process). The wavelength characteristics of the light source 99 are assumed to be known. Examples of "subjects with known wavelength characteristics" include white paper and a color chart.

[0046] In this state, the image signals output from the four types of pixels (pixels corresponding to the polarization filter elements 214A to 214D) of the image sensor 210 are assumed to be x0, x45, x135, and x90.

[0047] Similarly, part (b) of FIG. 11 shows a state in which the aperture region 135B that transmits light in the wavelength band λ2 (polarization angle 45°) is opened, and the remaining aperture regions are shielded by the light-shielding member 131B. Image signals output from the image sensor 210 in this state are designated y0, y45, y135, and y90. The process (processing) in which the imaging control unit 232 acquires these image signals is the information acquisition process (information acquisition processing). Part (c) of FIG. 11 shows a state in which the aperture region 135C that transmits light in the wavelength band λ3 (polarization angle 90°) is opened, and the remaining aperture regions are shielded by the light-shielding member 131C. Image signals output from the image sensor 210 in this state are designated z0, z45, z135, and z90. The process in which the imaging control unit 232 acquires these image signals is also the information acquisition process.

[0048] In the first embodiment and each of the following embodiments (including modified examples), it is not necessary to open the opening regions (acquire image signals) in the order of the opening regions 135A to 135C.

[0049] In this way, the imaging control unit 232 (processor) acquires four image signals (multiple image signals; first image signals) for light (multiple lights) in the wavelength bands λ1, λ2, and λ3 as information indicating the wavelength characteristics of the light source 99 (subject) (information acquisition process, information acquisition processing).

[0050] In the above example, one of the three opening areas (opening areas 135A to 135C) is opened and the remaining two opening areas are shielded. However, when obtaining an interference removal parameter according to the present invention, the number of shielded opening areas is not limited to two, and it is sufficient that at least one opening area is shielded.

[0051] 12 is a diagram showing the main imaging (second imaging) in the first mode. In the main imaging, the imaging control unit 232 (processor) acquires an image signal in a state where no light blocking member is placed.

[0052] [Calculation example of interference cancellation parameters in the first embodiment] [About the first opening area] If the intensity (known) of the light passing through the first aperture region (aperture region 135A) is Iλ1 and the interference cancellation matrix (matrix composed of interference cancellation parameters) is “interference cancellation matrix A,” the following (Equation 1) holds.

[0053]

number

[0054] Left-hand side matrix (x0,x45,x135,x90) T The components of the matrix (Iλ1,0,0) on the right side are the sensor intensities of the pixels at polarization angles of 0°, 45°, 135°, and 90° of the image sensor 210 (first image signals which are a plurality of image signals for a plurality of lights). T The components of are the intensities of light in wavelength bands λ1, λ2, and λ3 that have passed through the filter unit 134. (Equation 1) means that "if interference is removed from the sensor output intensity when only the first aperture area is open, the intensity of light in wavelength band λ1 should be Iλ1, and the intensities of light in the other wavelength bands λ2 and λ3 should be 0." Note that in the following, the matrix (x0, x45, x135, x90) on the left side of the equation T is sometimes referred to as "matrix X".

[0055] [About the second opening area] As in the case of the first aperture region, if the intensity (known) of the light passing through the second aperture region (designated aperture region 135B) is Iλ2 and the interference cancellation matrix (matrix composed of interference cancellation parameters) is “interference cancellation matrix A,” the following (Equation 2) holds.

[0056]

number

[0057] Left side matrix (y0,y45,y135,y90) TThe components of the matrix (0, Iλ2, 0) on the right side are the sensor intensities of the pixels at polarization angles of 0°, 45°, 135°, and 90° of the image sensor 210 (first image signals, which are multiple image signals for multiple lights). T The components of are the intensities of light in wavelength bands λ1, λ2, and λ3 that have passed through the filter unit 134. (Equation 2) means that "if interference is removed from the sensor output intensity when only the second aperture area is open, the intensity of light in wavelength band λ2 should be Iλ2, and the intensities of light in the other wavelength bands λ1 and λ3 should be 0." Note that in the following, the matrix (y0, y45, y135, y90) on the left side of the equation T is sometimes referred to as "matrix Y".

[0058] [About the third opening area] As in the cases of the first and second aperture regions, if the intensity (known) of the light passing through the third aperture region (designated aperture region 135C) is Iλ3 and the interference cancellation matrix (matrix composed of interference cancellation parameters) is "interference cancellation matrix A," then the following (Equation 3) holds.

[0059]

number

[0060] Left-hand side matrix (z0,z45,z135,z90) T The components of the matrix (0, 0, Iλ3) on the right side are the sensor intensities of the pixels at polarization angles of 0°, 45°, 135°, and 90° of the image sensor 210 (first image signals, which are multiple image signals for multiple lights). T The components of are the intensities of light in wavelength bands λ1, λ2, and λ3 that have passed through filter unit 134. (Equation 3) means that "if interference is removed from the sensor output intensity when only the third aperture area is open, the intensity of light in wavelength band λ3 should be Iλ3, and the intensities of light in the other wavelength bands λ1 and λ2 should be 0." Note that in the following, the matrix (z0, z45, z135, z90) on the left side T is sometimes referred to as "matrix Z".

[0061] The parameter acquisition unit 236 (processor) acquires a second interference removal parameter used to remove interference from the second image signals (multiple image signals in the second shooting (main shooting)) based on the above-mentioned (Equation 1) to (Equation 3) (by referring to the first image signals (information indicating the wavelength characteristics of the subject) acquired for the first to third opening areas) (second parameter acquisition step, second parameter acquisition process).

[0062] Combining (Equation 1) to (Equation 3), we can write it as (Equation 4) below.

[0063]

number

[0064] The second matrix on the left side (the matrix composed of matrices X, Y, and Z) is called "Matrix B," and its pseudo-inverse matrix is ​​called "Matrix B" -1 ", the parameter acquisition unit 236 acquires the second interference cancellation parameter (interference cancellation matrix A) by the following (Equation 5) using the pseudo-inverse matrix formula (second parameter acquisition step, second parameter acquisition process).

[0065]

number

[0066] [Numerical example (part 1) in the first embodiment] [About the first opening area] As shown in part (a) of FIG. 11, when a white paper (subject with known wavelength characteristics) is pre-photographed (first photographed) with the light-shielding member 131A shielding all but the first aperture region (referred to as aperture region 135A), the output of the image sensor 210 is expressed as a matrix (x0, x45, x135, x90). T =(1,0.5,0.5,0) TAssuming that, the following (Equation 6) holds for the first aperture region. Note that the light source 99 is white paper, and the "intensity (known) of light that passed through the first aperture region = 1" (the intensities for all wavelength bands λ1 to λ3 are 1).

[0067]

number

[0068] Equation 6 means that "if interference is removed from the sensor output intensity when only the first aperture area is open, the light intensity of wavelength band λ1 should be 1, and the light intensities of the other wavelength bands λ2 and λ3 should be 0."

[0069] [About the second opening area] As with the first aperture area, as shown in part (b) of FIG. 11, when a white paper (subject with known wavelength characteristics) is photographed preliminarily (first photographing) with the light-shielding member 131B shielding the second aperture area (referred to as aperture area 135B) as shown in part (b) of FIG. 11, the output of the image sensor 210 is expressed as a matrix (y0, y45, y135, y90). T =(0.5,1.0,0.0,0.5) T If so, the following (Equation 7) holds for the second aperture region. Note that the light source 99 is white paper, and the "intensity (known) of the light that passed through the second aperture region = 1" (the intensities for the wavelength bands λ1 to λ3 are all 1).

[0070]

number

[0071] Equation 7 means that "if interference is removed from the sensor output intensity when only the second aperture area is open, the light intensity of wavelength band λ2 should be 1, and the light intensities of the other wavelength bands λ1 and λ3 should be 0."

[0072] [About the third opening area] As with the first and second aperture areas, as shown in part (c) of FIG. 11, when all areas except the third aperture area (referred to as aperture area 135C) are shielded by light blocking member 131C and a white paper (subject with known wavelength characteristics) is pre-photographed (first photograph), the output of image sensor 210 is expressed as a matrix (z0, z45, z135, z90). T =(0.0,0.5,0.5,1.0) T If so, the following (Equation 8) holds for the third aperture region. Note that the light source 99 is white paper, and the "intensity (known) of the light that passed through the third aperture region = 1" (the intensities for all wavelength bands λ1 to λ3 are 1).

[0073]

number

[0074] Equation 8 means that "if interference is removed from the sensor output intensity when only the third aperture area is open, the light intensity of wavelength band λ3 should be 1, and the light intensities of the other wavelength bands λ1 and λ2 should be 0."

[0075] Combining the above (Equation 6) to (Equation 8), the following (Equation 9) holds.

[0076]

number

[0077] The matrix on the right side of the left side of (Equation 9) is called "Matrix B", and its pseudo-inverse matrix is ​​called "Matrix B -1 ", the parameter acquisition unit 236 acquires the second interference cancellation parameter (interference cancellation matrix A) by the following (Equation 10) (second parameter acquisition step, second parameter acquisition process).

[0078]

number

[0079] [Verification of the results for the numerical example (part 1)] In the actual shooting (second shooting), when the intensity of light passing through the first to third aperture regions is (1, 2, 3), the output of the image sensor 210 in the interference state (before interference removal) is (2, 4, 2, 4) for the pixels with polarization angles of 0°, 45°, 135°, and 90°. This output is expressed as the matrix (1, 2, 3) T can be obtained by multiplying a matrix formed by the outputs of the image sensor 210 in three preliminary captures (first captures) as in the following (Equation 11).

[0080]

number

[0081] When removing this interference, the interference removal unit 238 (processor) performs interference removal according to the following (Equation 12) (interference removal step, interference removal processing).

[0082]

number

[0083] As is clear from (Equation 12), the above-mentioned processes (first and second parameter acquisition step, first and second parameter acquisition process, information acquisition step, information acquisition process, interference removal step, interference removal process) have correctly removed interference (the intensities of light passing through the first to third aperture regions) to (1, 2, 3). That is, the difference between the information acquired by the first shooting (first image signal, information indicating the wavelength characteristics of the subject) and the second image signal from which interference has been removed using the second interference removal parameter is smaller than the difference between the information acquired by the first shooting and the second image signal from which interference has been removed using the first interference removal parameter. Specifically, in (Equation 6) to (Equation 8), the interference removal matrix is ​​determined so as to reduce the difference between the result (image signal after interference removal) of multiplying the interference removal matrix A on the left side by the output of image sensor 210 and the information of the subject on the right side of the equation (acquired information) (in the above example, so that the difference becomes zero).

[0084] [Image signal output] The display control unit 240 (processor) can display an image (plurality of image signals) corresponding to the image signal after interference removal (in the above example, image signals (1, 2, 3)) on the display device 300 (output device). Also, the recording control unit 242 (processor) can store the image (plurality of image signals) corresponding to the image signal after interference removal in the storage device 310 (output device).

[0085] [Numerical example (part 2) in the first embodiment] [About the first opening area] As shown in part (a) of FIG. 11, when a white paper (a subject with known wavelength characteristics) is photographed with the light-shielding member 131A shielding all but the first aperture region (referred to as aperture region 135A), the output of the image sensor 210 is expressed as a matrix (x0, x45, x135, x90). T =(0.8,0.4,0.4,0.2) T Assuming that, the following (Equation 13) holds for the first aperture region. Note that the light source 99 is white paper, and the "intensity (known) of light that passed through the first aperture region = 1" (the intensities for the wavelength bands λ1 to λ3 are all 1).

[0086]

number

[0087] Equation 13 means that "if interference is removed from the sensor output intensity when only the first aperture area is open, the light intensity of wavelength band λ1 should be 1, and the light intensities of the other wavelength bands λ2 and λ3 should be 0."

[0088] [About the second opening area] As with the first aperture area, as shown in part (b) of FIG. 11, when a white paper (subject with known wavelength characteristics) is photographed with the light-shielding member 131B shielding the second aperture area (referred to as aperture area 135B) and the output of the image sensor 210 is expressed as a matrix (y0, y45, y135, y90). T =(0.3,0.9,0.1,0.3) TIf so, the following (Equation 14) holds for the second aperture region. Note that the light source 99 is white paper, and the "intensity (known) of the light that passed through the second aperture region = 1" (the intensities for the wavelength bands λ1 to λ3 are all 1).

[0089]

number

[0090] Equation 14 means that "if interference is removed from the sensor output intensity when only the second aperture area is open, the light intensity of wavelength band λ2 should be 1, and the light intensities of the other wavelength bands λ1 and λ3 should be 0."

[0091] [About the third opening area] As with the first and second aperture areas, as shown in part (c) of FIG. 11, when all areas except the third aperture area (referred to as aperture area 135C) are shielded by light-shielding member 131C and a white paper (subject with known wavelength characteristics) is photographed, the output of image sensor 210 is expressed as a matrix (z0, z45, z135, z90). T =(0.0,0.5,0.5,0.1) T If so, the following (Equation 15) holds for the third aperture region. Note that the light source 99 is white paper, and the "intensity (known) of light that has passed through the third aperture region = 1" (the intensities for all wavelength bands λ1 to λ3 are 1).

[0092]

number

[0093] Equation 15 means that "if interference is removed from the sensor output intensity when only the third aperture area is open, the light intensity of wavelength band λ3 should be 1, and the light intensities of the other wavelength bands λ1 and λ2 should be 0."

[0094] Combining the above (Equation 13) to (Equation 15), the following (Equation 16) holds.

[0095]

number

[0096] The matrix on the right side of the left side of (Equation 16) is called "Matrix B", and its pseudo-inverse matrix is ​​called "Matrix B -1 ", the parameter acquisition unit 236 acquires the interference cancellation parameter (interference cancellation matrix A) using the following (Equation 17) (parameter acquisition step, parameter acquisition process).

[0097]

number

[0098] [Verification of the results for the numerical example (part 2)] In the actual shooting (second shooting), when the intensity of light passing through the first to third aperture areas is (1, 2, 3), the output of the image sensor 210 in the interference state (before interference removal) is (1.4, 3.7, 2.1, 3.8) for the pixels with polarization angles of 0°, 45°, 135°, and 90°. This output is expressed by the matrix (1, 2, 3) T can be obtained by multiplying a matrix formed by the outputs of the image sensor in three preliminary photographs (first photographs) as shown in the following (Equation 18).

[0099]

number

[0100] When removing this interference, the interference removal unit 238 (processor) performs interference removal according to the following (Equation 19) (interference removal step, interference removal processing).

[0101]

number

[0102] As is clear from (Equation 19), the above processing correctly removes interference, with (the intensities of the light that has passed through the first to third aperture regions) being (1, 2, 3).

[0103] [Modification of the first embodiment] In the first embodiment, all but a portion of the opening region is shielded (light-shielded) using a light-blocking member that does not transmit light, but light may also be blocked using a polarizing filter (second polarizing member) whose polarization direction (polarization angle) is orthogonal to that of the polarizing filter placed in the opening region. For example, light can be blocked by placing a polarizing filter with a polarization angle of 90 degrees in the first opening region (opening region 135A, polarizing filter 139 has a polarization angle of 0 degrees). The same applies to the second and third opening regions.

[0104] [Second aspect] In the second embodiment, the light blocking member as in the first embodiment is not used, but the wavelength band of the illumination light is changed and optical blocking is performed as shown in FIG. 13. Preliminary photography is performed on a subject with known wavelength characteristics in an optically blocked state. Specifically, as shown in part (a) of FIG. 13, in the first preliminary photography, light passes through the first opening region (here, opening region 135A in which color filter 138A of the wavelength band λ1 of the transmitted light is arranged) and passes through the second and third opening regions (color filters 138A of the wavelength band λ2). B The opening area 135 in which B The color filter 138C is irradiated with light of a wavelength band that is not transmitted through the opening region 135C in which the color filter 138C of the wavelength band λ3 is disposed.

[0105] In the second preliminary photograph, as shown in part (b) of Figure 13, light of a wavelength band that is transmitted through the second opening region (here, opening region 135B in which color filter 138B of wavelength band λ2 of transmitted light is arranged) but is not transmitted through the first and third opening regions (opening region 135A in which color filter 138A of wavelength band λ1 is arranged, and opening region 135C in which color filter 138C of wavelength band λ3 is arranged) is irradiated.

[0106] Similarly, in the third preliminary photograph, as shown in part (c) of Figure 13, light of a wavelength band that is transmitted through the third opening region (here, opening region 135C in which color filter 138C of wavelength band λ3 of transmitted light is arranged) but is not transmitted through the first and second opening regions (opening region 135A in which color filter 138A of wavelength band λ1 is arranged, and opening region 135B in which color filter 138B of wavelength band λ2 is arranged) is irradiated.

[0107] These preliminary shots In the shadows The illumination can be performed using a laser light source or an LED (Light-Emitting Diode) light source that emits monochromatic light such as red, green, or blue. As shown in Fig. 14, a diffuser 99A may be placed in front of the light source 99 (between the light source 99 and the lens device 100) to uniformly disperse the light from the light source. Alternatively, a device such as a monochromator or monochromator may be used to extract light of a desired wavelength band using an optical element such as a prism or diffraction grating from light containing multiple wavelength bands, and the extracted light may be used for illumination in the preliminary photography.

[0108] In the second aspect, the imaging control unit 232, the image acquisition unit 234, and the parameter acquisition unit 236 (processor) can acquire interference removal parameters based on information indicating the wavelength characteristics of the subject (wavelength intensity of the subject) acquired in preliminary shooting, in the same manner as described above for the first aspect (information acquisition step, information acquisition processing, second parameter acquisition step, second parameter acquisition processing).

[0109] [Third aspect] In the first and second embodiments described above, a polarizing filter is disposed in the aperture region, and an image sensor 210 including a polarizing filter element array layer 213 is used. In contrast, the third embodiment is implemented using a color sensor without a polarizing filter. FIG. 15 is a diagram showing the preliminary image capture (first image capture) in the third embodiment. As shown in the figure, a color filter is disposed in the filter unit 134A, but a polarizing filter is not disposed. Furthermore, the image sensor 210A does not have a polarizing filter element array layer, but instead includes color filters 212A to 212C having transmission wavelength bands corresponding to the transmission wavelength bands of the color filters 138A to 138C (see FIG. 6) of the filter unit 134A. These color filters 212A to 212C constitute the color filter array layer 212.

[0110] [Preliminary shooting in the third mode] 15(a), the image acquisition unit 234 (processor) acquires image signals for light in wavelength bands λ1 to λ3 (plurality of image signals for plural light beams) as "information indicating the wavelength characteristics of the subject" (information acquisition step, information acquisition process) in a state where the light blocking member 140A blocks the opening regions 135A and 135B and the opening region 135C is open, i.e., in a state where only light in wavelength band λ3 passes through the filter unit 134. Also, the image acquisition unit 234 acquires image signals for light in wavelength bands λ1 to λ3 (plurality of image signals for plural light beams) as "information indicating the wavelength characteristics of the subject" (information acquisition step, information acquisition process) in a state where the light blocking member 140B blocks the opening regions 135A and 135C and the opening region 135B is open, i.e., in a state where only light in wavelength band λ2 passes through the filter unit 134. Furthermore, as shown in part (c) of Figure 15, in a state where the light-shielding member 140C shields the opening areas 135B and 135C and the opening area 135A is left open, i.e., in a state where only light in the wavelength band λ1 passes through the filter unit 134, the image acquisition unit 234 acquires image signals for light in the wavelength bands λ1 to λ3 (multiple image signals for multiple light beams) as "information indicating the wavelength characteristics of the subject" (information acquisition step, information acquisition process).

[0111] The parameter acquisition unit 236 (processor) can acquire interference removal parameters in the same manner as described above for the first and second aspects (parameter acquisition step, parameter acquisition processing), and the interference removal unit 238 (processor) can perform interference removal using the acquired interference removal parameters (interference removal processing, interference removal step).

[0112] 16 is a diagram showing the state of actual photography (second photography) in the third mode. As shown in the drawing, actual photography is performed without using the light blocking members 140A to 140C.

[0113] [Fourth aspect] In the fourth mode, color filters 138A-138C arranged in opening regions 135A-135C and color filters of transmitted wavelength bands (wavelength band selection filters) are arranged closer to the subject (light source) than filter unit 134, and preliminary photography (first photography) is performed on a subject with known wavelength characteristics. Parts (a)-(c) of Figure 17 show how color filters 142A-142C, whose transmitted light wavelength bands are λ3, λ2, and λ1, respectively, are arranged closer to the subject than filter unit 134.

[0114] When arranging a color filter "closer to the subject than filter unit 134," instead of directly attaching the color filter to filter unit 134 as in the example shown in Fig. 17, color filters 142A to 142C may be separately attached to frame 132 separate from frame 135 and inserted closer to the subject than filter unit 134, as shown in Fig. 18. Also, as shown in Fig. 19, color filter 144 may be attached closer to the subject than first lens 110 (in this figure, the part of lens device 100 closest to the subject), and then replaced.

[0115] [Fifth aspect] In the fifth mode, with the color filters 138A to 138C not attached to the frame 135 or with the filter unit 134 not inserted in the slit 108, a color filter (wavelength selection filter) having the same transmission wavelength band as the color filters 138A to 138C is arranged closer to the subject than the frame 135. That is, the color filter is arranged closer to the subject than the color filters 138A to 138C would be if the color filters 138A to 138C were attached to the frame 135. Alternatively, the color filter is arranged closer to the subject than the frame 135 would be if the filter unit 134 (frame 135) were inserted in the slit 108. The image acquisition unit 234 (processor) acquires an image signal (first information) about a subject whose wavelength characteristics are unknown by performing preliminary imaging (first imaging) in this state (information acquisition step, information acquisition process). Parts (a) to (c) of Figure 20 show the state in which color filters 146A to 146C, whose transmitted light wavelength bands are λ3, λ2, and λ1, respectively, are arranged closer to the subject than the frame body 135, and the image acquisition unit 234 acquires the first image signal in the state shown in each part of the figure.

[0116] Furthermore, with the color filters 138A to 138C attached to the frame body 135 or with the filter unit 134 inserted into the slit 108, the image acquisition unit 234 acquires an image signal (second information) of a subject whose wavelength characteristics are unknown, in the same manner as described above with reference to Figures 11 and 17 (information acquisition process, information acquisition processing).

[0117] The parameter acquisition unit 236 (processor) acquires a parameter for correcting the second information as an interference removal parameter based on the first information (parameter acquisition step, parameter acquisition processing). The interference removal unit 238 (processor) can perform interference removal using the acquired interference removal parameter (interference removal step, interference removal processing). According to the fifth aspect, it is possible to acquire a highly accurate interference removal parameter even for a subject whose wavelength characteristics are unknown.

[0118] Although the embodiments and modifications of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the spirit of the present invention. [Explanation of symbols]

[0119] 10. Imaging System 99 light source 99A Diffuser 100 Lens device 102 Lens barrel 104 First Lever 106 Second Lever 108 Slit 110 First lens 120 Second lens 131 Light blocking material 131A Light blocking member 131B Light-shielding member 131C Light blocking material 132 Frame 133 Frame 133A Opening area 133B Opening area 133C Opening area 134 Filter unit 134A Filter Unit 135 Frame 135A opening area 135B Opening area 135C opening area 135D opening area 135E Light blocking material 135G Center of gravity 137 Filter Sets 137A Filter Set 137B Filter Set 137C Filter Set 137D Filter Set 138 Image Sensor 138A Color Filter 138B Color Filter 138C Color Filter 138D Color Filter 139 Polarizing Filter 139A Polarizing Filter 139B Polarizing Filter 139C Polarizing Filter 139D Polarizing Filter 140A Light blocking material 140B Light blocking member 140C Light blocking material 142A Color Filter 142B Color Filter 142C Color Filter 144 Color Filter 146A Color Filter 146B Color Filter 146C Color Filter 200 Imaging device body 210 Image Sensor 210A image sensor 211 Pixel Array Layer 211A Photodiode 212 Color filter array layer 212A Color Filter 212B Color Filter 212C Color Filter 213 Polarization filter element array layer 214A Polarizing filter element 214B Polarization filter element 214C Polarizing filter element 214D Polarization filter element 215 Microlens Array Layer 216 Microlens 230 processors 232 Imaging control unit 234 Image Acquisition Unit 236 Parameter Acquisition Unit 238 Interference removal unit 240 Display control unit 242 Recording control section 244 flash memory 246 RAM 300 display device 310 Storage device 320 Operation Device L optical axis X matrix Y matrix Z matrix λ1 wavelength band λ2 wavelength band λ3 wavelength band

Claims

1. 1. An information processing method for acquiring interference removal parameters for a split-pupil multispectral camera, executed by an information processing device including a processor, comprising: The split-pupil multispectral camera includes: a plurality of aperture regions arranged at or near the pupil; a plurality of optical filters arranged in the plurality of aperture regions, each of which transmits a plurality of light beams having at least some different wavelength bands; an image sensor that outputs a plurality of image signals corresponding to the plurality of lights; Equipped with The processor: a first parameter acquisition step of acquiring a first interference removal parameter used for removing interference from the plurality of image signals; an information acquiring step of acquiring, by a first photographing, first image signals which are the plurality of image signals for the plurality of light beams as information indicating wavelength characteristics of the subject; a second parameter acquisition step of acquiring second interference removal parameters used for removing interference from second image signals, which are a plurality of image signals obtained by the second imaging, by referring to the information acquired by the first imaging; Run the processor, in the second parameter acquisition step, acquires the second interference removal parameter that makes a difference between the information acquired by the first imaging and the second image signal from which interference has been removed using the second interference removal parameter smaller than a difference between the information acquired by the first imaging and the second image signal from which interference has been removed using the first interference removal parameter; The processor: In the information acquisition step, acquiring, as first information, the information about the subject whose wavelength characteristics are unknown, by photographing in a state in which the plurality of optical filters are not arranged in the plurality of opening regions and an optical filter that transmits one of the plurality of light beams and does not transmit the remaining light beams is arranged closer to the subject than the plurality of optical filters; acquiring, as second information, the information about the subject having unknown wavelength characteristics by photographing in a state in which the plurality of optical filters are arranged in the plurality of opening regions and an optical filter that transmits one of the plurality of light beams and does not transmit the remaining light beams is arranged closer to the subject than the plurality of optical filters; In the second parameter acquisition step, An information processing method for acquiring, as the second interference removal parameter, a parameter for correcting the second information based on the first information.

2. The first photographing is a preliminary photographing, and the second photographing is a main photographing, The information processing method according to claim 1 , wherein the focus position of the first photographing is the same as the focus position of the second photographing.

3. In the information acquisition step, the processor The information processing method according to claim 1 or 2, further comprising acquiring wavelength intensities of the subject in a state where light is not transmitted through some of the plurality of opening regions.

4. In the information acquisition step, the processor 3. The information processing method according to claim 1, wherein the information is obtained by capturing an image of a subject having known wavelength characteristics while blocking some of the plurality of opening areas and leaving the remaining opening areas open.

5. In the information acquisition step, the processor The information processing method according to claim 4 , wherein the first image signal as the information is acquired in a state where the partial opening region is physically blocked by arranging a light-blocking member that does not transmit light in the partial opening region.

6. the split-pupil multispectral camera includes a plurality of first polarizing members arranged in the plurality of aperture regions and transmitting light having different polarization angles; In the information acquisition step, the processor The information processing method of claim 4, wherein the first image signal is acquired as the information by optically blocking the partial opening region by arranging a second polarizing element having a polarization angle perpendicular to that of the first polarizing element arranged in the partial opening region.

7. In the information acquisition step, the processor The information processing method according to claim 4, wherein the first image signal is obtained by photographing a subject having known wavelength characteristics while an optical filter that transmits one of the plurality of light beams and does not transmit the remaining light beams is positioned closer to the subject than the plurality of optical filters.

8. The split-pupil multispectral camera comprises: a plurality of first polarizing members arranged in the plurality of opening regions and transmitting light beams with different polarization angles; a plurality of second polarizing members disposed on the image sensor, each of which transmits light having a polarization angle corresponding to the polarization angles of the plurality of first polarizing members; Equipped with In the information acquisition step, the processor The information processing method according to claim 1 or 2, wherein a plurality of image signals corresponding to the polarization angles of the plurality of first polarizing members are obtained as the information.

9. A plurality of aperture regions arranged at or near the pupil; a plurality of optical filters arranged in the plurality of aperture regions, each of which transmits a plurality of light beams having at least some different wavelength bands; an image sensor that outputs a plurality of image signals corresponding to the plurality of lights; An information processing device for acquiring interference removal parameters for a split-pupil multispectral camera, comprising: a processor; The processor: a first parameter acquisition process for acquiring a first interference removal parameter used for removing interference from the plurality of image signals; an information acquisition process of acquiring the plurality of image signals for the plurality of light beams as information indicating wavelength characteristics of the subject by first capturing the image; a second parameter acquisition process for acquiring second interference removal parameters used for interference removal of second image signals, which are a plurality of image signals obtained by the second image capture, by referring to the information acquired by the first image capture; Run the processor, in the second parameter acquisition process, acquires the second interference removal parameter that makes a difference between the information acquired by the first imaging and the second image signal from which interference has been removed using the second interference removal parameter smaller than a difference between the information acquired by the first imaging and the second image signal from which interference has been removed using the first interference removal parameter; The processor: In the information acquisition process, acquiring, as first information, the information about the subject whose wavelength characteristics are unknown, by photographing in a state in which the plurality of optical filters are not arranged in the plurality of opening regions and an optical filter that transmits one of the plurality of light beams and does not transmit the remaining light beams is arranged closer to the subject than the plurality of optical filters; acquiring, as second information, the information about the subject having unknown wavelength characteristics by photographing in a state in which the plurality of optical filters are arranged in the plurality of opening regions and an optical filter that transmits one of the plurality of light beams and does not transmit the remaining light beams is arranged closer to the subject than the plurality of optical filters; In the second parameter acquisition process, an information processing device that acquires, based on the first information, a parameter for correcting the second information as the second interference removal parameter;

10. An information processing device having a processor, a plurality of aperture regions arranged at or near the pupil; a plurality of optical filters arranged in the plurality of aperture regions, each of which transmits a plurality of light beams having at least some different wavelength bands; an image sensor that outputs a plurality of image signals corresponding to the plurality of lights; An information processing program for executing an information processing method for acquiring interference removal parameters for a split-pupil multispectral camera, comprising: the processor, a first parameter acquisition step of acquiring a first interference removal parameter used for removing interference from the plurality of image signals; an information acquiring step of acquiring the plurality of image signals for the plurality of light beams as information indicating wavelength characteristics of the subject by first capturing the image; a second parameter acquisition step of acquiring second interference removal parameters used for removing interference from second image signals, which are a plurality of image signals obtained by the second imaging, by referring to the information acquired by the first imaging; Execute in the second parameter acquisition step, causing the processor to acquire the second interference elimination parameter that makes a difference between the information acquired by the first imaging and the second image signal from which interference has been eliminated using the second interference elimination parameter smaller than a difference between the information acquired by the first imaging and the second image signal from which interference has been eliminated using the first interference elimination parameter; the processor, In the information acquisition step, In a state where the plurality of optical filters are not arranged in the plurality of opening regions, an optical filter that transmits one of the plurality of light beams and does not transmit the remaining light beams is arranged closer to the subject than the plurality of optical filters, thereby acquiring the information of the subject whose wavelength characteristics are unknown as first information; In a state where the plurality of optical filters are arranged in the plurality of opening regions, an optical filter that transmits one of the plurality of light beams and does not transmit the remaining light beams is arranged closer to the subject than the plurality of optical filters, thereby acquiring the information of the subject whose wavelength characteristics are unknown as second information; In the second parameter acquisition step, an information processing program for acquiring, as the second interference removal parameter, a parameter for correcting the second information based on the first information;

11. A non-transitory computer-readable recording medium on which the program described in claim 10 is recorded.

12. A plurality of aperture regions arranged at or near the pupil; a plurality of optical filters arranged in the plurality of aperture regions, each of which transmits a plurality of light beams having at least some different wavelength bands; an image sensor that outputs a plurality of image signals corresponding to the plurality of lights; a split-pupil multispectral camera comprising: The information processing device according to claim 9 ; An information processing system comprising:

13. The processor, performing interference removal on the second image signal using the interference removal parameter; The information processing system according to claim 12, wherein the plurality of image signals in a state where the interference has been removed are output to an output device.

Images