Information processing device, method and program and multi-spectrum camera

Abstract

To provide an information processing device, method and program for satisfactorily removing interference components included in a plurality of spectrum images outputted from an image sensor of a multi-spectrum camera, and a multi-spectrum camera.SOLUTION: A processor of an information processing device 2 acquires a first spectrum image and a second spectrum image outputted from an image sensor 30 of a pupil division type multi-spectral camera 1, performs first interference reduction processing for reducing interference between respective spectrum images to the first spectrum image and the second spectrum image by a calculation using a first crosstalk constant, and determines whether an influence of the interference by the second spectrum image exceeds a threshold to the first spectrum image undergoing the first interference reduction processing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, method, program, and multispectral camera, and more particularly to a technique for reducing interference components contained in a plurality of spectral images output from an image sensor of a split-pupil multispectral camera. [Background technology]

[0002] Conventional split-pupil multispectral cameras of this type are known from Patent Documents 1 and 2.

[0003] The split-pupil multispectral camera (imaging device) described in Patent Documents 1 and 2 is equipped with a filter unit that is positioned at or near the pupil position of the photographing lens and has multiple transmitted light regions with different transmitted wavelength bands and polarization characteristics (wavelength selectivity), and an image sensor in which multiple polarizing filters with different polarization characteristics are arranged for each pixel, and captures images of multiple wavelength bands (multiple spectral images) from multiple pixel groups with different polarization characteristics of the image sensor.

[0004] In addition, in order to reduce the effects of crosstalk (interference) between the spectral images output from the image sensors, the multispectral camera calculates a crosstalk constant (interference ratio) in advance and performs interference removal processing based on the interference ratio. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-13504 [Patent Document 2] Patent No. 7285322 Summary of the Invention [Problem to be solved by the invention]

[0006] One embodiment of the technique disclosed herein provides an information processing device, a method, a program, and a multispectral camera that effectively remove interference components contained in a plurality of spectral images output from an image sensor of a multispectral camera. [Means for solving the problem]

[0007] A first aspect of the invention is an information processing device including a processor, wherein the processor acquires a first spectral image and one or more second spectral images corresponding to a first wavelength band and one or more second wavelength bands, respectively, which are spectral images for each of a plurality of wavelength bands output from an image sensor of a multispectral camera when a first image of a subject is captured by the multispectral camera, performs a first interference reduction process for reducing interference on the first spectral image and the second spectral image by calculation using a first crosstalk constant, and determines whether or not an influence of interference due to the second spectral image exceeds a threshold for the first spectral image that has been subjected to the first interference reduction process.

[0008] In the information processing device according to the second aspect of the present invention, in the first aspect, a first spectral image and a second spectral image output from an image sensor are denoted as yλ1 and yλ2, respectively, a theoretical first spectral image without interference and a theoretical second spectral image without interference are denoted as xλ1 and xλ2, respectively, a ratio of interference between the first spectral image (xλ1) and the second spectral image (xλ2) is denoted as α, and a ratio of interference between the second spectral image (xλ2) and the first spectral image (xλ1) is denoted as β, the first spectral image (yλ1) and the second spectral image (yλ2) are expressed by the following formulas (1) and (2): yλ1=xλ1-αxλ1+βxλ2……(1) yλ2=xλ2+αxλ1-βxλ1……(2) The first interference reduction process is a process of calculating a theoretical first spectral image (xλ1) without interference and a theoretical second spectral image (xλ2) without interference using equations (1) and (2).

[0009] In the information processing device according to the third aspect of the present invention, in the first or second aspect, the processor preferably calculates a first representative value of an image signal indicating a first spectral image subjected to the first interference reduction processing, and a second representative value of an image signal indicating a second spectral image subjected to the first interference reduction processing, and determines whether or not the influence of interference exceeds a threshold based on whether or not a difference between the first representative value and the second representative value exceeds a first threshold.

[0010] In the information processing device according to the fourth aspect of the present invention, in the first or second aspect, the processor preferably calculates a first representative value of an image signal indicating the first spectral image after the first interference reduction processing and a third representative value of an image signal indicating the first spectral image before the first interference reduction processing, and determines whether or not the influence of interference exceeds a threshold based on whether or not a difference between the first representative value and the third representative value exceeds a second threshold.

[0011] In the information processing device according to a fifth aspect of the present invention, in any of the first to fourth aspects, it is preferable that, in response to a determination that the influence of interference exceeds a threshold, the processor notifies a second image of the subject by a multispectral camera, with a filter that reduces the transmittance of the first wavelength band interposed therebetween.

[0012] In the information processing device according to a sixth aspect of the present invention, in any of the first to fourth aspects, it is preferable that, in response to a determination that the influence of interference exceeds a threshold, the processor performs a second image of the subject using a multispectral camera, with a filter that reduces the transmittance of the first wavelength band interposed therebetween.

[0013] In the information processing device according to a seventh aspect of the present invention, in the fifth or sixth aspect, when a first image capture and a second image capture are performed on the same subject, it is preferable that the processor acquires two sets of first spectral images output from the image sensor corresponding to the first image capture and the second image capture, and performs a second interference reduction process to reduce interference with the first spectral image by calculating a difference between the two sets of first spectral images.

[0014] An information processing device according to an eighth aspect of the present invention is, in the seventh aspect, a pair of first spectral images output from the image sensor corresponding to the first and second imaging are denoted by yλ1 and y′λ1, respectively, a theoretical first spectral image without interference and a theoretical second spectral image without interference are denoted by xλ1 and xλ2, respectively, a ratio of interference between the first spectral image (xλ1) and the second spectral image (xλ2) is denoted by α, a ratio of interference between the second spectral image (xλ2) and the first spectral image (xλ1) is denoted by β, and a transmittance of the first wavelength band by the filter is denoted by k, the first spectral images (yλ1) and (y′λ1) are expressed by the following formulas (3) and (4): yλ1= xλ1- αxλ1+βxλ2……(3) y'λ1=kxλ1-kαxλ1+βxλ2...(4) The second interference reduction process is a process of subtracting equation (4) from equation (3) to calculate the first spectral image (xλ1).

[0015] In the information processing device according to a ninth aspect of the present invention, in the seventh or eighth aspect, it is preferable that the processor performs interference reduction for the second spectral image by a first interference reduction process.

[0016] In an information processing device according to a tenth aspect of the present invention, in any of the fifth to eighth aspects, it is preferable that the processor acquires the second spectral image output from the image sensor when the second imaging is performed as an image in which interference due to the second spectral image has been reduced.

[0017] In the information processing device according to the eleventh aspect of the present invention, or the fifth or sixth aspect, it is preferable that, when a first image capture and a second image capture are performed on the same subject, the processor acquires two sets of a first spectral image and a second spectral image output from the image sensor corresponding to the first image capture and the second image capture, calculates a second crosstalk constant based on the two sets of the first spectral image and the second spectral image, and performs a third interference reduction process to reduce interference on the first spectral image and the second spectral image by calculation using the second crosstalk constant.

[0018] In an information processing device according to a twelfth aspect of the present invention, in any of the first to eleventh aspects, it is preferable that the processor sets a spectral image set by a user among the multiple spectral images output from the image sensor as a first spectral image, and sets the other spectral images as one or more second spectral images.

[0019] In an information processing device according to a thirteenth aspect of the present invention, in any of the first to eleventh aspects, it is preferable that the processor sets, among the multiple spectral images output from the image sensor, a spectral image having the smallest output signal as a first spectral image, and sets the other spectral images as one or more second spectral images.

[0020] A fourteenth aspect of the invention is a multispectral camera including the information processing device of any one of the first to eleventh aspects.

[0021] A fifteenth aspect of the invention is an information processing method for processing spectral images for multiple wavelength bands output from an image sensor of a split-pupil multispectral camera when a first image of a subject is captured by the multispectral camera, the information processing method including the steps of: a processor acquiring the multiple spectral images output from the image sensor, the first spectral image and one or more second spectral images corresponding to a first wavelength band and one or more second wavelength bands, respectively; a first interference reduction process performed by the processor to reduce interference on the first spectral image and the second spectral image by calculation using a first crosstalk constant; and a processor determining whether or not an influence of interference due to the second spectral image exceeds a threshold for the first spectral image that has been subjected to the first interference reduction process.

[0022] An information processing method according to a 16th aspect of the present invention, in the 15th aspect, preferably includes the steps of: a step in which the processor determines a first representative value of an image signal indicating a first spectral image subjected to a first interference reduction process, and a second representative value of an image signal indicating a second spectral image subjected to the first interference reduction process; and a step in which the processor determines whether or not the influence of interference exceeds a threshold based on whether or not the difference between the first threshold and the second representative value exceeds the first threshold.

[0023] An information processing method according to a 17th aspect of the present invention, in the 15th aspect, preferably includes the steps of: a step in which the processor determines a first representative value of an image signal indicating a first spectral image after the first interference reduction processing, and a third representative value of an image signal indicating the first spectral image before the first interference reduction processing; and a step in which the processor determines whether or not an influence of interference exceeds a threshold based on whether or not a difference between the first representative value and the third representative value exceeds a second threshold.

[0024] An information processing method according to an 18th aspect of the present invention, in any of the 15th to 17th aspects, preferably includes a step of notifying or causing a second image of the subject to be captured by a multispectral camera with a filter that reduces the transmittance of the first wavelength band interposed when the processor determines that the influence of interference exceeds a threshold.

[0025] An information processing method according to a 19th aspect of the present invention preferably includes, in the 18th aspect, the steps of: when the first imaging and the second imaging are performed, the processor acquires two sets of first spectral images output from the image sensor corresponding to the first imaging and the second imaging; and, when the first imaging and the second imaging are performed, the processor performs a second interference reduction process to reduce interference with the first spectral image by calculating a difference between the two sets of first spectral images.

[0026] An information processing method according to a 20th aspect of the present invention, in the 18th aspect, preferably includes the steps of: acquiring, when a first image pickup and a second image pickup are performed on the same object, two sets of first and second spectral images output from the image sensor corresponding to the first and second image pickup; calculating, by the processor, a second crosstalk constant based on the two sets of the first and second spectral images; and performing, by the processor, a third interference reduction process for reducing interference on the first and second spectral images by calculation using the second crosstalk constant.

[0027] An invention according to a 21st aspect is an information processing program that causes a computer to realize a function of acquiring spectral images for a plurality of wavelength bands output from an image sensor of a split-pupil multispectral camera when a first image of a subject is captured by the multispectral camera, a first spectral image and one or more second spectral images corresponding to a first wavelength band and one or more second wavelength bands, respectively; a function of performing a first interference reduction process to reduce interference on the first spectral image and the second spectral image by calculation using a first crosstalk constant; and a function of determining whether or not an influence of interference by the second spectral image exceeds a threshold for the first spectral image that has been subjected to the first interference reduction process. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram showing the system configuration of an imaging system including an information processing device according to the present invention. [Diagram 2] FIG. 2 is a perspective view showing a filter unit disposed in the imaging optical system of the multispectral camera. [Diagram 3] FIG. 3 is a perspective view showing a schematic configuration of an image sensor of a multispectral camera. [Figure 4] FIG. 4 is a plan view of an image sensor of a multispectral camera, and in particular shows a micro-polarizer disposed at each pixel of the image sensor. [Diagram 5]FIG. 5 is a perspective view showing the external appearance of a main part of the multispectral camera. [Figure 6] FIG. 6 is an enlarged perspective view of the filter unit shown in FIG. [Figure 7] 7 is a plan view of a filter set attached to the frame of the filter unit shown in FIG. [Figure 8] FIG. 8 is a block diagram showing an embodiment of a hardware configuration of an information processing device according to the present invention. [Figure 9] Figures 9(A) and (B) show two spectral images without crosstalk captured separately by a multispectral camera of a surface light source with a low light intensity of wavelength λ1 (720 nm) and a surface light source with a high light intensity of wavelength λ2 (615 nm). [Figure 10] Figures 10(A) and (B) show two spectral images with crosstalk captured simultaneously by a multispectral camera of a surface light source with a low light intensity of wavelength λ1 (720 nm) and a surface light source with a high light intensity of wavelength λ2 (615 nm). [Figure 11] 11(A) and (B) are images showing the effect of crosstalk, obtained by dividing the spectral images with crosstalk shown in FIGS. 10(A) and (B), respectively, by the spectral images without crosstalk shown in FIGS. 10(A) and (B), respectively. [Figure 12] FIG. 12 is a conceptual diagram showing a second interference reduction processing method for a first spectral image that is significantly affected by crosstalk. [Figure 13] FIG. 13 shows a spectral image generated to confirm the actual effect of the second interference reduction processing. [Figure 14] FIG. 14 is an image showing the influence of crosstalk, obtained by dividing the first spectral image after the second interference reduction processing shown in FIG. 13(B) by the ideal first spectral image without interference shown in FIG. 13(A). [Figure 15] FIG. 15 is a flowchart showing a first embodiment of an information processing method according to the present invention. [Figure 16]FIG. 16 is a flowchart showing a second embodiment of the information processing method according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, preferred embodiments of an information processing device, method, program, and multispectral camera according to the present invention will be described with reference to the accompanying drawings.

[0030] [Imaging system overview] FIG. 1 is a diagram showing the system configuration of an imaging system including an information processing device according to the present invention.

[0031] The imaging system shown in FIG. 1 includes a split-pupil multispectral camera 1 and an information processing device 2.

[0032] The multispectral camera 1 includes a photographing optical system 10 and an image sensor 30 .

[0033] The photographing optical system 10 of this example has a front lens group 10A and a rear lens group 10B, and a filter unit 20 is disposed between the front lens group 10A and the rear lens group 10B at or near the pupil position.

[0034] FIG. 2 is a perspective view showing a filter unit disposed in the imaging optical system of the multispectral camera.

[0035] As shown in FIG. 2, the filter unit 20 has opening areas that transmit different transmission wavelength bands. In the example shown in FIG. 2, the opening areas are three equally spaced circular opening areas, and the filter unit 20 has a bandpass filter BPF1 that transmits light of a first wavelength band, a bandpass filter BPF2 that transmits light of a second wavelength band, and a bandpass filter BPF3 that transmits light of a third wavelength band.

[0036] Furthermore, multiple (three in this example) polarizing filters PL1, PL2, PL3 with different polarization directions are arranged in the opening region of the filter unit 20. Polarizing filter PL1 absorbs the left-right component of unpolarized light in Fig. 2 and transmits the up-down component. Hereinafter, the direction of linearly polarized light (polarization direction) that transmits polarizing filter PL1 is defined as 0°.

[0037] 2, the polarization direction of the linearly polarized light transmitted through the polarizing filter PL2 is 45°, and the polarization direction of the linearly polarized light transmitted through the polarizing filter PL3 is 90°. That is, the three polarizing filters PL1, PL2, PL3 are arranged so that their polarization directions are different. Note that the polarization directions of the polarizing filters PL1, PL2, PL3 are not limited to 0°, 45°, and 90°, and may be, for example, 0°, 60°, and 120°.

[0038] Returning to FIG. 2, light emitted from the subject passes through the three aperture regions of the photographing optical system 10 and enters the image sensor 30 as subject light having different wavelength bands and different polarization directions.

[0039] FIG. 3 is a perspective view showing a schematic configuration of an image sensor of a multispectral camera.

[0040] 3 is a CMOS (Complementary Metal-Oxide Semiconductor) type image sensor, and is a monochrome imaging element having a pixel array layer 32, a polarizing filter element array layer 34, and a microlens array layer 36. The layers are arranged in the order of the pixel array layer 32, the polarizing filter element array layer 34, and the microlens array layer 36 from the image plane side to the object side. Note that the image sensor 30 is not limited to a CMOS type, and may be an XY address type or a CCD (Charge Coupled Device) type image sensor.

[0041] The pixel array layer 32 is configured by arranging a large number of photodiodes two-dimensionally. One photodiode constitutes one pixel. The photodiodes are regularly arranged along the horizontal direction (x direction) and the vertical direction (y direction).

[0042] The polarizing filter element array layer 34 is configured by two-dimensionally arranging four types of micro-polarizers with different polarization directions (polarization directions of light to be transmitted) corresponding to the respective pixels.

[0043] FIG. 4 is a plan view of an image sensor of a multispectral camera, and in particular shows a micro-polarizer disposed at each pixel of the image sensor.

[0044] As shown in Fig. 4, four micro-polarizers P1, P2, P3, and P4 with different polarization directions are arranged in each pixel of the image sensor 30. The micro-polarizer P1 transmits the vertical components of the incident light in Fig. 4. If the polarization direction of the micro-polarizer P1 is 0°, the polarization directions of the micro-polarizers P2, P3, and P4 are 45°, 90°, and 135°.

[0045] If four adjacent micro-polarizers P1 to P4 form a basic array pattern F, then in each pixel of the image sensor 30, the micro-polarizers P1 to P4 of the basic array pattern F are repeatedly arranged in the vertical and horizontal directions in FIG.

[0046] Moreover, the microlens array layer 36 shown in FIG. 3 is a microlens arrayed for each pixel.

[0047] The polarizing filters PL1, PL2, and PL3 (FIG. 2) arranged in the three aperture regions of the filter unit 20, and the micro polarizers P1, P2, P3, and P4 arranged on each pixel of the image sensor 30 function as a pupil division section that divides the subject light of different wavelength bands that passes through each of the three aperture regions of the photographing optical system 10 into pupils and causes each to enter a pixel of the image sensor 30 that corresponds to the polarization direction. That is, in this example, a pupil division section of a "polarization method" that uses the polarization of light to separate subject light of different wavelength bands is adopted, which has the advantages of making it easy to make the camera compact and being suitable for close-up photography because images (spectral images) for each wavelength band can be obtained without parallax.

[0048] However, this type of pupil division method cannot effectively separate subject light of different wavelength bands, and the signal obtained from each pixel of the image sensor 30 contains a mixture of information on multiple spectral images. This is called "interference" or "crosstalk." In addition, when interference occurs, the pupil position and aperture shape of the aperture region that transmits each wavelength band in the filter unit 20 cause problems such as image distortion and multiple edges.

[0049] In particular, in the case of the "polarized light method," when the number of subject light beams to be separated is three or more (three or more opening areas), each pixel of the image sensor 30 receives a relatively large amount of subject light beams having a polarization direction different from the polarization direction of the subject light beams having the same polarization direction as the polarization direction of the micro-polarizer provided in that pixel.

[0050] Furthermore, when there are two subject lights to be separated (two aperture areas), two polarizing filters with perpendicular polarization directions are provided in the two aperture areas, and two micro-polarizers with perpendicular polarization directions are provided in each pixel of the image sensor. However, even in this case, interference can occur due to a slight angle misalignment between the polarizing filters and the micro-polarizers.

[0051] The multispectral camera 1 in this example captures subject light in three wavelength bands by pupil-splitting the subject light that passes through three aperture regions in which three types of bandpass filters BPF1 to BPF3 are arranged. However, as described above, subject light in three wavelength bands is incident on each pixel of the image sensor 30, and therefore the spectral images for the three wavelength bands output from the image sensor 30 contain interference components.

[0052] FIG. 5 is a perspective view showing the external appearance of a main part of the multispectral camera.

[0053] As shown in FIG. 5, a slit 12 is formed in the lens barrel of the multispectral camera 1 at or near the pupil position of the imaging optical system 10 (FIG. 1), and a filter unit 20 is inserted into this slit 12.

[0054] A filter 14, which will be described later, is detachably attached to the incident light side (front lens surface) of the lens barrel of the multispectral camera 1.

[0055] An image signal indicating a plurality of spectral images output from the image sensor 30 of the multispectral camera 1 is output to an information processing device 2 (see FIG. 1).

[0056] FIG. 6 is an enlarged perspective view of the filter unit shown in FIG.

[0057] The filter unit 20 shown in FIG. 6 is configured by attaching a filter set 23 shown in FIG. 7 to a frame 22 having a plurality of openings 22A (four in this example).

[0058] As shown in FIG. 2, similar to the filter unit 20, the filter set 23 is configured by overlapping three polarizing filters PL1 to PL3 having different polarization directions and three bandpass filters BPF1 to BPF3.

[0059] Here, when a subject having wavelengths λ1, λ2, and λ3 of 720 nm, 615 nm, and 850 nm is captured using multispectral camera 1, bandpass filters BPF1 to BPF3 are prepared which transmit the first to third wavelength bands, respectively, having λ1, λ2, and λ3 as representative wavelengths (e.g., central wavelengths of the wavelength bands).

[0060] In FIG. 6, one of the four openings 22A of the frame 22 is covered with a black Mylar filter 24.

[0061] When capturing images of subjects with wavelengths λ1, λ2, and λ3 using the multispectral camera 1 configured as described above, in addition to the light of wavelength λ1 that passes through bandpass filter BPF1, light of wavelengths λ2 and λ3 that pass through bandpass filter BPF2 and bandpass filter BPF3 are also incident on the pixel where micro-polarizer P1 of image sensor 30 is located.

[0062] In particular, the light of wavelength λ2 that passes through bandpass filter BPF2 has a polarization direction of 45° with respect to the polarization direction of light of wavelength λ1 that passes through bandpass filter BPF1, which is 0°, so that a relatively large amount of light enters the pixel in which micro-polarizer P1 is arranged, whereas the light of wavelength λ3 that passes through bandpass filter BPF3 has a polarization direction of 90° with respect to the polarization direction of light of wavelength λ1 that passes through bandpass filter BPF1, which is 0°, so that a small amount of light enters the pixel in which micro-polarizer P1 is arranged. Similarly, light of three wavelengths (λ1, λ2, λ3) is incident on the pixels in which the other micro-polarizers P2 to P4 are arranged.

[0063] Therefore, in order to extract spectral images of only the wavelengths λ1, λ2, and λ3, it is necessary to perform image processing (interference reduction processing).

[0064] [Information processing device] The information processing device 2 can be configured, for example, by a PC (personal computer) as shown in FIG. 1, and acquires unprocessed imaging data (also referred to as "RAW data") immediately after it is output from the image sensor 30 of the multispectral camera 1 via wired or wireless communication, or via a recording medium (memory card), and performs information processing on the acquired RAW data, including various types of interference reduction processing described below, to generate spectral images for each wavelength (λ1, λ2, λ3).

[0065] FIG. 8 is a block diagram showing an embodiment of a hardware configuration of an information processing device according to the present invention.

[0066] As shown in FIG. 8, the information processing device 2 constituted by a PC includes a processor 100, a memory 110, a display 120, an input / output interface 130, an operation unit 140, and the like.

[0067] The processor 100 is composed of a CPU (Central Processing Unit) and the like, and controls each part of the information processing device 2 and functions as a variety of information processing parts including an interference reduction process.

[0068] The memory 110 includes a flash memory, a read-only memory (ROM), a random access memory (RAM), a hard disk drive, etc. The flash memory, the ROM, or the hard disk drive is a non-volatile memory that stores an operation system, an information processing program according to the present invention, various parameters used for information processing, etc.

[0069] The processor 100 can obtain an information processing program and various parameters from a server of the manufacturer of the multispectral camera 1 or from a recording medium attached to the multispectral camera 1 in advance by operating the operation unit 140, and store them in the memory 110. In addition, the flash memory or hard disk device stores the spectral images after various information processing such as interference reduction processing.

[0070] The RAM functions as a working area for information processing by the processor 100, and also temporarily stores information processing programs and the like stored in a flash memory etc. The processor 100 may incorporate a part of the memory 110 (RAM).

[0071] The display 120 displays a screen for operating the information processing device 2, and also displays spectral images for each wavelength (λ1, λ2, λ3) after information processing.

[0072] The input / output interface 130 includes a connection unit connectable to an external device, a communication unit connectable to a network, etc. Examples of the connection unit connectable to an external device include a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI) (HDMI is a registered trademark), and a card slot into which a memory card is inserted.

[0073] The operation unit 140 includes a pointing device such as a mouse or a touch pad, a keyboard, etc., and is a part that accepts various operational inputs from the user.

[0074] <Example of the effects of crosstalk> Figures 9(A) and (B) show spectral images of wavelengths λ1 and λ2 without crosstalk, captured separately by a multispectral camera of a surface light source with a low light intensity of wavelength λ1 (720 nm) and a surface light source with a high light intensity of wavelength λ2 (615 nm).

[0075] On the other hand, Figures 10(A) and (B) show two spectral images with crosstalk captured simultaneously by a multispectral camera of a surface light source with a low light intensity of wavelength λ1 (720 nm) and a surface light source with a high light intensity of wavelength λ2 (615 nm).

[0076] The range of luminance values ​​of the spectral images of wavelengths λ1 and λ2 shown in FIGS. 10(A) and (B) is normalized to be the same as the range of luminance values ​​of the spectral images of wavelengths λ1 and λ2 shown in FIGS. 9(A) and (B).

[0077] 11(A) and (B) are images showing the effect of crosstalk, obtained by dividing the spectral images with crosstalk shown in FIGS. 10(A) and (B), respectively, by the spectral images without crosstalk shown in FIGS. 10(A) and (B), respectively.

[0078] The image showing the effects of crosstalk shown in FIG. 11(A) shows a distribution in the range of 0.8 to 1 (20% range), whereas the image showing the effects of crosstalk shown in FIG. 11(B) shows a distribution in the range of 0.992 to 1.002 (1% range).

[0079] As shown in Figure 11, the spectral image of the wavelength λ1 with a small amount of light is significantly affected by the crosstalk from the surface light source of the wavelength λ2 with a large amount of light, while the spectral image of the wavelength λ2 with a large amount of light is less affected by the crosstalk from the surface light source of the wavelength λ1 with a small amount of light.

[0080] [Summary of the Invention] Next, an overview of the present invention will be described.

[0081] A phenomenon called crosstalk is a problem with the multispectral camera 1. In particular, if the amount of light in a spectral image (first spectral image) of a specific wavelength band (center wavelength λ1) to be measured is smaller than the amount of light of a disturbing wavelength (λ2), the amount (proportion) of crosstalk becomes larger according to the ratio of the amount of light, which directly leads to screen unevenness of the first spectral image of wavelength λ1.

[0082] In the case of such a first spectral image, even if a normal interference reduction process ("first interference reduction process" to be described later) is performed, it is not possible to effectively reduce interference.

[0083] Here, a spectral image of a first wavelength band (central wavelength λ1) which is a specific wavelength band is referred to as a first spectral image, and a spectral image of a second wavelength band (central wavelength λ2) which is another wavelength band that causes disturbance is referred to as a second spectral image.

[0084] For example, if the ratio of the light amounts of the first and second spectral images input to the multispectral camera 1 is 1:10, and theoretical first and second spectral images without crosstalk are output from the image sensor 30 of the multispectral camera 1, the ratio of the luminance values ​​of the output first and second spectral images will also be 1:10, making it possible to extract the original signal (wavelength information).

[0085] On the other hand, if the ratio of the amount of light between the first and second spectral images input to the multispectral camera 1 is 1:100 and 1% of light in the second wavelength band of the second spectral image is incident on a pixel of the image sensor 30 (a pixel in which the micro-polarizer P1 is arranged) where light in the first wavelength band of the first spectral image is incident, due to crosstalk, the ratio of the luminance values ​​of the first and second spectral images output from each pixel of the image sensor 30 (the pixel in which the micro-polarizer P1 is arranged and the pixel in which the micro-polarizer P2 is arranged) will be 2:99, and the discrepancy between the ratio of the amount of light between the first and second spectral images input to the multispectral camera and the ratio of the luminance values ​​of the output first and second spectral images will become a problem.

[0086] Therefore, the processor 100 of the information processing device 2 acquires spectral images for each of a plurality of wavelength bands output from the image sensor of the multispectral camera, and determines whether or not a first spectral image of a specific wavelength band to be measured among these spectral images is significantly affected by crosstalk from a second spectral image of another wavelength band (above a threshold).

[0087] When the processor 100 determines that the first spectral image output from the image sensor 30 of the multispectral camera 1 is not significantly affected by crosstalk, it performs a first interference reduction process and generates a first spectral image and a second spectral image after the first interference reduction process in which interference is reduced for the first spectral image and the second spectral image output from the image sensor 30.

[0088] On the other hand, when the processor 100 determines that the first spectral image output from the image sensor 30 of the multispectral camera 1 is significantly affected by crosstalk, it issues an instruction via the display 120 or other notification means to capture a second image of the same subject by the multispectral camera 1, with a filter that reduces the transmittance of the first wavelength band interposed therebetween.

[0089] Furthermore, the processor 100 may cause the multispectral camera 1 to perform a second image capture of the same subject, instead of or in addition to the notification of the second image capture.

[0090] The processor 100 can set, as the first spectral image, a spectral image set by a user using a user interface including the operation unit 140 and the display 120, among the multiple spectral images output from the image sensor 30 of the multispectral camera 1. In this case, the other spectral images among the multiple spectral images are set as one or more second spectral images.

[0091] The processor 100 determines whether the first spectral image thus set is significantly affected by crosstalk from the second spectral image.

[0092] In addition, the processor 100 may set the spectral image with the smallest output signal among the multiple spectral images output from the image sensor 30 of the multispectral camera 1 as the first spectral image, and set the other spectral images as one or more second spectral images.

[0093] Next, when the processor 100 determines that the first spectral image is significantly affected by crosstalk from the second spectral image of another wavelength band, the processor 100 causes the multispectral camera 1 to perform first and second imaging.

[0094] The processor 100 acquires at least two pairs of first spectral images out of two pairs of first and second spectral images output from the image sensor 30 corresponding to the first and second imaging, and performs a second interference reduction process different from the first interference reduction process to generate a first spectral image that is less affected by crosstalk.

[0095] FIG. 12 is a conceptual diagram showing a second interference reduction processing method for a first spectral image that is significantly affected by crosstalk.

[0096] In FIG. 12, first, a first spectral image and a second spectral image of a wavelength λ1 to be measured are obtained from the image sensor 30 by a first imaging (first imaging) of a subject by the multispectral camera 1.

[0097] Next, a first spectral image of the small light wavelength λ1 is obtained from the image sensor 30 by capturing an image of the same subject a second time (second image capturing) using the multispectral camera 1. When capturing an image of the same subject a second time (second image capturing), a filter 14 is inserted to reduce the transmittance of the first wavelength band including the small light wavelength λ1 to X%.

[0098] 5, filter 14 can be attached to the front of the lens of multispectral camera 1. Filter 14 preferably has a low transmittance (transmittance X%) for the first wavelength band and a high transmittance for the second wavelength band, and for example, optical filters such as a notch filter that blocks the first wavelength band and transmits the other wavelength bands, a bandpass filter that transmits the second wavelength band and blocks the other wavelength bands including the first wavelength band, and an edge filter that cuts off wavelengths equal to or higher than a specific lower limit wavelength in the first wavelength band can be used.

[0099] The processor 100 subtracts (performs a difference calculation) the first spectral image acquired from the image sensor 30 by the second imaging with the filter 14 interposed therebetween from the first spectral image acquired from the image sensor 30 by the first imaging, thereby performing interference reduction processing (second interference reduction processing) that reduces interference (crosstalk) of the second spectral image with respect to the first spectral image acquired by the first imaging.

[0100] Finally, the entire first spectral image after the second interference reduction process is multiplied by 100 / (100-X). This process is not necessarily required because it adjusts the luminance value of the first spectral image after the second interference reduction process.

[0101] When the processor 100 performs the second interference reduction processing by difference calculation between the two sets of first spectral images acquired from the image sensor 30 by the first imaging and the second imaging, it is preferable to perform image processing to align the position of one of the first spectral images with the other first spectral image so that no image misalignment occurs between the two sets of first spectral images.

[0102] On the other hand, the second spectral image of wavelength λ2, which is less affected by crosstalk, can be obtained by performing a first interference reduction process based on the first spectral image of wavelength λ1 and the second spectral image of wavelength λ2 acquired from the image sensor 30 by the first imaging.

[0103] <Imaging sequence> The order of imaging between the first and second imaging is not limited to the first imaging followed by the second imaging, but may be such that the second imaging with filter 14 attached is performed first, and then the first imaging with filter 14 removed.

[0104] Regardless of the imaging order described above, a second interference reduction process is performed in which the first spectral image acquired by the second imaging is subtracted from the first spectral image acquired by the first imaging, thereby reducing crosstalk from the second spectral image to the first spectral image after the second interference reduction process.

[0105] On the other hand, the processor 100 may obtain, as the second spectral image, a spectral image in which crosstalk from the first spectral image has been reduced (by executing a first interference reduction process) through a calculation using the first and second spectral images acquired by the first imaging and a first crosstalk constant, which will be described later, or may directly acquire the second spectral image acquired by the second imaging with the filter 14 attached.

[0106] <Method for reducing crosstalk in spectral images with three or more wavelengths> When acquiring a first spectral image, a second spectral image, and a third spectral image for each of three wavelengths (λ1, λ2, λ3) from the image sensor 30 of the multispectral camera 1, if the amount of light of the second spectral image and the third spectral image is sufficiently large compared to the amount of light of the first spectral image input to the multispectral camera 1, the first spectral image output from the image sensor 30 is significantly affected by crosstalk from the second spectral image and the third spectral image.

[0107] In this case, similarly to the above-described case of two wavelengths, the second imaging is performed with a filter 14 that reduces the transmittance of the wavelength λ1 interposed therebetween. The second interference reduction process for reducing crosstalk from the second and third spectral images to the first spectral image is performed by subtracting the first spectral image acquired by the second imaging from the first spectral image acquired by the first imaging.

[0108] On the other hand, an interference reduction process for reducing crosstalk from other spectral images to the second and third spectral images, which are less affected by crosstalk, can be performed by a calculation (first interference reduction process) using the first spectral image, the second spectral image, and the third spectral image acquired by the first imaging and a first crosstalk constant, which will be described later.

[0109] On the other hand, when the light amount of the third spectral image is small and the influence of crosstalk from the second spectral image is large like the first spectral image, a third imaging is performed with another filter interposed that reduces the transmittance of the third wavelength band including the wavelength λ3 with a small light amount. Then, a second interference reduction process is performed in which the third spectral image acquired by the third imaging is subtracted from the third spectral image acquired by the first imaging, thereby reducing crosstalk from the other spectral images in the third spectral image acquired by the first imaging.

[0110] In the case of four or more wavelengths, a color multispectral camera is used in which, in addition to the micro-polarizers P1, P2, P3, and P4 in the image sensor, color filter elements that transmit light in the red (Red, R) wavelength band, the green (Green, G) wavelength band, and the blue (Blue, B) wavelength band are arranged in each pixel, and spectral images of four or more wavelengths output from the image sensor of the color multispectral camera are acquired, as shown in Patent Document 2. Then, when there are multiple spectral images that are significantly affected by crosstalk from other wavelength bands among the spectral images of four or more wavelengths, multiple images are taken with multiple filters corresponding to each of the multiple spectral images interposed in sequence, thereby performing a second interference reduction process that reduces the effect of crosstalk on the multiple spectral images that are significantly affected by crosstalk.

[0111] [Method for reducing crosstalk in two-wavelength spectral images] Let yλ1 and yλ2 be the first spectral image of wavelength λ1 and the second spectral image of wavelength λ2 output from the image sensor 30 of the multispectral camera 1, respectively, let xλ1 and xλ2 be the theoretical first spectral image without interference and the theoretical second spectral image without interference, respectively, let α be the ratio of interference between the first spectral image (xλ1) and the second spectral image (xλ2), and let β be the ratio of interference between the second spectral image (xλ2) and the first spectral image (xλ1). The first spectral image (yλ1) and the second spectral image (yλ2) are expressed by the following formulas (1) and (2): yλ1=xλ1-αxλ1+βxλ2……(1) yλ2=xλ2+αxλ1-βxλ2……(2) It can be expressed as follows.

[0112] The normal interference reduction process (first interference reduction process) described in Patent Documents 1 and 2 is a process of calculating a theoretical first spectral image (xλ1) without interference and a theoretical second spectral image (xλ2) without interference by using formulas (1) and (2). That is, the interference ratios α and β (first crosstalk constants) in formulas (1) and (2) can be obtained by, for example, acquiring spectral images of wavelengths λ1 and λ2 without crosstalk by separately capturing an area light source of wavelength λ1 and an area light source of wavelength λ2 by the multispectral camera 1, acquiring spectral images of wavelengths λ1 and λ2 with crosstalk by simultaneously capturing an area light source of wavelength λ1 and an area light source of wavelength λ2 by the multispectral camera 1, and comparing the spectral images of wavelengths λ1 and λ2 without crosstalk with the spectral images of wavelengths λ1 and λ2 with crosstalk. The crosstalk constant differs depending on the imaging conditions such as the focal length and aperture value of the imaging optical system 10, and the image height of the image sensor 30, and is therefore obtained according to the imaging conditions and image height.

[0113] Then, the processor 100 determines whether or not the first spectral image is significantly affected by crosstalk from the second spectral image (exceeding a threshold value) as follows.

[0114] The first determination method determines a first representative value (the average value of the entire screen of the first spectral image (xλ1), or the average value within a range (an appropriately selected range) in the first spectral image (yλ1) where crosstalk removal is desired) of an image signal indicating a first spectral image (xλ1) calculated using equations (1) and (2), and a second representative value (the same average value as above) of an image signal indicating a second spectral image (xλ2) calculated using equations (1) and (2), and determines whether the influence of interference exceeds a threshold value (first threshold value) based on whether the difference (ratio) between the first representative value and the second representative value exceeds a threshold value.

[0115] In other words, the ratio of the light amounts of the first spectral image (xλ1) and the second spectral image (xλ2) input to the multispectral camera 1 can be determined from the ratio between the first representative value of the first spectral image (xλ1) after the first interference reduction processing and the second representative value of the second spectral image (xλ2) after the first interference reduction processing. The larger this light amount ratio is, the more the first spectral image (xλ1) with a small amount of light is affected by crosstalk from the second spectral image (xλ2) with a large amount of light.

[0116] In this example, when the ratio between the first representative value of the first spectral image (xλ1) and the second representative value of the second spectral image (xλ2) exceeds a first threshold (e.g., 10), it is determined that the influence of interference from the second spectral image (xλ2) on the first spectral image (xλ1) that has been subjected to the first interference reduction process exceeds a threshold. That is, although the first spectral image (xλ1) is an image that has been subjected to the first interference reduction process, when the ratio exceeds the first threshold, it is determined that the first interference reduction process cannot sufficiently reduce the influence of crosstalk from the second spectral image (xλ2) with a large amount of light on the first spectral image (xλ1) with a small amount of light.

[0117] The representative value is not limited to the average value, but may be the median value, mode value, or the like of the image signal of the entire screen of the spectral image or a selected range.

[0118] In addition, the present invention is not limited to calculating the ratio between the first representative value of the image signal representing the first spectral image (xλ1) after the first interference reduction processing and the second representative value of the image signal representing the second spectral image (xλ2) after the first interference reduction processing. It is also possible to calculate the ratio between the first representative value of the image signal representing the first spectral image (xλ1) after the first interference reduction processing and a representative value (third representative value) of the image signal representing the first spectral image (yλ1) before the first interference reduction processing.

[0119] When the first spectral image is significantly affected by crosstalk due to the second spectral image, the first spectral image (xλ1) after the first interference reduction processing will be significantly larger than the first spectral image (yλ1) before the first interference reduction processing.

[0120] Therefore, the processor 100 calculates the ratio between the first representative value of the first spectral image (xλ1) and the third representative value of the first spectral image (yλ1), and can determine whether the influence of crosstalk due to the second spectral image (xλ2) subjected to the first interference reduction processing exceeds a threshold value based on whether the calculated ratio exceeds a threshold value (second threshold value).

[0121] Next, a description will be given of the second interference reduction process and the third interference reduction process performed by the processor 100. The second interference reduction process and the third interference reduction process are interference reduction processes performed when it is determined that the influence of crosstalk cannot be sufficiently reduced by the first interference reduction process.

[0122] <Second interference reduction processing> As described above, in addition to the first imaging by the multispectral camera 1, a second imaging is performed with the filter 14 attached to the front of the lens.

[0123] Filter 14 has a low transmittance in a first wavelength band and a high transmittance in a second wavelength band, but in the following embodiment, the transmittance of the first wavelength band including the wavelength (λ1) is k (X%), and the transmittance of the second wavelength band including the wavelength (λ2) is 1 (100%).

[0124] If the first spectral image of wavelength λ1 and the second spectral image of wavelength λ2 output from the image sensor 30 of the multispectral camera 1 by the second imaging are denoted by y′λ1 and y′λ2, respectively, the first spectral image (y′λ1) and the second spectral image (y′λ2) are expressed by the following formulas (3) and (4): y'λ1=kxλ1-kαxλ1+βxλ2...(3) y'λ2= xλ2+kαxλ1-βxλ2……(4) In addition, xλ1, xλ2, α, and β in the formulas (3) and (4) are the same as those shown in the formulas (1) and (2).

[0125] Subtracting equation (3) from equation (1) and rearranging gives equation (5).

[0126] yλ1-y'λ1=(1-k)(1-α)xλ1……(5) The first spectral image (xλ1) can be expressed by equation (6), which is a modification of equation (5).

[0127] xλ1=A(yλ1-y'λ1) ……(6) where A=1 / {(1-k)(1-α)} A is a constant determined by k and α. Therefore, the first spectral image (xλ1) can be obtained by multiplying the difference image between the two pairs of first spectral images (yλ1, y′λ1) acquired by the image sensor 30 of the multispectral camera 1 corresponding to the first and second imaging by the constant A.

[0128] On the other hand, the second spectral image (xλ2) with a large amount of light is less affected by crosstalk from the first spectral image (xλ1) with a small amount of light, so it is possible to use the image calculated using equations (1) and (2) (the image subjected to the first interference reduction process).

[0129] Moreover, the second spectral image (xλ2) with a large amount of light can be obtained by approximating kα≒0 in the above-mentioned formula (4). That is, by approximating kα≒0, formula (4) can be rewritten as formula (7).

[0130] y'λ2=xλ2-βxλ2 =(1-β)xλ2……(7) Therefore, the second spectral image (xλ2) can be obtained by multiplying the second spectral image (y′λ2) acquired from the image sensor 30 of the multispectral camera 1 in response to the second imaging by a constant (1 / (1−β)).

[0131] The transmittance k (X%) of the first wavelength band including the wavelength (λ1) of the filter 14 can be set to, for example, 10%. Furthermore, by making the transmittance k not ≈ 0, even during the second imaging using the filter 14, the first interference reduction processing based on the formulas (3) and (4) is performed on the first spectral image (y'λ1) and the second spectral image (y'λ2) acquired from the image sensor 30 of the multispectral camera 1, so that videos of the first interference reduction processed first spectral image (xλ1) and second spectral image (xλ2) can be simultaneously acquired.

[0132] <Third interference reduction processing> The third interference reduction process obtains a first spectral image (yλ1) and a second spectral image (yλ2) output from the image sensor 30 of the multispectral camera 1 by the first imaging, and a first spectral image (y'λ1) and a second spectral image (y'λ2) output from the image sensor 30 of the multispectral camera 1 by the second imaging in which a filter 14 is attached to the front of the lens.

[0133] The first spectral image (yλ1) and the second spectral image (yλ2), and the first spectral image (y'λ1) and the second spectral image (y'λ2) can be expressed by the above-mentioned formulas (1) to (4), respectively.

[0134] Then, four unknowns (xλ1, xλ2, α, β) are found from these four equations (equations (1) to (4)). That is, in the second interference reduction process, the first crosstalk constants (α, β) are treated as unknowns, and a crosstalk constant (α, β) (second crosstalk constant) suitable for the current imaging is calculated. The calculated second crosstalk constant (α, β) is used as the crosstalk constant in equations (1) and (2), thereby calculating the first spectral image (xλ1) and the second spectral image (xλ2) based on equations (1) and (2). In addition, since the four unknowns (xλ1, xλ2, α, β) can be obtained from the four equations (1) to (4), the first spectral image (xλ1) and the second spectral image (xλ2) may be calculated directly based on equations (1) to (4) instead of calculating the first spectral image (xλ1) and the second spectral image (xλ2) based on equations (1) and (2) after obtaining the second crosstalk constants (α, β).

[0135] [Method for reducing crosstalk in three-wavelength spectral images] The first spectral image of wavelength λ1, the second spectral image of wavelength λ2, and the third spectral image of wavelength λ3 output from the image sensor 30 of the multispectral camera 1 are denoted as yλ1, yλ2, and yλ3, respectively, and the theoretical first spectral image without interference, the theoretical second spectral image without interference, and the theoretical third spectral image without interference are denoted as xλ1, xλ2, and xλ3, respectively. The first spectral image (xλ1) is a second spectral image (xλ2) and a third spectral image ( Let α2 and α3 be the ratios of interference between the second spectral image (xλ2) and the first spectral image (xλ1) and the third spectral image (xλ3), respectively, β1 and β3 be the ratios of interference between the second spectral image (xλ2) and the first spectral image (xλ1) and the third spectral image (xλ3), respectively, and γ1 and γ2 be the ratios of interference between the third spectral image (xλ3) and the first spectral image (xλ1) and the second spectral image (xλ2), respectively. The first spectral image (yλ1), the second spectral image (yλ2), and the third spectral image (yλ3) are expressed by the following formulas (8) to (10): yλ1=xλ1-α1xλ1+β1xλ2+γ1xλ3……(8) yλ2=xλ2+α2xλ1-β2xλ2+γ2xλ3……(9) yλ3=xλ3+α3xλ1+β3xλ2-γ3xλ3……(10) It can be expressed as follows.

[0136] In addition, the interference ratios α1 to α3, β1 to β3, and γ1 to γ3 (first crosstalk constants) in equations (8) to (10) can be calculated in advance. In addition, in equations (8) to (10), α1+α2+α3=0, β1+β2+β3=0, and γ1+γ2+γ3=0.

[0137] Here, a case will be described in which it is determined that the first interference reduction process using equations (8) to (10) cannot sufficiently reduce the influence of crosstalk from the second spectral image (xλ2) and the third spectral image (xλ3) having a large amount of light with respect to the first spectral image (xλ1) having a small amount of light.

[0138] In this case, similar to the crosstalk reduction method for two-wavelength spectral images, a second image is captured with a filter 14 attached to the front of the lens in addition to the first image captured by the multispectral camera 1. The filter 14 has a low transmittance in the first wavelength band and a high transmittance in the second and third wavelength bands, where the transmittance of the first wavelength band including the wavelength (λ1) is k, and the transmittance of the second wavelength band including the wavelength λ2 and the third wavelength band including the wavelength λ3 is 1.

[0139] If the first spectral image at wavelength λ1, the second spectral image at wavelength λ2, and the second spectral image at wavelength λ3 output from the image sensor 30 of the multispectral camera 1 by the second imaging are denoted as y′λ1, y′λ2, and y′λ3, respectively, the first spectral image (y′λ1), the second spectral image (y′λ2), and the third spectral image (y′λ3) are expressed by the following formulas (11) to (134): y'λ1=kxλ1-kα1xλ1+β1xλ2+γ1xλ3...(11) y'λ2= xλ2+kα2xλ1-β2xλ2+γ2xλ3……(12) y'λ3= xλ3+kα3xλ1+β3xλ2-γ3xλ3……(13) In addition, xλ1 to xλ3, α1 to α3, β1 to β3, and γ1 to γ3 in formulas (11) to (13) are the same as those shown in formulas (8) to (10).

[0140] Subtracting equation (11) from equation (8) and rearranging gives equation (14).

[0141] yλ1-y'λ1=(1-k)(1-α1)xλ1……(14) The first spectral image (xλ1) can be expressed by equation (15), which is a modification of equation (14).

[0142] xλ1=B(yλ1-y'λ1) ……(15) Here, B = 1 / {(1-k)(1-α1)} B is a constant determined by k and α1. Therefore, the first spectral image (xλ1) can be obtained by multiplying the difference image between the two pairs of first spectral images (yλ1, y′λ1) acquired from the image sensor 30 of the multispectral camera 1 corresponding to the first and second imaging by the constant B.

[0143] On the other hand, if the second crosstalk constants (α1 to α3, β1 to β3, γ1 to γ3) are newly calculated and applied to equations (8) to (10) to obtain the first to third spectral images (xλ1) to (xλ3), the number of equations is insufficient relative to the unknowns.

[0144] Therefore, a filter other than filter 14 (a filter with a different wavelength band that reduces transmittance) is attached to the front of the lens of multispectral camera 1 to perform a third image capture, and spectral images of three wavelengths are obtained from image sensor 30 of multispectral camera 1 by each of the first to third images capture, thereby increasing the number of equations and finding the unknowns.

[0145] <Example> FIG. 13 shows a spectral image generated to confirm the actual effect of the second interference reduction processing.

[0146] FIG. 13A is a diagram showing an ideal first spectrum image without interference acquired from the image sensor 30 when an image of a surface light source with a small amount of light of wavelength λ1 (720 nm) is captured by a multispectral camera (first image capture).

[0147] On the other hand, FIG. 13(B) shows a first spectral image after the second interference reduction process, in which a first image and a second image are captured by the multispectral camera 1 of a surface light source when a surface light source with a small light amount of wavelength λ1 (720 nm) and a surface light source with a large light amount of wavelength λ2 (615 nm) are simultaneously emitted, and then a second interference reduction process is performed to obtain a difference image between the two sets of first spectral images acquired by the image sensor 30 corresponding to the first and second images.

[0148] In the second imaging, the filter 14 attached to the front of the lens of the multispectral camera 1 had a transmittance of 10% for the wavelength λ1 (720 nm).

[0149] The first spectral image after the second interference reduction processing shown in FIG. 13(B) is close to the ideal first spectral image without interference shown in FIG. 13(A).

[0150] FIG. 14 is an image showing the influence of crosstalk, obtained by dividing the first spectral image after the second interference reduction processing shown in FIG. 13(B) by the ideal first spectral image without interference shown in FIG. 13(A).

[0151] The image showing the effect of crosstalk shown in FIG. 14 shows a distribution in the range of 0.975 to 1.025 (a range of 5%).

[0152] It can be seen that the image shown in FIG. 14 with a distribution in the range of 0.975 to 1.025 (5% range) has reduced crosstalk, compared to the image shown in FIG. 11(A) with a distribution in the range of 0.8 to 1 (20% range) (an image showing the effect of crosstalk in the first spectral image before the second interference reduction processing).

[0153] [Information processing method] First Embodiment FIG. 15 is a flowchart showing a first embodiment of an information processing method according to the present invention.

[0154] 15, a range in which crosstalk removal is desired to be performed is selected by a user operation from within the imaging range captured by the multispectral camera 1 (step S10). That is, the user uses a user interface including the operation unit 140 and the display 120 to select the range of the subject to be measured, for example, while referring to the first spectral image displayed on the display 120. The processor 100 accepts the range selected by the user operation as the range in which crosstalk is to be removed.

[0155] Also, in step S10, the user selects a wavelength to be measured, and the wavelength selected by the user is accepted by the processor 100. In this example, it is assumed that the wavelength λ1 is selected.

[0156] Next, the user operates the multispectral camera 1 to perform a first image capture of the subject using the multispectral camera 1 (step S12). When the first image capture is performed in step S12, the processor 100 acquires spectral images for each of a plurality of wavelength bands output from the image sensor 30 of the multispectral camera 1 (in this example, a first spectral image (yλ1) at wavelength λ1 and a second spectral image (yλ2) at wavelength λ2).

[0157] The processor 100 performs a first interference reduction process for reducing interference on the acquired first spectral image (yλ1) and second spectral image (yλ2) by calculation using the first crosstalk constant (step S14).

[0158] The first interference reduction process is a process of calculating a theoretical first spectral image (xλ1) without interference and a theoretical second spectral image (xλ2) without interference using the above-mentioned equations (1) and (2).

[0159] Specifically, equations (1) and (2) can be rewritten as the following determinant (16).

[0160]

number

[0161]

number

[0162] The interference cancellation matrix shown in determinant (17) can be calculated in advance based on imaging conditions such as the focal length and aperture value of the imaging optical system 10 of the multispectral camera 1, and the image height of the image sensor 30, and stored in the memory 110. When performing the first interference cancellation process on the first spectral image (yλ1) and the second spectral image (yλ2) acquired from the image sensor 30 of the multispectral camera 1, the processor 100 reads out an interference cancellation matrix corresponding to the imaging conditions, etc. from the memory 110, and multiplies the read interference cancellation matrix by the first spectral image (yλ1) and the second spectral image (yλ2) to calculate the first spectral image (xλ1) and the second spectral image (xλ2).

[0163] Next, based on the first spectral image (xλ1) and the second spectral image (xλ2) calculated in step S14, the processor 100 determines whether the first spectral image (xλ1) of the wavelength λ1 selected by the user is significantly affected (above a threshold) by crosstalk from the second spectral image (xλ2) of another wavelength λ2 (step S20).

[0164] That is, the processor 100 acquires a first spectral image (xλ1) and a second spectral image (xλ2) of the range selected in step S10, and determines an average value of image signals indicating the first spectral image (xλ1) of the selected range as a first representative value, and an average value of image signals indicating the second spectral image (xλ2) of the selected range as a second representative value. If a ratio between the determined first average value and the second representative value exceeds a preset first threshold value (e.g., 10), the processor 100 determines that the first spectral image (xλ1) is significantly affected by crosstalk due to the second spectral image (xλ2).

[0165] Whether or not the influence of crosstalk is large may be determined based on the ratio between the first spectral image (xλ1) after the first interference reduction processing and the first spectral image (yλ1) before the first interference reduction processing.

[0166] If the processor 100 determines in step S20 that the influence of crosstalk is not large (in the case of “No”), the processor 100 proceeds to step S50. In this case, in step S50, the processor 100 outputs the first spectral image (xλ1) and the second spectral image (xλ2) after the first interference reduction process obtained in step S14.

[0167] On the other hand, if the processor 100 determines in step S20 that the influence of crosstalk is large (in the case of "Yes"), the processor 100 transitions to step S30.

[0168] When the processor 100 determines that the influence of crosstalk is large, the processor 100 notifies the user of a second image capture of the subject by the multispectral camera 1 having a filter 14 attached to the front of the lens, the filter 14 reducing the transmittance of the first wavelength band including the wavelength λ1, or causes the second image capture to be performed (step S30). That is, in step S30, in response to the notification of the second image capture, the user attaches the filter 14 to the front of the lens, and the multispectral camera 1 to which the filter 14 is attached performs a second image capture of the same subject. In addition, in the case of the multispectral camera 1 having a filter replacement function or an attachment / detachment function, the processor 100 may operate the filter replacement function or the attachment / detachment function to interpose the filter 14 on the optical path of the imaging optical system, and cause the multispectral camera 1 to perform a second image capture of the same subject. In step S30, when the second image capture is performed, the processor 100 acquires at least the first spectral image (y'λ1) of the first spectral image (y'λ1) and the second spectral image (y'λ2) output from the image sensor 30 of the multispectral camera 1.

[0169] Next, the processor 100 performs a process (second interference reduction process) of subtracting the first spectral image (y'λ1) acquired by the second imaging with a filter in step S30 from the first spectral image (yλ1) acquired by the first imaging without a filter in step S12, and obtains the subtracted difference image as the first spectral image (xλ1) (step S40).

[0170] Note that the first spectral image (xλ1) calculated when the influence of crosstalk is large does not necessarily have to be calculated by subtracting the two first spectral images (yλ1) and (y'λ1) (second interference reduction process). The first spectral image (xλ1) may be calculated by performing a process (third interference reduction process) of calculating at least the unknown quantity (xλ1) among the four unknown quantities (xλ1, xλ2, α, β) from the above-mentioned four equations (equations (1) to (4)) into which the first spectral image (yλ1) and the second spectral image (yλ2) acquired in the first imaging without a filter in step S12, and the first spectral image (y'λ1) and the second spectral image (y'λ2) acquired in the second imaging with a filter in step S30 are substituted.

[0171] In step S50, if it is determined that the influence of crosstalk is not large as described above, the first spectral image (xλ1) and the second spectral image (xλ2) obtained in step S14 after the first interference reduction processing are output. However, if it is determined that the influence of crosstalk is large, the first spectral image (xλ1) after the second interference reduction processing obtained in step S40 is output instead of the first spectral image (xλ1) after the first interference reduction processing.

[0172] <Second embodiment> FIG. 16 is a flowchart showing a second embodiment of the information processing method according to the present invention.

[0173] In FIG. 16, the same steps as those in the first embodiment shown in FIG. 15 are denoted by the same step numbers, and detailed descriptions thereof will be omitted.

[0174] 16 differs from the first embodiment shown in Fig. 15 in that the user selects the wavelength λ1 of the first spectral image for determining whether the influence of crosstalk is large, whereas the processor 100 automatically selects the wavelength λ1. Also, the multiple spectral images captured by the multispectral camera 1 are spectral images of wavelengths λ1, λ2, ..., λn, and are not limited to two spectral images.

[0175] 16, the processor 100 sets a specific wavelength when determining whether or not the influence of crosstalk from spectral images of other wavelengths on the spectral image of the specific wavelength is large. In this case, the processor 100 sets the wavelength λmin corresponding to the spectral image having the smallest output signal among the multiple spectral images output from the image sensor 30 by the first imaging of the multispectral camera 1 in step S12 as the specific wavelength λ1, and sets the other one or more wavelengths as wavelengths λ2, ..., λn.

[0176] Moreover, each output signal of the plurality of spectral images may be a representative value (e.g., an average value) of the image signals indicating each spectral image in the range selected in step S10. Furthermore, instead of the plurality of spectral images output from the image sensor 30, a plurality of spectral images that have been subjected to the first interference reduction process in step S14 may be used to set the specific wavelength λ1, etc., as described above.

[0177] In step S22, it is determined whether or not the influence of crosstalk from the spectral images of the other wavelengths λ2, . . . , λn is large with respect to the spectral image of the specific wavelength λ1 set as described above.

[0178] If the processor 100 determines in step S22 that the influence of crosstalk is not large (in the case of "No"), the processor 100 proceeds to step S52. In this case, in step S52, the processor 100 outputs the spectral images of the wavelengths λ1, λ2, ..., λn after the first interference reduction process obtained in step S14.

[0179] On the other hand, if the processor 100 determines in step S22 that the influence of crosstalk is large (in the case of "Yes"), the processor 100 transitions to step S32.

[0180] When the processor 100 determines that the influence of crosstalk is large, it notifies or causes the second imaging of the subject to be performed by the multispectral camera 1 having a filter 14 attached to the front of the lens, the filter 14 reducing the transmittance of the first wavelength band including the specific wavelength λ1 (λmin) (step S32). Then, when the second imaging is performed in step S32, the processor 100 calculates the first spectral image (y'λ1), the second spectral image (y'λ2), ..., the n-th spectral image (y'λ n ), at least the first spectral image (y′λ1) is acquired.

[0181] Next, the processor 100 performs a process (second interference reduction process) of subtracting the first spectral image (y'λ1) acquired by the second imaging with a filter in step S32 from the first spectral image (yλ1) acquired by the first imaging without a filter in step S12, and obtains the subtracted difference image as the first spectral image (xλ1) (step S42).

[0182] In step S52, if it is determined that the influence of crosstalk is not large as described above, the spectral images of wavelengths λ1, λ2, ..., λn after the first interference reduction processing obtained in step S14 are output. However, if it is determined that the influence of crosstalk is large, the first spectral image of wavelength λ1 after the second interference reduction processing obtained in step S42 is output instead of the first spectral image of wavelength λ1 after the first interference reduction processing.

[0183] [others] The information processing device 2 of this embodiment is a device separate from the multispectral camera 1, but the present invention may also be a device in which the multispectral camera 1 and the information processing device 2 are integrated, that is, a multispectral camera 1 equipped with an information processing device 2.

[0184] In addition, the crosstalk constant (matrix indicating crosstalk) indicated by the (2 × 2) matrix of determinant (16) corresponding to equations (1) and (2) is not limited to that shown in determinant (16), and the numerical values ​​of the (2 × 2) matrix can be values ​​appropriately determined by simulation or the like.

[0185] In this embodiment, the hardware structure of a processing unit that executes various processes, such as the processor 100 of the information processing device 2, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electric circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically for executing specific processes.

[0186] One processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0187] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0188] The present invention also includes an information processing program that, when installed in a computer, causes the computer to function as the information processing device according to the present invention, and a non-volatile storage medium on which this information processing program is recorded.

[0189] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0190] 1. Multispectral camera 2. Information processing device 10...Photographic optical system 10A…Front group lens 10B…Rear lens 12...Slit 14…Filter 20...Filter unit 22...Frame 22A…Opening 23...Filter set 24…Black Mylar filter 30...Image sensor 32…Pixel array layer 34...Polarizing filter element array layer 36...Microlens array layer 100…Processor 110…Memory 120…Display 130... Input / Output interface 140...Operation unit BPF1~BPF3 Bandpass Filters P1~P4 Micro polarizers PL1~PL3 Polarizing Filters S10~S52…Step

Claims

1. In an information processing device having a processor, The processor, acquiring spectral images for a plurality of wavelength bands output from an image sensor of a multispectral camera of a first image of a subject captured by the multispectral camera, the first spectral image and one or more second spectral images corresponding to a first wavelength band and one or more second wavelength bands, respectively; performing a first interference reduction process for reducing interference on the first spectral image and the second spectral image by a calculation using a first crosstalk constant; determining whether or not an influence of interference caused by the second spectral image on the first spectral image that has been subjected to the first interference reduction process exceeds a threshold; Information processing device.

2. The first and second spectral images output from the image sensor are respectively represented by yλ 1 , y 2 Let the theoretical first spectrum image without interference and the theoretical second spectrum image without interference be xλ 1 , x 2 and the first spectral image (xλ 1 ) is the second spectral image (xλ 2 ) is the interference ratio to the second spectral image (xλ 2 ) is the first spectral image (xλ 1 ) is the interference ratio β, the first spectral image (yλ 1 ) and the second spectral image (yλ 2 ) are represented by the following formulas (1) and (2): yλ 1 =aλ 1 -axl 1 +βkλ 2 ……(1) yλ 2 =aλ 2 +axl 1 -βxλ 2 ……(2) and The first interference reduction process is to obtain a theoretical first spectral image (xλ) without interference by using the formulas (1) and (2). 1 ) and the theoretical second spectral image (xλ 2 ) is calculated. The information processing device according to claim 1 .

3. The processor, determining a first representative value of an image signal representing the first spectral image subjected to the first interference reduction processing, and a second representative value of an image signal representing the second spectral image subjected to the first interference reduction processing; determining whether the influence of the interference exceeds a threshold value based on whether a difference between the first representative value and the second representative value exceeds a first threshold value; The information processing device according to claim 1 .

4. The processor, determining a first representative value of an image signal representing the first spectral image after the first interference reduction processing and a third representative value of an image signal representing the first spectral image before the first interference reduction processing; determining whether the influence of the interference exceeds a threshold value based on whether a ratio between the first representative value and the third representative value exceeds a second threshold value; The information processing device according to claim 1 .

5. The processor, in response to a determination that the influence of the interference exceeds a threshold, notifying a second image of the subject by the multispectral camera, the second image being captured using a filter that reduces a transmittance of the first wavelength band; The information processing device according to claim 1 .

6. The processor, In response to the determination that the influence of the interference exceeds a threshold, a second image of the subject is captured by the multispectral camera through a filter that reduces the transmittance of the first wavelength band. The information processing device according to claim 1 .

7. The processor, When the first image capturing and the second image capturing are performed on the same subject, two sets of the first spectral images are acquired from the image sensor corresponding to the first image capturing and the second image capturing, performing a second interference reduction process for reducing interference with the first spectral images by calculating a difference between the two sets of the first spectral images; 7. The information processing device according to claim 5 or 6.

8. Two sets of the first spectral images output from the image sensor corresponding to the first imaging and the second imaging are respectively represented as yλ 1 , y′λ 1 Let the theoretical first spectrum image without interference and the theoretical second spectrum image without interference be xλ 1 , x 2 and the first spectral image (xλ 1 ) is the second spectral image (xλ 2 ) is the interference ratio to the second spectral image (xλ 2 ) is the first spectral image (xλ 1 ) is β, and the transmittance of the first wavelength band by the filter is k, the first spectral image (yλ 1 ) and (y′λ 1 ) is expressed by the following formula (3) and formula (4): yλ 1 = xλ 1 -axl 1 + + +++ 2 ……(3) y'λ 1 =kxλ 1 -kaxl 1 +βkλ 2 ……(4) and The second interference reduction process subtracts the formula (4) from the formula (3) to obtain the first spectral image (xλ 1 ) is calculated. The information processing device according to claim 7.

9. The processor performs the first interference reduction process to reduce interference on the second spectral image. The information processing device according to claim 7.

10. The processor acquires the second spectral image output from the image sensor when the second imaging is performed, as an image in which interference caused by the second spectral image is reduced.

7. The information processing device according to claim 5 or 6.

11. The processor, When the first image capturing and the second image capturing are performed on the same subject, two sets of the first spectral image and the second spectral image are acquired which are output from the image sensor in response to the first image capturing and the second image capturing, Calculating a second crosstalk constant based on the two sets of the first spectral image and the second spectral image; performing a third interference reduction process for reducing interference on the first spectral image and the second spectral image by a calculation using the second crosstalk constant; 7. The information processing device according to claim 5 or 6.

12. the processor sets a spectral image set by a user among the plurality of spectral images output from the image sensor as the first spectral image, and sets other spectral images as one or more of the second spectral images; The information processing device according to claim 1 .

13. the processor sets a spectral image having a smallest output signal among the plurality of spectral images output from the image sensor as the first spectral image, and sets other spectral images as one or more of the second spectral images; The information processing device according to claim 1 .

14. A multispectral camera comprising the information processing device according to any one of claims 1 to 6, 12 and 13.

15. 1. An information processing device that processes spectral images for a plurality of wavelength bands output from an image sensor of a split-pupil multispectral camera when a first image of a subject is captured by the multispectral camera, comprising: a processor acquiring a plurality of spectral images output from the image sensor, the first spectral image and one or more second spectral images corresponding to a first wavelength band and one or more second wavelength bands, respectively; performing a first interference reduction process for reducing interference on the first spectral image and the second spectral image by a calculation using a first crosstalk constant by the processor; determining whether or not an influence of interference caused by the second spectral image on the first spectral image subjected to the first interference reduction processing exceeds a threshold; An information processing method comprising:

16. determining a first representative value of an image signal representing the first spectral image subjected to the first interference reduction processing, and a second representative value of an image signal representing the second spectral image subjected to the first interference reduction processing, by the processor; determining whether the influence of the interference exceeds a threshold value based on whether a difference between the first representative value and the second representative value exceeds a first threshold value; The information processing method according to claim 15 , comprising:

17. determining a first representative value of an image signal representing the first spectral image after the first interference reduction processing, and a third representative value of an image signal representing the first spectral image before the first interference reduction processing; determining whether the influence of the interference exceeds a threshold value based on whether a difference between the first representative value and the third representative value exceeds a second threshold value; The information processing method according to claim 15 , comprising:

18. When the processor determines that the influence of the interference exceeds a threshold, the processor notifies the multispectral camera of a second image of the subject using a filter that reduces the transmittance of the first wavelength band, or causes the multispectral camera to perform the second image. The information processing method according to claim 15.

19. acquiring, by the processor, two sets of the first spectral images output from the image sensor corresponding to the first imaging and the second imaging when the first imaging and the second imaging are performed; performing a second interference reduction process by the processor to reduce interference in the first spectral image by calculating a difference between the two sets of the first spectral images; The information processing method according to claim 18 , comprising:

20. acquiring, by the processor, two sets of the first spectral image and the second spectral image output from the image sensor corresponding to the first image capturing and the second image capturing, when the first image capturing and the second image capturing are performed on the same object; calculating a second crosstalk constant based on two sets of the first spectral image and the second spectral image by the processor; performing a third interference reduction process by the processor to reduce interference on the first spectral image and the second spectral image by a calculation using the second crosstalk constant; The information processing method according to claim 18 , comprising:

21. a function of acquiring, when a first image of a subject is captured by the split-pupil multispectral camera, spectral images for a plurality of wavelength bands output from an image sensor of the multispectral camera, a first spectral image and one or more second spectral images corresponding to a first wavelength band and one or more second wavelength bands, respectively; a function of performing a first interference reduction process for reducing interference on the first spectral image and the second spectral image by calculation using a first crosstalk constant; a function of determining whether or not an influence of interference caused by the second spectral image on the first spectral image that has been subjected to the first interference reduction process exceeds a threshold; An information processing program that enables a computer to realize the above.

Images