Image processing method and spectroscopic camera system

Abstract

To provide an image processing method and a spectroscopic camera system that can create a color composite image with high color reproducibility regardless of the performance of a spectroscopic camera.SOLUTION: An image processing method is to convert a plurality of pieces of spectroscopic image data of spectral wavelengths picked up by a spectroscopic camera into a color image by using processors, and the processors acquire the plurality of pieces of spectroscopic image data from a storage unit, multiply the spectroscopic spectrum of pixels in the plurality of pieces of spectroscopic image data by a correction constant set for every wavelength to calculate a correction value, add up the correction values at the same pixel position to calculate a color conversion value, and create a color composite image based on the color conversion value. The correction constant is set such that the total sum spectrum obtained by adding up characteristic spectra obtained by multiplying the sensitivity characteristics for the spectral wavelengths of the spectroscopic camera and the correction constants corresponding to the wavelengths matches a target spectrum of an arbitrary color filter.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing method for processing a spectroscopic image, and a spectroscopic camera system. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known an image processing device that uses a spectroscopic camera to capture an image of an object, acquires spectral images for a plurality of wavelengths, and generates a color image of the object based on these spectral images (see, for example, Patent Document 1). The image processing device described in Patent Document 1 acquires spectral image data of an object at multiple wavelengths and corrects the transmittance, sensitivity, and other parameters of the spectral image data for each wavelength band λi. Then, the luminance value of each pixel in the spectral image data is multiplied by pseudocolor conversion functions r′(λi), g′(λi), and b′(λi) or color matching functions r￣(λi), g￣(λi), and b￣(λi) to obtain tristimulus values ​​R, G, and B. In other words, RGB values ​​are calculated by applying any color filter (e.g., r filter, g filter, b filter) to the luminance value of each pixel. The RGB values ​​of each pixel in the spectral image data for each wavelength band are then integrated for each pixel to calculate R, G, and B data for each pixel. A color composite image is then generated based on the calculated R, G, and B data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-33222 Summary of the Invention [Problem to be solved by the invention]

[0004] The image processing device described in Patent Document 1 multiplies the luminance value by the same pseudo-color conversion function or color matching function as a correction coefficient, regardless of the characteristics of the spectroscopic camera, i.e., regardless of the spectroscopic performance at a wavelength when capturing an image at that wavelength using the spectroscopic camera. This is simply multiplication of the camera's luminance value by the filter function of an arbitrary color filter. In this configuration, if the spectral element incorporated in the spectroscopic camera has high resolution and can transmit desired wavelengths with a narrow half-width, it is possible to generate a color composite image with high color reproducibility for the imaged object. However, as the half-width of the spectral element becomes wider and the wavelength resolution becomes lower, there is a problem in that errors increase and the color reproducibility of the color composite image decreases. In other words, the conventional image processing method as described in Patent Document 1 has a problem in that the color reproducibility of the color composite image is affected by the characteristics of the spectroscopic camera. [Means for solving the problem]

[0005] An image processing method according to a first aspect of the present disclosure is an image processing method for converting spectral image data of a plurality of spectral wavelengths captured by a spectroscopic camera into a color image using one or more processors, the method comprising causing the one or more processors to: acquire from a storage unit a data cube including spectral spectra based on luminance values ​​at the same pixel position in a plurality of the spectral image data corresponding to each of the plurality of spectral wavelengths; calculate a correction value by multiplying the spectral spectrum of each pixel by a correction constant set for each wavelength; calculate a color conversion value by summing the correction values ​​at the same pixel position in the plurality of the spectral image data; and generate a color composite image based on the color conversion value for each pixel; wherein a characteristic spectrum is a spectrum obtained by multiplying the sensitivity characteristic spectrum of the spectroscopic camera for the spectral wavelengths by the correction constant corresponding to each wavelength; and a total spectrum is a spectrum obtained by summing the characteristic spectra corresponding to the plurality of the spectral wavelengths, the correction constant is set so that the total spectrum matches a target spectrum, which is the spectrum of an arbitrary color filter.

[0006] In the image processing method of this aspect, the one or more processors are further caused to acquire the sensitivity characteristic spectra for the plurality of spectral wavelengths in the spectroscopic camera, acquire the target spectrum, and calculate the correction constants corresponding to each wavelength so that the total spectrum matches the target spectrum.

[0007] In the image processing method of this aspect, the target spectrum is F(λ), and the wavelength λ i The correction constant for a i and the spectroscopic wavelength Λ i The sensitivity characteristic when capturing the spectral image data is S i (λ), the correction constant a i is Σ({Σa i S i (λ)}-F(λ)) 2 is the minimum value.

[0008] a correction value calculation unit that calculates a correction value by multiplying the spectral spectrum of each pixel by a correction constant set for each wavelength; and an image synthesis unit that calculates a color conversion value by summing the correction values ​​for the same pixel position in the plurality of pieces of spectral image data, and generates a color composite image based on the color conversion value for each pixel. The correction constant is set so that when a characteristic spectrum is defined as a spectrum obtained by multiplying the sensitivity characteristic spectrum of the spectral wavelength of the spectroscopic camera by the correction constant corresponding to each wavelength, and a spectrum obtained by integrating each of the characteristic spectra corresponding to the plurality of spectral wavelengths is defined as a total spectrum, the correction constant is set so that the total spectrum matches a target spectrum, which is the spectrum of an arbitrary color filter.

[0009] In the spectroscopic camera system of this aspect, the correction constants are pre-recorded in the storage unit.

[0010] In the spectroscopic camera system of this aspect, the memory unit pre-stores the sensitivity characteristic spectra of a plurality of the spectroscopic cameras, each having a different sensitivity characteristic spectrum, and the correction constant for each of the sensitivity characteristic spectra, and the system further includes a camera characteristic acquisition unit that acquires the sensitivity characteristic spectrum of the spectroscopic camera, and the correction value calculation unit reads out the correction constant corresponding to the acquired sensitivity characteristic spectrum from the memory unit and calculates the correction value.

[0011] The spectroscopic camera system of this aspect further includes a camera characteristic acquisition unit that acquires the sensitivity characteristic spectra for the plurality of spectroscopic wavelengths in the spectroscopic camera, and a constant calculation unit that calculates the correction constant corresponding to each wavelength so that the total spectrum matches the target spectrum. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of a spectroscopic camera system according to a first embodiment. [Figure 2] 4 is a flowchart showing a method for calculating a correction constant according to the first embodiment. [Figure 3] FIG. 4 is an image diagram showing an example of a method for calculating a correction constant in the first embodiment. [Figure 4] FIG. 10 is a diagram showing an example of filter data. [Figure 5] 10 is a flowchart showing a method for generating a color composite image in the first embodiment. [Figure 6] FIG. 10 is an image diagram showing an example of a method for generating a color composite image. [Figure 7] FIG. 10 is a schematic diagram showing a schematic configuration of a spectroscopic camera system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] [First embodiment] A spectroscopic camera system according to an embodiment of the present invention will be described below. FIG. 1 is a schematic diagram showing the general configuration of a spectroscopic camera system. As shown in FIG. 1, the spectroscopic camera system of this embodiment includes a spectroscopic camera 10 that captures a spectroscopic image of an object, and an image processing device 20 that receives spectroscopic image data of the spectroscopic image captured by the spectroscopic camera 10. The spectroscopic camera system may also be provided with an optical element (illumination unit) that irradiates the object to be imaged with light. In the spectroscopic camera system of this embodiment, the image processing device 20 combines the spectroscopic images of the multiple spectral wavelengths captured by the spectroscopic camera 10 to generate a color image. Each component of such a spectroscopic camera system will be described below.

[0014] [Configuration of Spectroscopic Camera 10] As shown in FIG. 1, the spectroscopic camera 10 includes an optical lens system 11, a spectroscopic element 12, an image sensor 13, a camera control unit 14, and the like. The optical lens system 11 is composed of, for example, a plurality of lenses that guide incident light reflected by an object to be imaged and incident on the spectroscopic camera 10 to the spectroscopic element 12 and the image sensor 13. Note that, although Fig. 1 illustrates a plurality of lenses that constitute an incident optical system as the optical lens system 11, for example, one or more lenses that constitute an imaging optical system may be provided between the spectroscopic element 12 and the image sensor 13, and various other lenses that constitute a telecentric optical system may also be provided.

[0015] The spectroscopic element 12 receives the incident light guided by the optical lens system 11 and transmits light having a predetermined spectral wavelength as its center. Note that, in this embodiment, an example is shown in which the spectroscopic element 12 transmits light having a desired spectral wavelength as its center toward the image sensor 13, but the spectroscopic element 12 may be configured to reflect light having a desired spectral wavelength as its center toward the image sensor 13. The spectroscopic element 12 is an element capable of switching the spectral wavelength of light transmitted toward the image sensor 13, and may be, for example, a wavelength-tunable Fabry-Perot etalon. A Fabry-Perot etalon is an element in which a pair of mirrors are arranged opposite each other, and incident light is multiple-reflected between these mirrors, causing reinforced interference and transmitting light of a predetermined spectral wavelength. In such a Fabry-Perot etalon, the spectral wavelength can be switched by changing the gap between the mirrors using an actuator element such as an electrostatic actuator. The spectroscopic element 12 is not limited to the Fabry-Perot etalon as described above, and may be, for example, an AOTF (Acousto-Optic Tunable Filter) or an LCTF (Liquid Crystal Tunable Filter).

[0016] The imaging element 13 receives light centered on a desired spectral wavelength dispersed by the spectral element 12, and captures a spectral image. As the imaging element 13, a general image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) can be used.

[0017] The camera control unit 14 is configured to include, for example, a spectroscopic control circuit that controls the spectroscopic element 12, an imaging control circuit that controls the imaging element 13, a microcomputer that controls the overall operation of the spectroscopic camera 10, and a camera memory that stores various data. The spectroscopic control circuit outputs a predetermined drive signal to the spectroscopic element 12 under the control of the microcomputer, and switches the spectroscopic wavelengths to be separated by the spectroscopic element 12 . The imaging control circuit drives the imaging element 13 under the control of the microcomputer, and outputs an image signal to the microcomputer according to the amount of light received by each pixel. The microcomputer controls the spectroscopic control circuit to switch the spectroscopic wavelength of the spectroscopic element 12, and controls the imaging control circuit to acquire an image signal from the imaging element 13. Then, based on the image signal input from the imaging element 13, it generates spectroscopic image data. The camera memory records various data for controlling the spectroscopic element 12 and the like.

[0018] Furthermore, the camera memory may store a sensitivity characteristic spectrum (hereinafter simply referred to as sensitivity characteristic) of the spectroscopic camera 10. The sensitivity characteristic of the spectroscopic camera 10 is spectral data indicating sensitivity to each wavelength when the spectroscopic camera 10 captures a spectral image for a predetermined spectral wavelength. This sensitivity characteristic is a value obtained by multiplying the optical lens characteristic of the optical lens system 11, the spectral characteristic of the spectroscopic element 12, and the imaging sensitivity characteristic of the imaging element 13. Note that in the case where the spectroscopic camera system irradiates an imaging target with illumination light and captures an image of the reflected light, the sensitivity characteristic is a value obtained by multiplying the optical lens characteristic of the optical lens system 11, the spectral characteristic of the spectroscopic element 12, the imaging sensitivity characteristic of the imaging element 13, and the spectrum (emission spectrum) of the illumination light. Here, the optical characteristic of the optical lens system 11 is the transmittance of light passing through the optical lens system 11 for each wavelength. The spectral characteristics of the spectroscopic element 12 are the transmittance for each wavelength when light of a spectral wavelength Λ is transmitted through the spectroscopic element 12. In this embodiment, the spectroscopic element 12 can switch between a plurality of spectral wavelengths Λ. That is, assuming that the number of switchable spectral wavelengths Λ is K, the spectroscopic element 12 can switch between spectral wavelengths Λ1 to Λ2. K Therefore, the spectroscopic element 12 can select K wavelengths up to each spectroscopic wavelength Λ i It has K spectral characteristics depending on The imaging sensitivity characteristic of the imaging element 13 indicates the sensitivity of the imaging element 13 to each wavelength.

[0019] [Configuration of image processing device 20] The image processing device 20 can be configured by a general computer such as a smartphone, a tablet terminal, or a personal computer, and includes at least a storage unit 21 and one or more processors 22, as shown in FIG.

[0020] The storage unit 21 is an information storage device configured with a memory, a hard disk, and the like. The information stored in the storage unit 21 includes various programs including an image processing program for performing image processing by the image processing device 20, and various data used when executing the image processing program and other various programs. The various data include, for example, spectroscopic image data, filter data, sensitivity characteristics of the spectroscopic camera 10, and correction constants.

[0021] The spectroscopic image data is image data of a spectroscopic image captured by the spectroscopic camera 10. In this embodiment, the spectroscopic wavelength is switched for the imaging target, and spectroscopic images are captured for each of a plurality of spectroscopic wavelengths. Therefore, spectroscopic image data for a plurality of spectroscopic wavelengths for the same imaging target are recorded in association with each other by an image ID or the like.

[0022] As described above, the sensitivity characteristics of the spectroscopic camera 10 are spectral data obtained by multiplying the optical lens characteristics of the optical lens system 11, the spectral characteristics of the spectroscopic element 12, and the imaging sensitivity characteristics of the imaging element 13. When a single spectroscopic camera 10 is used in a spectroscopic camera system, it is sufficient to record the sensitivity characteristics of the spectroscopic camera 10. Furthermore, when the spectroscopic cameras 10 are detachable in the spectroscopic camera system and any spectroscopic camera 10 can be selected, the sensitivity characteristics of each spectroscopic camera 10 may be recorded.

[0023] The filter data contains spectral data of the target color filters used to form a color composite image from the spectral spectrum. This filter data is used to calculate correction constants. For example, when generating an RGB color image from the spectral spectrum that is similar to an image of an object captured using RGB color filters, correction constants are required to convert the spectral spectrum into R, G, and B color conversion values. In this case, the transmittance spectrum of the R color filter, the transmittance spectrum of the G color filter, and the transmittance spectrum of the B color filter are stored in the storage unit 21 as filter data used to calculate the correction constants. It is also possible to record multiple filter data corresponding to multiple color systems. For example, color filters for the RGB color system, color filters for the XYZ color system, and color filters for the Lab color system may be recorded. In this case, the user can select the desired color system.

[0024] The correction constant is a constant used to calculate the correction value. As described above, in this embodiment, the optical spectrum is converted into color conversion values ​​corresponding to any color filter, and a color composite image is generated that is similar to an image captured by capturing an image of the target using the color filter. Therefore, the correction constant is set for each color filter and each wavelength. For example, when generating a color composite image to be a color image using RGB color filters, a correction constant for each wavelength corresponding to the R filter, a correction constant for each wavelength corresponding to the G filter, and a correction constant for each wavelength corresponding to the B filter are recorded.

[0025] The storage unit 21 may also store the sensitivity characteristics of the spectroscopic camera 10 incorporated into the spectroscopic camera system.

[0026] The processor 22 functions as a data acquisition unit 221, a camera characteristic acquisition unit 222, a filter selection unit 223, a constant calculation unit 225, a correction value calculation unit 224, and an image synthesis unit 226 by reading and executing the programs stored in the memory unit 21. The data acquisition unit 221 acquires from the storage unit 21 a data cube including a spectral spectrum based on luminance values ​​at the same pixel position in a plurality of spectral image data corresponding to each of a plurality of spectral wavelengths. That is, the data acquisition unit 221 acquires data including a pixel position, a spectral wavelength, and a luminance value for the spectral wavelength at the pixel position. Specifically, the data acquisition unit 221 may acquire spectral image data corresponding to each of a plurality of spectral wavelengths, may acquire the luminance value of each pixel (x, y) of each spectral image data, or may acquire the change in luminance value of a line region along the X direction for each position in the Y direction. In either case, it is possible to obtain the luminance value (spectral spectrum) of each wavelength for the same pixel. The camera characteristic acquisition unit 222 acquires the sensitivity characteristic of the spectroscopic camera 10. The camera characteristic acquisition unit 222 may acquire the sensitivity characteristic stored in the camera memory of the spectroscopic camera 10, or may acquire the sensitivity characteristic input by the user.

[0027] The filter selection unit 223 selects a color filter when generating a color composite image from a plurality of spectral image data, and also selects a target color filter when calculating a correction constant. In this embodiment, a configuration is exemplified in which the user can select any color filter, but if color filters of a single color system are used, the filter selection unit 223 may not be necessary.

[0028] The constant calculation unit 225 calculates the correction constant and stores it in the storage unit 21. Although details will be described later, in this embodiment, the constant calculation unit 225 calculates the correction constant using, for example, the sensitivity characteristics of the spectroscopic camera 10 and arbitrary filter data.

[0029] The correction value calculation unit 224 reads out correction constants corresponding to the sensitivity characteristics of the spectroscopic camera 10 and corresponding to the color filters of the target color system from the storage unit 21. Then, the correction value calculation unit 224 calculates correction values ​​by correcting the signal values ​​of each pixel of the multiple pieces of spectroscopic image data for the imaging target with the corresponding correction constants.

[0030] The image synthesis unit 226 calculates a color conversion value that is the sum of the correction values ​​calculated by the correction value calculation unit 224, and generates a color synthesis image.

[0031] [Image processing method] Next, an image processing method for generating a color composite image in the spectroscopic camera system of this embodiment will be described. (Calculation of correction constant) In the spectroscopic camera system of this embodiment, correction constants are recorded in advance in the storage unit 21. In this embodiment, the image processing device 20 is capable of calculating these correction constants, and a method for calculating the correction constants will be described below.

[0032] Fig. 2 is a flowchart showing a method for calculating the correction constants, and Fig. 3 is a conceptual diagram showing an example of a method for calculating the correction constants. First, the camera characteristic acquisition unit 222 of the image processing device 20 acquires the sensitivity characteristic of the spectroscopic camera 10 incorporated in the spectroscopic camera system (step S1). The sensitivity characteristics of the spectroscopic camera 10 can be obtained by, for example, reading the sensitivity characteristics if they are recorded in the camera memory or storage unit 21 of the camera control unit 14 of the spectroscopic camera 10 as described above.

[0033] The sensitivity characteristics of the spectroscopic camera 10 are usually measured during manufacturing and inspection of the spectroscopic camera 10 . For example, the spectroscopic element 12 is set to a predetermined spectroscopic wavelength Λ i , laser light of a single wavelength is incident on the spectroscopic element 12, and the light intensity of the transmitted light is measured. The wavelength λ of the laser light is changed in sequence within the wavelength range of the spectroscopic image captured by the spectroscopic camera 10, and the light intensity of the transmitted light is measured. i This operation is performed by the spectroscopic element 12, and the spectroscopic characteristics of each wavelength λ can be obtained by setting i (i=an integer from 1 to K) in order, the spectroscopic element 12 is switched to each spectroscopic wavelength Λ i The spectral characteristics of each of the cases where the setting is The optical lens characteristics of the optical lens system 11 can be obtained by, for example, irradiating a laser beam of a single wavelength onto the optical lens system 11 and measuring the light intensity of the transmitted light. The wavelength of the laser beam is sequentially changed within the wavelength range of the spectroscopic image captured by the spectroscopic camera 10, and the light intensity of the transmitted light is measured. Similarly, the imaging sensitivity characteristics of the imaging element 13 can be obtained by, for example, receiving laser light of a single wavelength at the imaging element 13 and measuring the intensity of the received light (output light-receiving signal). The wavelength of the laser light is sequentially changed within the wavelength range of the spectroscopic image captured by the spectroscopic camera 10, and the light-receiving signal is measured. The imaging sensitivity characteristics of the imaging element 13 can be measured for each pixel.

[0034] The sensitivity characteristics of the spectroscopic camera 10 are obtained by multiplying the spectral characteristics of the spectroscopic element 12, the optical lens characteristics of the optical lens system 11, and the imaging sensitivity characteristics of the imaging element 13. Therefore, the sensitivity characteristics of the spectroscopic camera 10 are determined by the spectral wavelength λ set by the spectroscopic element 12, as shown in FIG. iThe spectral data differs for each pixel, and the sensitivity characteristics are set for each pixel of the image sensor 13. Hereinafter, the spectroscopic element 12 is set to the spectroscopic wavelength Λ i The sensitivity characteristic when set to S i The subscript i indicates the spectral wavelength that can be switched by the spectroscopic element 12, and when the number of possible switches is K, i is an integer from 1 to K. The spectral characteristics of the spectroscopic element 12, the optical lens characteristics of the optical lens system 11, and the imaging sensitivity characteristics of the imaging element 13 are determined by the spectrum of wavelengths λ1 to λ2. K Therefore, the sensitivity characteristic S i (λ) can be expressed as a matrix with K rows and K columns, as shown in FIG. In addition, the spectral wavelength Λ i and wavelength λ i That is, Λ i =λ i It is preferable that:

[0035] Next, the filter selection unit 223 selects a color filter to be used when generating the color composite image, and reads the filter data from the storage unit 21 (step S2). The filter data read here becomes the target spectrum. If the color filters to be used are set in advance, the filter data for the set color filters can be read in. For example, when generating an RGB color composite image, the filter data for the R color filter, the filter data for the G color filter, and the filter data for the B color filter in the RGB color system are read in. Furthermore, if color filters corresponding to a plurality of color systems are recorded in the storage unit 21, the color system designated by the user's operation can be selected and the filter data for that system can be read. Although this embodiment shows an example in which filter data corresponding to one color system is read, filter data corresponding to multiple color systems may be read in. For example, filter data corresponding to the RGB color system and filter data corresponding to the Lab color system may be read in. 4 is an example of filter data, showing the filter data of a B color filter in an RGB color filter. As shown in FIG. 4, the filter data is spectral data showing the relationship between the wavelength of light transmitted through the filter and the light intensity of the transmitted light, and is divided into wavelengths λ1 to λ2. K It can be represented as a K-row, 1-column matrix consisting of K elements up to

[0036] Thereafter, the constant calculation unit 225 calculates a correction constant (step S3). In step S3, as shown in FIG. 3, the constant calculation unit 225 calculates a spectrum obtained by multiplying the sensitivity characteristic of the spectroscopic camera 10 obtained in step S1 by the correction constant as a characteristic spectrum, and calculates a plurality of spectral wavelengths Λ i A correction constant is calculated so that the total spectrum obtained by summing up the characteristic spectra of the filters coincides or substantially coincides with the filter data (see FIG. 4) obtained in step S2. For example, the constant calculation unit 225 uses the least squares method to calculate Σ({Σa i S i (λ)}-F(λ)) 2 The wavelength λ is chosen so that i Correction constant a for i Calculate. More specifically, as described above, K spectral wavelengths Λ i Sensitivity characteristics S i (λ) are the K wavelengths λ j The element S corresponding to i (λ j ), and the filter data F(λ) includes K wavelengths λ j The element F(λ j Therefore, the constant calculation unit 225 calculates the correction constant a , which is the minimum value of X shown in the following formula (1): i Calculate.

[0037]

number

[0038] After that, the constant calculation unit 225 calculates the calculated correction constant ai is stored in the storage unit 21 as appropriate (step S4). In this embodiment, the sensitivity characteristics of the spectroscopic camera 10 are set for each pixel, so the correction constant a i can also be calculated for each pixel. As a result, the correction constant a corresponding to the color filter of the color system selected in step S2 is i For example, as a correction constant for color image synthesis in the RGB color system, each wavelength λ for the R color filter is calculated. i Correction constant a for i , G color filter for each wavelength λ i Correction constant a for i , and each wavelength λ for the B color filter i Correction constant a for i will be calculated. In step S2, filter data for a plurality of color systems may be selected as described above, and in this case, the processes of steps S3 and S4 may be performed for each color filter.

[0039] [Generating color composite images] Next, a method for generating a color composite image from a plurality of sets of spectral image data of an object captured by the spectroscopic camera 10 will be described. Fig. 5 is a flowchart showing a method for generating a color composite image, and Fig. 6 is a conceptual diagram showing an example of a method for generating a color composite image.

[0040] When a user performs an operation to command capturing of a spectral image, the spectroscopic camera 10 detects the spectral wavelength Λ i While switching sequentially, each spectral wavelength Λ i An image of the object is captured (step S11). Each spectral image data obtained by imaging is associated with, for example, wavelength data and an imaging ID. The wavelength data is the spectral wavelength Λ of the spectroscopic element 12 when the spectral image data is captured. i In addition, the data shows the multiple spectral wavelengths Λ obtained in one imaging process. iThe same imaging ID is assigned to each of the spectroscopic image data. The spectroscopic image data captured by the spectroscopic camera 10 is transmitted to the image processing device 20 and stored in the storage unit 21 of the image processing device 20 as appropriate.

[0041] Next, when the data acquisition unit 221 of the image processing device 20 receives an operation from the user to command the generation process of a color composite image, it acquires a data cube including the spectral spectrum of each pixel (e.g., spectral image data for multiple spectral wavelengths) from the memory unit 21 (step S12). For example, the data acquisition unit 221 reads from the storage unit 21 a plurality of sets of spectroscopic image data to which an imaging ID designated by the user is assigned.

[0042] Furthermore, the filter selection unit 223 selects a color filter to be used when synthesizing the color composite image (step S13). For example, the filter selection unit 223 may notify the user of selectable color filters to prompt the user to perform a selection operation, and may acquire the color filter selected by the user as the color filter to be used. Note that if there is only one color filter available, or if the color filter that the user will use has been fixed in advance, the process of step S13 may be skipped.

[0043] Then, the correction value calculation unit 224 calculates the spectral spectrum of each pixel based on the spectral image data read out in step S12 using the correction constant a corresponding to the color filter selected in step S13. i and calculates the correction value (step S14). Here, as mentioned above, Λ i =λ i If the spectral wavelength Λ i The brightness value P of each pixel of the spectral image data i and wavelength λ i The correction constant a corresponding to i Therefore, the correction value a i P i can be easily calculated. On the other hand, the spectral wavelength Λi and the wavelength λ, which is a spectral element i If different from the above, a spectrum is calculated from the luminance value of each pixel of each spectral image data, and the wavelength λ in the calculated spectrum is calculated. i The brightness value P corresponding to i and the wavelength λ i The correction constant a corresponding to i Using the correction value a i P i Calculate.

[0044] For example, when generating a color composite image in the RGB color system, the correction constant a corresponding to the R color filter is ri , the correction constant a corresponding to the G color filter gi , and the correction constant a corresponding to the B color filter bi Using this, the correction value (R correction value, G correction value, B correction value) = (a ri P i ,a gi P i ,a bi P i ) are K wavelengths λ i are calculated for each.

[0045] Then, the image synthesis unit 226 calculates a color conversion value by summing up the correction values ​​calculated in step S14 (step S15). Specifically, the image synthesis unit 226 calculates the correction value a for each color filter as shown in the following formula (2): i P i The color conversion value C is calculated for each pixel by summing up the above.

[0046]

number

[0047] As described above, when the correction values ​​(R correction value, G correction value, B correction value) are calculated, the R color conversion value C corresponding to RGB is calculated. R , G color conversion value C G , B color conversion value C B is calculated for each pixel. Thereafter, the image synthesis unit 226 generates a color synthesis image in which the parameters of each pixel are set based on these color conversion values ​​(step S16). For example, in a color synthesis image in the RGB color system, (R, G, B)=(C R ,C G ,C B ) to generate an RGB color composite image with the color parameters of each pixel set as

[0048] [Effects of this embodiment] The image processing device 20 of this embodiment is equipped with one or more processors 22, and the processors 22 function as a data acquisition unit 221, a correction value calculation unit 224, and an image synthesis unit 226 by reading and executing programs stored in the memory unit 21. In step S12, the data acquisition unit 221 acquires a plurality of spectral image data corresponding to a plurality of spectral wavelengths from the storage unit 21. In step S14, the correction value calculation unit 224 calculates a correction constant a set for each wavelength for the spectral spectrum of each pixel based on the luminance value of each pixel of the plurality of spectral image data. i The image synthesis unit 226 calculates a color conversion value by summing the correction values ​​at the same pixel position in the plurality of spectral image data, and generates a color synthesis image based on the color conversion value of each pixel. In this embodiment, the sensitivity characteristics of the spectroscopic camera and the wavelengths λ i The correction constant a corresponding to i As a characteristic spectrum obtained by multiplying i The correction constant a for each wavelength is set so that the total spectrum obtained by summing the characteristic spectra corresponding to each wavelength matches the target spectrum, which is the spectrum of an arbitrary color filter. i is set.

[0049] In this embodiment, the correction constants are set based on the sensitivity characteristics of the spectroscopic camera 10 so that any color filter can be realized, thereby improving the color reproducibility when a color composite image is generated. That is, the spectroscopic element 12 is configured to emit light at a desired spectral wavelength Λ iIdeally, only light with a spectral wavelength of Λ would be transmitted. i It is difficult to block all light other than the spectroscopic wavelength Λ, and even if it were possible, the amount of light transmitted through the spectroscopic element 12 would be significantly small. For this reason, in a spectroscopic camera 10 that captures a spectroscopic image, the spectroscopic wavelength Λ is usually i The spectroscopic camera 10 is configured to transmit light with a predetermined half-width centered on the center of the spectrum. However, in this case, the half-width varies depending on the performance of the spectroscopic camera 10. Therefore, if a spectrum based on spectroscopic image data is multiplied by a filter function corresponding to an arbitrary color filter, as in the past, the color reproducibility of the generated color composite image will vary depending on the performance of the spectroscopic camera 10. In contrast, in this embodiment, correction constants are set according to the sensitivity characteristics of the spectroscopic camera so that a spectrum similar to that obtained when light incident on the spectroscopic camera is input to an arbitrary color filter is obtained. Therefore, a color composite image with high color reproducibility can be generated regardless of the performance of the spectroscopic camera 10.

[0050] In the image processing device 20 of this embodiment, the processor 22 further functions as a camera characteristics acquisition unit 222, a filter selection unit 223, and a constant calculation unit 225. The camera characteristics acquisition unit 222 acquires a plurality of spectral wavelengths Λ of the spectroscopic camera 10. i The filter selection unit 223 acquires the target spectrum. The constant calculation unit 225 calculates the correction constant a so that the total spectrum matches the target spectrum. i Calculate. This allows the image processing device 20 to calculate correction constants according to the individual spectroscopic cameras 10 incorporated in the spectroscopic camera system. For example, even if the spectroscopic camera 10 is replaced, the image processing device 20 can calculate correction constants according to the replaced spectroscopic camera.

[0051] The constant calculation unit 225 of the image processing device 20 of this embodiment calculates the target spectrum as F(λ) and the wavelength λ i The correction constant for a i and the spectroscopic camera 10 detects the spectral wavelength Λ iThe sensitivity characteristics when capturing the spectral image data are i (λ), then Σ({Σa i S i (λ)}-F(λ)) 2 The correction constant a that minimizes i Calculate. This makes it possible to appropriately calculate the correction constants that can reproduce the target spectrum.

[0052] [Second embodiment] In the above embodiment, the constant calculation unit 225 of the image processing device 20 calculates the correction constant a corresponding to the sensitivity characteristic of the spectroscopic camera 10. i However, the correction constant may be stored in the storage unit 21 in advance. In the second embodiment, an example will be shown in which correction constants corresponding to a plurality of spectroscopic cameras 10 that can be incorporated into a spectroscopic camera system are stored in advance in a storage unit. In the following description, the same reference numerals will be used to designate components that have already been described, and their description will be omitted or simplified.

[0053] FIG. 7 is a schematic diagram showing a schematic configuration of a spectroscopic camera system according to the second embodiment. Similar to the first embodiment, the spectroscopic camera system of this embodiment includes a spectroscopic camera 10 and an image processing device 20. In this embodiment, the spectroscopic camera 10 is replaceable, and the sensitivity characteristics of the spectroscopic camera 10 that can be used are stored in advance in a storage unit 21. The storage unit 21 also stores correction constants a corresponding to the sensitivity characteristics of the spectroscopic camera 10. i is stored in advance in the storage unit 21. 7, the processor 22 functions as a data acquisition unit 221, a camera characteristics acquisition unit 222, a filter selection unit 223, a correction value calculation unit 224, and an image synthesis unit 226. In other words, the function of the constant calculation unit 225 for calculating the correction constant is not required.

[0054] In this embodiment, a color composite image is generated in substantially the same manner as in the first embodiment, but in this embodiment, after steps S11 to S13 are performed, the camera characteristic acquisition unit 222 acquires the sensitivity characteristics of the spectroscopic camera 10. Then, in step S14, the correction value calculation unit 224 calculates the correction constant a i Among these, the correction constant a corresponds to the sensitivity characteristic of the spectroscopic camera 10 and corresponds to the color filter selected in step S13. i is read out and the correction value is calculated. Thereafter, similarly to the first embodiment, the processes of steps S15 and S16 are carried out to generate a color composite image.

[0055] [Effects of this embodiment] In this embodiment, the same effects as those of the first embodiment can be achieved, and further, the following effects can be achieved. In the spectroscopic camera system of this embodiment, the correction constants are calculated in advance at the time of shipping from the factory and are recorded in the storage unit 21. This eliminates the need to calculate correction constants, and the configuration of the image processing device 20 can be simplified.

[0056] In the spectroscopic camera system of this embodiment, the storage unit 21 stores the sensitivity characteristics of a plurality of spectroscopic cameras 10 and the correction constants a for each sensitivity characteristic. i The camera characteristic acquisition unit 222 of the processor 22 acquires the sensitivity characteristic of the spectroscopic camera 10 incorporated in the spectroscopic camera system, and the correction value calculation unit 224 calculates the correction constant a corresponding to the acquired sensitivity characteristic. i is read from the storage unit 21 and a correction value is calculated. This allows the spectroscopic camera 10 to be detachably attached to the spectroscopic camera system, and even when a different spectroscopic camera 10 is attached, there is no need to calculate a new correction constant, and the correction value can be easily calculated simply by reading the correction constant corresponding to the attached spectroscopic camera 10. In addition, even when a plurality of spectroscopic cameras 10 are connected to the image processing device 20, the correction constant a corresponding to the spectroscopic camera 10 that captured the spectroscopic image is used. i It is possible to generate a color composite image with high color reproducibility corresponding to the spectroscopic camera 10 used.

[0057] [Variations] The present invention is not limited to the above-described embodiments, and the present invention includes modifications, improvements, and configurations obtained by appropriately combining the embodiments within the scope that can achieve the object of the present invention.

[0058] (Variation 1) In the above embodiment, a configuration including the spectroscopic camera 10 and the image processing device 20 has been exemplified as the spectroscopic camera system, but the spectroscopic camera 10 and the image processing device 20 may be configured as an integrated unit. In this case, there is no need to store the sensitivity characteristics of multiple spectroscopic cameras 10 in the storage unit 21, and as long as the sensitivity characteristic of only one spectroscopic camera 10 is recorded, the correction constant corresponding to that spectroscopic camera 10 can be calculated. Furthermore, as long as the correction constant corresponding to the sensitivity characteristic of that spectroscopic camera 10 is calculated and recorded in the storage unit 21 at the time of factory shipment, the sensitivity characteristic does not need to be recorded in the storage unit 21.

[0059] (Variation 2) In the above embodiment, the correction constant a for each pixel is calculated based on the sensitivity characteristics of each pixel of the spectroscopic camera 10. i However, if the sensitivity characteristics of each pixel are uniform, a common correction constant may be set regardless of the pixel position.

[0060] (Variation 3) In the above embodiment, the sensitivity characteristics are recorded in the camera memory of the spectroscopic camera 10, but the present invention is not limited to this. For example, the sensitivity characteristics may be stored in the storage unit 21 or in another external device that can communicate with the spectroscopic camera system. An example of the other external device is a data server provided by the manufacturer of the spectroscopic camera 10, which records a camera ID such as a serial number that can identify the spectroscopic camera 10 and the sensitivity characteristics of the spectroscopic camera 10. The image processing device 20 may then transmit the camera ID of the spectroscopic camera 10 incorporated in the spectroscopic camera system to the external device via the Internet or the like, and download the sensitivity characteristics of the corresponding spectroscopic camera 10 from the external device.

[0061] The spectroscopic camera system may also incorporate a sensitivity measurement unit that measures the sensitivity characteristics of the spectroscopic camera 10. The sensitivity measurement unit may include, for example, a laser light source that can change the wavelength of the laser light, and may change the wavelength of the laser light from λ1 to λ K , and the wavelength λ i This is repeated while sequentially changing the spectral wavelength Λi set by the spectral element 12, thereby measuring the light intensity of each spectral wavelength Λ i The sensitivity characteristics to

[0062] Summary of this disclosure An image processing method according to a first aspect of the present disclosure is an image processing method that converts spectral image data of a plurality of spectral wavelengths captured by a spectroscopic camera into a color image using one or more processors, the method comprising causing the one or more processors to: acquire a plurality of pieces of spectral image data corresponding to each of the plurality of spectral wavelengths from a storage unit; calculate a correction value by multiplying a spectral spectrum based on the luminance value of each pixel of the plurality of pieces of spectral image data by a correction constant set for each wavelength; calculate a color conversion value by summing the correction values ​​for the same pixel position in the plurality of pieces of spectral image data; and generate a color composite image based on the color conversion value for each pixel; wherein a characteristic spectrum is a spectrum obtained by multiplying the sensitivity characteristic spectrum of the spectroscopic camera for the spectral wavelengths by the correction constant corresponding to each wavelength; and a total spectrum is a spectrum obtained by summing the characteristic spectra corresponding to the plurality of spectral wavelengths, the correction constant is set so that the total spectrum matches a target spectrum, which is the spectrum of an arbitrary color filter.

[0063] This allows the correction constants to be set so that any color filter can be realized based on the sensitivity characteristics of the spectroscopic camera, making it possible to generate a color composite image with high color reproducibility regardless of the performance of the spectroscopic camera.

[0064] In the image processing method of this aspect, the one or more processors are further caused to acquire the sensitivity characteristic spectra for the plurality of spectral wavelengths in the spectroscopic camera, acquire the target spectrum, and calculate the correction constants corresponding to each wavelength so that the total spectrum matches the target spectrum. This makes it possible to calculate a correction constant suitable for the spectroscopic camera that captures the spectroscopic image. For example, even if the spectroscopic camera is replaced, it is possible to calculate a correction constant suitable for the replaced spectroscopic camera.

[0065] In the image processing device of this aspect, the target spectrum is F(λ), and the wavelength λ iThe correction constant for a i and the spectroscopic wavelength Λ i The sensitivity characteristic when capturing the spectral image data is S i (λ), the correction constant a i is Σ({Σa i S i (λ)}-F(λ)) 2 is the minimum value. This makes it possible to appropriately calculate the correction constants that can reproduce the target spectrum.

[0066] a correction value calculation unit that calculates a correction value by multiplying a spectral spectrum based on a luminance value of each pixel of the plurality of spectral image data by a correction constant set for each wavelength; and an image synthesis unit that calculates a color conversion value by summing the correction values ​​for the same pixel position in the plurality of spectral image data, and generates a color composite image based on the color conversion value for each pixel, wherein a characteristic spectrum is defined as a spectrum obtained by multiplying the sensitivity characteristic spectrum of the spectral camera for the spectral wavelength by the correction constant corresponding to each wavelength, and a total spectrum is defined as a spectrum obtained by integrating each of the characteristic spectra corresponding to the plurality of spectral wavelengths, and the correction constant is set so that the total spectrum matches a target spectrum, which is the spectrum of an arbitrary color filter.

[0067] As a result, as in the first aspect, the correction constants are set so that any color filter can be realized based on the sensitivity characteristics of the spectroscopic camera, thereby realizing a spectroscopic camera system that can generate color composite images with high color reproducibility regardless of the performance of the spectroscopic camera.

[0068] In the spectroscopic camera system of this aspect, the correction constants are pre-recorded in the storage unit. In this case, there is no need to calculate the correction constants each time a spectroscopic image is captured by the spectroscopic camera. Alternatively, the correction constants may be pre-recorded in the storage unit at the time of factory shipment. In this case, there is no need to calculate the correction constants separately in the spectroscopic camera system, which simplifies the configuration.

[0069] In the spectroscopic camera system of this aspect, the memory unit pre-stores the sensitivity characteristic spectra of a plurality of the spectroscopic cameras, each having a different sensitivity characteristic spectrum, and the correction constant for each of the sensitivity characteristic spectra, and the system further includes a camera characteristic acquisition unit that acquires the sensitivity characteristic spectrum of the spectroscopic camera, and the correction value calculation unit reads out the correction constant corresponding to the acquired sensitivity characteristic spectrum from the memory unit and calculates the correction value.

[0070] According to this aspect, the correction value can be calculated by selecting a correction constant that corresponds to the sensitivity characteristics of the spectroscopic camera that captured the spectroscopic image from multiple correction constants stored in the memory unit.Therefore, there is no need to calculate a correction constant each time a spectroscopic image is captured with the spectroscopic camera.If the correction constants are recorded in the memory unit in advance at the time of factory shipment, there is no need to calculate the correction constants, which simplifies the configuration.

[0071] The spectroscopic camera system of this aspect further includes a camera characteristic acquisition unit that acquires the sensitivity characteristic spectra for the plurality of spectroscopic wavelengths in the spectroscopic camera, and a constant calculation unit that calculates the correction constant corresponding to each wavelength so that the total spectrum matches the target spectrum. According to this aspect, it is possible to calculate a correction constant corresponding to the spectroscopic camera used in the spectroscopic camera system. Therefore, even when the spectroscopic camera is replaced, it is possible to calculate an appropriate correction constant, thereby generating a color composite image with high color reproducibility. [Explanation of symbols]

[0072] 10...spectroscopic camera, 11...optical lens system, 12...spectroscopic element, 13...imaging element, 14...camera control unit, 20...image processing device, 21...memory unit, 22...processor, 221...data acquisition unit, 222...camera characteristic acquisition unit, 223...filter selection unit, 224...correction value calculation unit, 225...constant calculation unit, 226...image synthesis unit, 234...constant calculation unit.

Claims

1. An image processing method for synthesizing a color image from spectral image data of a plurality of spectral wavelengths captured by a spectroscopic camera using one or more processors, comprising: the one or more processors; acquiring, from a storage unit, a data cube including a spectral spectrum based on luminance values ​​at the same pixel position in the plurality of spectral image data corresponding to each of the plurality of spectral wavelengths; multiplying the spectrum of each pixel by a correction constant for each wavelength in each pixel of a plurality of color filters corresponding to the color system of the color image to be synthesized, to calculate a correction value for each wavelength in each pixel of the plurality of color filters; summing the correction values ​​for each wavelength at the same pixel position in the plurality of spectral image data to calculate a color conversion value for each color filter, and generating a color composite image based on the color conversion value for each color filter of each pixel; and the spectral camera is configured to multiply a sensitivity characteristic spectrum for the spectral wavelengths by the correction constant corresponding to each wavelength to obtain a characteristic spectrum, and the characteristic spectrum corresponding to the plurality of spectral wavelengths is configured to obtain a total spectrum, the correction constant being set so that the total spectrum matches a target spectrum, which is a spectrum of the plurality of color filters corresponding to a color system of the color image to be synthesized.

2. the one or more processors; acquiring the sensitivity characteristic spectrum for the plurality of spectral wavelengths in the spectroscopic camera; obtaining the target spectrum; calculating the correction constant corresponding to each wavelength so that the sum spectrum coincides with the target spectrum; The image processing method according to claim 1 , further comprising:

3. The target spectrum is F(λ), wavelength λ ｉ The correction constant for a ｉ year, The spectroscopic camera detects the spectral wavelength Λ ｉ The sensitivity characteristic when capturing the spectral image data is S ｉ (λ), The correction constant a ｉ is Σ({Σa ｉ S ｉ (λ)}-F(λ) ２ is the value at which 3. The image processing method according to claim 1.

4. a spectroscopic camera that captures an image by separating incident light into light having a predetermined spectral wavelength at its center and that can change the spectral wavelength to a plurality of wavelengths; a storage unit that stores spectroscopic image data captured by the spectroscopic camera; a data acquisition unit that acquires, from the storage unit, a data cube including a spectral spectrum based on luminance values ​​at the same pixel position in the plurality of spectral image data corresponding to the plurality of spectral wavelengths; a correction value calculation unit that calculates a correction value for each wavelength in each pixel of a plurality of color filters by multiplying the optical spectrum of each pixel by a correction constant for each wavelength in each pixel of a plurality of color filters that corresponds to the color system of a color image to be synthesized; an image synthesis unit that calculates a color conversion value for each of the color filters by summing the correction values ​​for each wavelength at the same pixel position in the plurality of spectral image data, and generates a color synthesis image based on the color conversion value for each of the color filters in each pixel, a characteristic spectrum is defined as a spectrum obtained by multiplying a sensitivity characteristic spectrum for the spectral wavelengths in the spectroscopic camera by the correction constant corresponding to each wavelength, and a spectrum obtained by integrating the characteristic spectra corresponding to the plurality of spectral wavelengths is defined as a total spectrum, and the correction constant is set so that the total spectrum matches a target spectrum, which is a spectrum of the plurality of color filters corresponding to a color system of the color image to be synthesized.

5. The spectroscopic camera system according to claim 4 , wherein the correction constant is pre-recorded in the storage unit.

6. the storage unit stores in advance the sensitivity characteristic spectra of the plurality of spectroscopic cameras, each having a different sensitivity characteristic spectrum, and the correction constant for each of the sensitivity characteristic spectra; a camera characteristic acquisition unit that acquires the sensitivity characteristic spectrum of the spectroscopic camera, The spectroscopic camera system according to claim 5 , wherein the correction value calculation unit reads out the correction constant corresponding to the acquired sensitivity characteristic spectrum from the storage unit and calculates the correction value.

7. a camera characteristic acquisition unit that acquires the sensitivity characteristic spectrum for the plurality of spectral wavelengths in the spectroscopic camera; The spectroscopic camera system according to claim 4 , further comprising: a constant calculation unit that calculates the correction constant corresponding to each wavelength so that the total spectrum coincides with the target spectrum.

Images