Information processing device, information processing method, and program
Patent Information
- Application Number
- JP2024511193
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-09
AI Technical Summary
Existing information processing technologies lack the ability to perform color adjustment with high degrees of freedom in pseudo-coloring multiple spectral images, particularly in generating channel images and pseudo-color images that effectively utilize subtraction operations and alignment processing.
An information processing device and method that assigns multiple spectral images to different channels through assignment and subtraction processes, incorporating product-sum calculations and negative coefficients, while ensuring the pseudo-color image falls within the display medium's expression range, and includes polarization and wavelength information.
This approach enables high-degree color adjustment and enhanced wavelength information discrimination in pseudo-color images, avoiding color saturation and improving image quality by aligning and processing multispectral data from an imaging device with polarizers and filters.
Abstract
Description
Information processing device, information processing method, and program
[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program.
[0002] International Publication No. 2019 / 151029 discloses an imaging device that includes a white light source unit that irradiates an object with white light, an imaging unit that captures a multispectral image of the object, an object identification unit that identifies, from the multispectral image of the object irradiated with white light, a wavelength of light that is optimal for analyzing the object as an effective wavelength, and a variable wavelength light source unit that irradiates the object with light of the effective wavelength.
[0003] Japanese Patent Application Laid-Open No. 2018-098341 discloses an imaging element having a first pixel equipped with a metal thin film filter that transmits light in a first frequency band, and a second pixel equipped with a color filter that transmits light in a second frequency band that is broader than the first frequency band.
[0004] Japanese Patent Application Laid-Open Publication No. 2021-135404 discloses a lens device including an optical system, an optical element, an irradiation device, and a control unit. The optical system includes a lens that forms an optical image of a subject. The optical element is an optical element arranged at or near the pupil position of the optical system, and includes a frame having multiple aperture regions, multiple optical filters arranged in the multiple aperture regions, the multiple optical filters including two or more optical filters that transmit light of at least a portion of different wavelength bands, and multiple polarizing filters arranged in the multiple aperture regions, the multiple polarizing filters having different polarization directions. The irradiation device irradiates illumination light onto the subject. The control unit controls at least one of the optical system, the optical element, and the irradiation device. The control unit changes the spectral characteristics of the light emitted from the optical system for the multiple aperture regions.
[0005] Japanese Patent Laid-Open Publication No. 2010-025750 discloses an image processing device that processes a multiband image captured using multiple band-pass filters. The image processing device includes a secondary light source designation means that designates a secondary light source separate from a primary light source, a spectral estimation means that obtains a spectral image from the multiband image, and an image separation means that separates the spectral image obtained by the spectral estimation means into an image under the primary light source and an image under the secondary light source based on the spectrum of the secondary light source designated by the secondary light source designation means.
[0006] As an example, one embodiment of the technology disclosed herein provides an information processing device, an information processing method, and a program that can achieve highly flexible color adjustment in pseudo-coloring a plurality of spectral images, compared to when a channel image is generated based on an operation that includes only addition of a first spectral image among a plurality of different spectral images.
[0007] A first aspect of the technology of the present disclosure is an information processing device including a processor, wherein the processor performs an allocation process for allocating different spectral images to different channels to generate a channel image for each channel, and generates a first pseudo-color image based on the multiple channel images, and the allocation process includes a process for generating the channel image based on an operation including subtraction on a first spectral image of the multiple spectral images.
[0008] A second aspect of the technology of the present disclosure is the information processing device according to the first aspect, wherein the operation is a product-sum operation including a first spectral image, and the subtraction is realized by including a negative value in the coefficient of the first spectral image in the product-sum operation.
[0009] A third aspect of the technology of the present disclosure is an information processing device according to the second aspect, in which the coefficient is set to a value such that the range of the first pseudo-color image falls within the representation area of the display medium on which the first pseudo-color image is displayed.
[0010] A fourth aspect of the technology of the present disclosure is an information processing device according to any one of the first to third aspects, wherein the plurality of spectral images include polarization information and / or first wavelength information, and the number of the plurality of spectral images is equal to or greater than the number of channels.
[0011] A fifth aspect of the technology of the present disclosure is an information processing device according to any one of the first to fourth aspects, wherein the plurality of spectral images are images obtained by capturing images using an image sensor having a polarizer, and the processor performs alignment processing on the first spectral images, and the subtraction is performed on the first spectral images that have been subjected to the alignment processing.
[0012] A sixth aspect of the technology of the present disclosure is an information processing device according to any one of the first to fifth aspects, in which the first pseudo-color image is an image in which the discrimination of the second wavelength information is enhanced compared to the second pseudo-color image in which the calculation includes only addition.
[0013] A seventh aspect of the technology of the present disclosure is an information processing device according to any one of the first to sixth aspects, in which the processor divides an image generated based on a plurality of spectral images into a plurality of regions, and calculations are performed based on color information set for the plurality of regions.
[0014] An eighth aspect of the technology of the present disclosure is the information processing device according to any one of the first to seventh aspects, wherein the different channels are channels of three primary colors.
[0015] A ninth aspect of the technology of the present disclosure is an information processing device according to any one of the first to eighth aspects, wherein the processor outputs data for displaying a first pseudo-color image on a display device.
[0016] A tenth aspect of the technology of the present disclosure is an information processing method comprising: performing an allocation process for allocating different spectral images to different channels, thereby generating a channel image for each channel, and generating a first pseudo-color image based on the plurality of channel images, wherein the allocation process includes a process for generating the channel image based on an operation including subtraction on a first spectral image of the plurality of spectral images.
[0017] An eleventh aspect of the technology of the present disclosure is a program for causing a computer to execute a specific processing, the specific processing including generating a channel image for each channel by performing an assignment processing for assigning different spectral images to different channels, and generating a first pseudo-color image based on the multiple channel images, and the assignment processing including processing for generating a channel image based on an operation including subtraction on a first spectral image of the multiple spectral images.
[0018] 1 is a block diagram showing an example of a hardware configuration of an imaging device according to an embodiment. FIG. 1 is an exploded perspective view showing an example of a photoelectric conversion element according to an embodiment. FIG. 2 is a block diagram showing an example of a functional configuration for realizing image display processing according to an embodiment. FIG. 2 is a block diagram showing an example of the operation of an output value acquisition unit and an interference removal processing unit according to an embodiment. FIG. 3 is a block diagram showing an example of the operation of a position alignment processing unit according to an embodiment. FIG. 3 is a block diagram showing an example of the operation of an initial image generation unit according to an embodiment. FIG. 4 is a block diagram showing an example of the operation of an initial image output unit according to an embodiment. FIG. 5 is a block diagram showing an example of the operation of an area setting determination unit and an area setting unit according to an embodiment. FIG. 6 is a block diagram showing an example of the operation of a color setting determination unit and a gain setting unit according to an embodiment. FIG. 7 is a block diagram showing an example of the operation of a pseudo color image generation unit according to an embodiment. FIG. 8 is a block diagram showing an example of the operation of a pseudo color image output unit according to an embodiment and a first example of a pseudo color image. FIG. 9 is a block diagram showing an example of the operation of a pseudo color image output unit according to an embodiment and a second example of a pseudo color image. FIG. 10 is a block diagram showing an example of the operation of a pseudo color image output unit according to an embodiment and a third example of a pseudo color image. FIG. 11 is a block diagram showing an example of the operation of a pseudo color image output unit according to an embodiment and a fourth example of a pseudo color image. FIG. 12 is a flowchart showing an example of the flow of image display processing according to an embodiment.
[0019] Hereinafter, exemplary embodiments of an information processing device, an information processing method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.
[0020] First, the terms used in the following description will be explained.
[0021] I / F is an abbreviation for "Interface". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". NVM is an abbreviation for "Non-Volatile Memory". RAM is an abbreviation for "Random Access Memory". CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". HDD is an abbreviation for "Hard Disk Drive". LiDAR is an abbreviation for "Light Detection and Ranging". TPU is an abbreviation for "Tensor processing unit". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". ASIC is an abbreviation for "Application Specific Integrated Circuit." FPGA is an abbreviation for "Field-Programmable Gate Array." PLD is an abbreviation for "Programmable Logic Device." SoC is an abbreviation for "System-on-a-chip." IC is an abbreviation for "Integrated Circuit."
[0022] In the description of this specification, "same" refers to the same in the sense of including, in addition to being completely the same, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, "orthogonal" refers to the same in the sense of including, in addition to being completely orthogonal, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, "straight line" refers to the same in the sense of including, in addition to being completely straight, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure.
[0023] 1 , an image capturing device 10 is a multispectral camera capable of outputting a pseudo-colored multispectral image, and includes an optical system 12, an image sensor 14, a control driver 16, an input / output I / F 18, a computer 20, a reception device 22, and a display 24. The image capturing device is an example of an “information processing device” according to the technology of the present disclosure.
[0024] The optical system 12 includes a first lens 26, a pupil division filter 28, and a second lens 30. The first lens 26, the pupil division filter 28, and the second lens 30 are arranged in this order along the optical axis OA of the imaging device 10, from the subject 4 side to the image sensor 14 side. The first lens 26 transmits light obtained by light emitted from the light source 2 being reflected by the subject 4 (hereinafter referred to as "subject light") through the pupil division filter 28. The second lens 30 forms an image of the subject light that has transmitted through the pupil division filter 28 on a light receiving surface 48A of a photoelectric conversion element 48 provided in the image sensor 14.
[0025] The pupil division filter 28 has a spectral filter 40 and a polarizing filter 42. The spectral filter 40 has filters 44A to 44H, and the polarizing filter 42 has polarizers 46A to 46H. For convenience, in Figure 1, the filters 44A to 44H are shown as being linearly arranged in a direction perpendicular to the optical axis OA, but the filters 44A to 44H are actually arranged in a direction around the optical axis OA.
[0026] The filter 44A has a first transmission wavelength band λ 1 and the filter 44B has a second transmission wavelength band λ 2 and the filter 44C has a third transmission wavelength band λ 3 and the filter 44D has a fourth transmission wavelength band λ 4 The filter 44E has a fifth transmission wavelength band λ 5 and the filter 44F has a sixth transmission wavelength band λ 6 and the filter 44G has a seventh transmission wavelength band λ 7 and the filter 44H has an eighth transmission wavelength band λ 8 It has the following characteristics.
[0027] First transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 are different wavelength bands. For example, in the example shown in FIG. 1, the first transmission wavelength band λ 1 is set to 435 nm, and the second transmission wavelength band λ 2 is set to 495 nm, and the third transmission wavelength band λ 3 is set to 555 nm, and the fourth transmission wavelength band λ 4 is set to 615 nm. 5 is set to 675 nm, and the sixth transmission wavelength band λ 6 is set to 735 nm, and the seventh transmission wavelength band λ 7 is set to 795 nm, and the eighth transmission wavelength band λ 8 The wavelength bands exemplified here are merely examples, and the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 may be set to any wavelength band, and preferably, they are different wavelength bands.
[0028] Hereinafter, when there is no need to distinguish between the filters 44A to 44H, the filters 44A to 44H will be referred to as "filters 44." 1 ~ 8th transmission wavelength band λ 8 When there is no need to distinguish between the first transmission wavelength band λ1 ~ 8th transmission wavelength band λ 8 is referred to as the "transmission wavelength band λ."
[0029] The polarizers 46A to 46H are respectively superimposed on the filters 44A to 44H. The polarizer 46A is a polarizer whose transmission axis angle is set to 0°, the polarizer 46B is a polarizer whose transmission axis angle is set to 20°, the polarizer 46C is a polarizer whose transmission axis angle is set to 40°, and the polarizer 46D is a polarizer whose transmission axis angle is set to 60°. The polarizer 46E is a polarizer whose transmission axis angle is set to 80°, the polarizer 46F is a polarizer whose transmission axis angle is set to 100°, the polarizer 46G is a polarizer whose transmission axis angle is set to 120°, and the polarizer 46H is a polarizer whose transmission axis angle is set to 140°. Hereinafter, when it is not necessary to distinguish between the polarizers 46A to 46H, the polarizers 46A to 46H will each be referred to as "polarizer 46."
[0030] 1, the number of filters 44 is eight, but the number of filters 44 may be any number as long as it is equal to or greater than the number of channels (see FIGS. 6 and 10) described below. Also, in the example shown in FIG. 1, the number of polarizers 46 is eight, but the number of polarizers 46 may be any number as long as it is the same as the number of filters 44.
[0031] The image sensor 14 includes a photoelectric conversion element 48 and a signal processing circuit 50. The image sensor 14 is, for example, a CMOS image sensor. In the present embodiment, a CMOS image sensor is used as the image sensor 14, but the technology of the present disclosure is not limited thereto. For example, the technology of the present disclosure can be applied even if the image sensor 14 is another type of image sensor, such as a CCD image sensor.
[0032] As an example, Fig. 1 shows a schematic configuration of the photoelectric conversion element 48. Also, as an example, Fig. 2 shows a specific configuration of a portion of the photoelectric conversion element 48. The photoelectric conversion element 48 has a pixel layer 52, a polarizing filter layer 54, and a spectral filter layer 56.
[0033] The pixel layer 52 has a plurality of pixels 58. The plurality of pixels 58 are arranged in a matrix and form the light receiving surface 48A of the photoelectric conversion element 48. Each pixel 58 is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light.
[0034] Hereinafter, the pixels 58 provided in the photoelectric conversion element 48 will be referred to as "physical pixels 58" to distinguish them from pixels that form a multispectral image. Also, the pixels that form the image displayed on the display 24 will be referred to as "image pixels."
[0035] The photoelectric conversion element 48 outputs the electrical signals output from the plurality of physical pixels 58 to the signal processing circuit 50 as imaging data 120. The signal processing circuit 50 digitizes the analog imaging data 120 input from the photoelectric conversion element 48.
[0036] The plurality of physical pixels 58 form a plurality of pixel blocks 60. Each pixel block 60 is formed by two vertical and two horizontal rows, for a total of four physical pixels 58. For convenience, in FIG. 1 , the four physical pixels 58 forming each pixel block 60 are shown as being linearly arranged in a direction perpendicular to the optical axis OA, but as shown in FIG. 2 as an example, the four physical pixels 58 are arranged adjacent to each other in the vertical and horizontal directions of the photoelectric conversion element 48.
[0037] The polarizing filter layer 54 has polarizers 62A to 62D. The polarizer 62A is a polarizer whose transmission axis angle is set to 0°. The polarizer 62B is a polarizer whose transmission axis angle is set to 45°. The polarizer 62C is a polarizer whose transmission axis angle is set to 90°. The polarizer 62D is a polarizer whose transmission axis angle is set to 135°. Hereinafter, when it is not necessary to distinguish between the polarizers 62A to 62D, the polarizers 62A to 62D will be referred to as "polarizer 62."
[0038] The spectral filter layer 56 has a B filter 64A, a G filter 64B, and an R filter 64C. The B filter 64A is a blue bandpass filter that transmits the most light in the blue wavelength band among the multiple wavelength bands of light. The G filter 64B is a green bandpass filter that transmits the most light in the green wavelength band among the multiple wavelength bands of light. The R filter 64C is a red bandpass filter that transmits the most light in the red wavelength band among the multiple wavelength bands of light. The B filter 64A, G filter 64B, and R filter 64C are assigned to each pixel block 60.
[0039] For convenience, Fig. 1 shows the B filter 64A, G filter 64B, and R filter 64C arranged in a line along a direction perpendicular to the optical axis OA, but as an example, as shown in Fig. 2, the B filter 64A, G filter 64B, and R filter 64C are arranged in a matrix in a predetermined pattern arrangement. In the example shown in Fig. 2, the B filter 64A, G filter 64B, and R filter 64C are arranged in a matrix in a Bayer array, which is an example of a predetermined pattern arrangement. Note that the predetermined pattern arrangement may be an RGB stripe array, an R / G checkerboard array, an X-Trans (registered trademark) array, a honeycomb array, or the like, in addition to the Bayer array.
[0040] Hereinafter, when there is no need to distinguish between the B filter 64A, the G filter 64B, and the R filter 64C, they will each be referred to as the "filter 64."
[0041] As an example, as shown in FIG. 1, the input / output I / F 18 is connected to a signal processing circuit 50, a control driver 16, a computer 20, a reception device 22, and a display 24.
[0042] The computer 20 includes a processor 70, an NVM 72, and a RAM 74. The processor 70 is an example of a "processor" according to the technology of the present disclosure. The processor 70 controls the entire imaging device 10. The processor 70 is, for example, an arithmetic processing device including a CPU and a GPU. The GPU operates under the control of the CPU and is responsible for executing image-related processing. Here, an arithmetic processing device including a CPU and a GPU is given as an example of the processor 70, but this is merely an example. The processor 70 may be one or more CPUs that integrate GPU functionality, or one or more CPUs that do not integrate GPU functionality. The processor 70, the NVM 72, and the RAM 74 are connected via a bus 76, which is connected to the input / output I / F 18.
[0043] The NVM 72 is a non-transitory storage medium that stores various parameters and programs. For example, the NVM 72 is a flash memory (e.g., an EEPROM). However, this is merely an example, and an HDD or the like may also be used as the NVM 72 in addition to the flash memory. The RAM 74 temporarily stores various information and is used as a work memory.
[0044] The processor 70 reads out necessary programs from the NVM 72 and executes the read programs in the RAM 74. The processor 70 controls the control driver 16 and the signal processing circuit 50 in accordance with the programs executed in the RAM 74. The control driver 16 controls the photoelectric conversion element 48 under the control of the processor 70.
[0045] The reception device 22 has, for example, a release button, a touch panel, and hard keys (all not shown), and receives instructions from a user, etc. The display 24 is, for example, a liquid crystal display, and displays various images.
[0046] As an example, as shown in FIG. 3 , an image display program 80 is stored in the NVM 72. The image display program 80 is an example of a "program" according to the technology of the present disclosure. The processor 70 reads the image display program 80 from the NVM 72 and executes the read image display program 80 on the RAM 74. The processor 70 executes image display processing for the imaging data 120 in accordance with the image display program 80 executing on the RAM 74. The image display processing is an example of a "specific processing" according to the technology of the present disclosure.
[0047] The image display process is realized by the processor 70 operating as an output value acquisition unit 82, an interference removal processing unit 84, a positioning processing unit 86, an initial image generation unit 88, an initial image output unit 90, an area setting determination unit 92, an area setting unit 94, a color setting determination unit 96, a gain setting unit 98, a pseudo-color image generation unit 100, and a pseudo-color image output unit 102 in accordance with the image display program 80. The image display process is started every time imaging data 120 is input from the image sensor 14 to the processor 70.
[0048] 4 , the output value acquisition unit 82 acquires an output value Y of each physical pixel 58 based on the imaging data 120 input from the image sensor 14 to the processor 70. The output value Y of each physical pixel 58 corresponds to the luminance value of each pixel included in the captured image 122 represented by the imaging data 120.
[0049] Here, the output value Y of each physical pixel 58 is a value including interference (i.e., crosstalk). 1 , second transmission wavelength band λ 2 , and the third transmission wavelength band λ 3 Since light of each transmission wavelength band λ is incident, the output value Y is 1 a value according to the amount of light, and a second transmission wavelength band λ 2 and the third transmission wavelength band λ 3 The value is a mixture of values according to the amount of light.
[0050] To acquire a multispectral image, the processor 70 needs to perform a process of separating and extracting values corresponding to each transmission wavelength band λ from the output value Y for each physical pixel 58, that is, an interference removal process that removes interference, on the output value Y. Therefore, in this embodiment, the interference removal processing unit 84 performs the interference removal process on the output value Y of each physical pixel 58 acquired by the output value acquisition unit 82.
[0051] Here, the interference removal process will be described. The output value Y of each physical pixel 58 includes the luminance values of red, green, and blue as components of the output value Y. The output value Y of each physical pixel 58 is expressed by equation (1).
[0052]
[0053] However, Y R is the red luminance value of the output value Y, and Y G is the luminance value of green in the output value Y, and Y B is the luminance value of blue in the output value Y.
[0054] The first to eighth spectral images 124A to 124H are images generated by performing interference removal processing on the captured images. The pixel value X of each image pixel included in the first to eighth spectral images 124A to 124H before pseudo-colorization, which will be described later, is a value within the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 The luminance value of each light is included as a component of the pixel value X. The pixel value X of each image pixel is expressed by equation (2).
[0055]
[0056] However, the brightness value X λ1 is the first transmission wavelength band λ of the pixel value X 1 is the brightness value of the light, and the brightness value X λ2 is the second transmission wavelength band λ of the pixel value X 2 is the brightness value of the light, and the brightness value X λ3 is the third transmission wavelength band λ of the pixel value X 3 is the brightness value of the light, and the brightness value X λ4is the fourth transmission wavelength band λ of the pixel value X 4 is the brightness value of the light.
[0057] In addition, the brightness value X λ5 is the fifth transmission wavelength band λ of the pixel value X 5 is the brightness value of the light, and the brightness value X λ6 is the sixth transmission wavelength band λ of the pixel value X 6 is the brightness value of the light, and the brightness value X λ7 is the seventh transmission wavelength band λ of the pixel value X 7 is the brightness value of the light, and the brightness value X λ8 is the eighth transmission wavelength band λ of the pixel value X 8 Hereinafter, the luminance value X λ1 ~ Brightness value X λ8 When there is no need to distinguish between the luminance value X λ1 ~ Brightness value X λ8 "Brightness value X λ "
[0058] When the interference matrix is A, the output value Y of each physical pixel 58 is expressed by equation (3).
[0059]
[0060] The interference matrix A is a matrix defined based on the spectrum of the subject light, the spectral transmittance of the first lens 26 , the spectral transmittance of the second lens 30 , the spectral transmittance of the plurality of filters 44 , and the spectral sensitivity of the image sensor 14 .
[0061] The interference cancellation matrix, which is the generalized inverse matrix of the interference matrix A, is defined as A + In this case, the pixel value X of each image pixel is expressed by equation (4).
[0062]
[0063] Interference cancellation matrix A + Similarly to the interference matrix A, the interference cancellation matrix A is a matrix defined based on the spectrum of the subject light, the spectral transmittance of the first lens 26, the spectral transmittance of the second lens 30, the spectral transmittance of the plurality of filters 44, and the spectral sensitivity of the image sensor 14. + is set by a user or the like and stored in advance in the NVM 72.
[0064] The interference cancellation processor 84 uses the interference cancellation matrix A stored in the NVM 72 + and the output value Y of each physical pixel 58 acquired by the output value acquisition unit 82. Then, the interference cancellation processing unit 84 uses the acquired interference cancellation matrix A + and the output value Y of each physical pixel 58, the pixel value X of each image pixel is calculated and output using equation (4).
[0065] Here, as described above, the pixel value X of each image pixel is the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 The brightness value of each light is included as a component of the pixel value X.
[0066] The first spectral image 124A of the captured image 122 is a first transmission wavelength band λ 1 The brightness value of the light X λ1 (i.e., the brightness values X λ1 The second spectral image 124B of the captured image 122 is an image based on the second transmission wavelength band λ 2 The brightness value of the light X λ2 (i.e., the brightness values X λ2 The third spectral image 124C of the captured image 122 is an image based on the third transmission wavelength band λ 3 The brightness value of the light X λ3 (i.e., the brightness values X λ3 The fourth spectral image 124D of the captured image 122 is an image based on the fourth transmission wavelength band λ 4 The brightness value of the light X λ4 (i.e., the brightness values X λ4 (Image based on ).
[0067] Furthermore, the fifth spectral image 124E of the captured image 122 has a fifth transmission wavelength band λ 5 The brightness value of the light X λ5 (i.e., the brightness values X λ5 The sixth spectral image 124F of the captured image 122 is an image based on the sixth transmission wavelength band λ 6 The brightness value of the light X λ6 (i.e., the brightness values Xλ6 The seventh spectral image 124G of the captured image 122 is an image based on the seventh transmission wavelength band λ 7 The brightness value of the light X λ7 (i.e., the brightness values X λ7 The eighth spectral image 124H of the captured image 122 is an image based on the eighth transmission wavelength band λ 8 The brightness value of the light X λ8 (i.e., the brightness values X λ8 Hereinafter, when there is no need to distinguish between the first to eighth spectral images 124A to 124H, the first to eighth spectral images 124A to 124H will be referred to as "spectral images 124."
[0068] In this way, the interference removal processing is performed by the interference removal processing unit 84, and the captured image 122 is filtered out by the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 The brightness value X of each light λ That is, the captured image 122 is separated into spectral images 124 for each of the transmission wavelength bands λ of the filters 44. The spectral images 124 are images obtained by capturing an image using the image sensor 14 having a polarizer 46 and a filter 44, and include polarization information corresponding to the polarizer 46 and wavelength information corresponding to the filter 44. The number of the spectral images 124 is equal to or greater than the number of channels (see FIGS. 6 and 10 ), which will be described later.
[0069] The plurality of spectral images 124 are an example of “different plurality of spectral images” and “first spectral image” according to the technology of the present disclosure. The wavelength information is an example of “first wavelength information” according to the technology of the present disclosure.
[0070] 5, the alignment processing unit 86 performs alignment processing on the multiple spectral images 124. The alignment processing includes, for example, processing to correct optical distortion and / or processing to geometrically correct distortion occurring during imaging. Processing to correct optical distortion includes, for example, processing such as distortion correction (e.g., correction of barrel aberration or pincushion aberration). Processing to geometrically correct distortion occurring during imaging includes, for example, processing such as keystone correction (i.e., projective transformation or affine transformation).
[0071] 6 , the initial image generation unit 88 performs an allocation process to allocate the R channel, G channel, and B channel to a plurality of different spectral images 124, thereby generating an R channel image 126A, a G channel image 126B, and a B channel image 126C for each of the R channel, G channel, and B channel. The initial image generation unit 88 then generates an initial image 128 by combining the generated R channel image 126A, G channel image 126B, and B channel image 126C.
[0072] In the allocation process by the initial image generation unit 88, the initial image generation unit 88 performs arithmetic processing on the multiple spectral images 124. In the arithmetic processing, an operation using a gain is performed. The operation is a product-sum operation including the multiple spectral images 124. That is, as shown in equation (5), image pixel components X 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 36 37 38 39 40 41 42 43 44 55 56 57 58 69 70 71 72 73 74 75 86 76 R is the first gain G set for the R channel R1 ~ 8th Gain G R8 and the transmission wavelength band λ 1 ~λ 8 Brightness value X λ1 ~X λ8 The first gain G R1 ~ 8th Gain G R8 is the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 are the gains corresponding to each of the above.
[0073]
[0074] Similarly, as shown in equation (6), the image pixel component X assigned to the G channel of each image pixel included in the multiple spectral images 124 is G is the first gain G set for the G channel G1 ~ 8th Gain G G8 and the transmission wavelength band λ 1 ~λ 8 Brightness value X λ1 ~X λ8 The first gain G G1 ~ 8th Gain G G8 is the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 are the gains corresponding to each of the above.
[0075]
[0076] Furthermore, as shown in equation (7), the image pixel component X assigned to the B channel among the image pixels included in the plurality of spectral images 124 is B is the first gain G set for the B channel B1 ~ 8th Gain G B8 and the transmission wavelength band λ 1 ~λ 8 Brightness value X λ1 ~X λ8 The first gain G B1 ~ 8th Gain G B8 is the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 are the gains corresponding to each of the above.
[0077]
[0078] Then, image pixel components X R , X G , and X B By calculating the above, the multiple spectral images 124 are assigned to the R channel, the G channel, and the B channel.
[0079] Table 1 shows the first gain G set for the R channel in the allocation process by the initial image generation unit 88. R1 ~ 8th Gain G R8 , the first gain G set for the G channel G1 ~ 8th Gain G G8 , and the first gain G set for the B channel B1 ~ 8th Gain G B8 An example is shown: The gains shown in Table 1 are for the initial image 128.
[0080]
[0081] As shown in Table 1, in the R channel, the fifth transmission wavelength band λ 5 The fifth gain G corresponding to R5 is set to "1", and the fifth transmission wavelength band λ 5 The remaining gains corresponding to the transmission wavelength bands other than λ5 An R channel image 126A containing only the R signal is obtained.
[0082] Similarly, in the G channel, the third transmission wavelength band λ 3 The third gain G corresponding to G3 is set to "1", and the third transmission wavelength band λ 3 The remaining gains corresponding to the transmission wavelength bands other than λ3 A G channel image 126B containing only the G color is obtained.
[0083] In addition, in the B channel, the first transmission wavelength band λ 1 The first gain G corresponding to B1 is set to "1", and the first transmission wavelength band λ 1 The remaining gains corresponding to the transmission wavelength bands other than λ1 A B channel image 126C containing only the B channel image 126C is obtained.
[0084] The initial image 128 generated by combining the R channel image 126A, G channel image 126B, and B channel image 126C thus obtained is a normal RGB image that has not been pseudocolorized. The initial image 128 is an example of an "image generated based on multiple spectral images" according to the technology of the present disclosure.
[0085] Hereinafter, when there is no need to distinguish between the R channel, G channel, and B channel, they will each be referred to as a "channel." Furthermore, when there is no need to distinguish between the R channel image 126A, G channel image 126B, and B channel image 126C, they will each be referred to as a "channel image 126."
[0086] 7 , the initial image output unit 90 outputs initial image data representing an initial image 128 generated by the initial image generation unit 88 to the display 24. The display 24 displays the initial image 128 based on the initial image data. In the example shown in FIG. 7 , the initial image 128 representing a person's left hand 140, a first object 144, a second object 146, and a background 148 is displayed on the display 24. The hand 140 includes blood vessels 142.
[0087] As an example, as shown in Figure 8, when the processor 70 displays the initial image 128 on the display 24, it sets the operating mode of the imaging device 10 to an area division mode in which a user or the like can divide the initial image 128 into multiple areas 132 through the reception device 22.
[0088] While the initial image 128 is displayed on the display 24, the user or the like issues an area division instruction to the reception device 22 to divide the initial image 128 into a plurality of areas 132. The area division instruction is issued to, for example, a touch panel or the like included in the reception device 22.
[0089] When the reception device 22 receives the area division instruction, it outputs area division instruction data indicating the area division instruction to the processor 70. The area setting determination unit 92 determines whether the area division instruction data has been input to the processor 70.
[0090] When the region setting determination unit 92 determines that region division instruction data has been input to the processor 70, the region setting unit 94 divides the initial image 128 into a plurality of regions 132 in accordance with the region division instruction data. The process of dividing the initial image 128 into a plurality of regions 132 uses various processes, such as a process of detecting boundaries between images included in the plurality of spectral images 124 and / or a process of dividing the plurality of regions 132 based on pre-stored spectroscopic characteristics.
[0091] As an example, in the example shown in FIG. 8 , the region division instructions include a first region instruction, a second region instruction, a third region instruction, a fourth region instruction, and a fifth region instruction. The first region instruction is an instruction to designate a region 132 (hereinafter also referred to as a “first region 132A”) corresponding to a portion of the hand 140 other than the blood vessels 142. The second region instruction is an instruction to designate a region 132 (hereinafter also referred to as a “second region 132B”) corresponding to the blood vessels 142 of the hand 140. The third region instruction is an instruction to designate a region 132 (hereinafter also referred to as a “third region 132C”) corresponding to a first object 144. The fourth region instruction is an instruction to designate a region 132 (hereinafter also referred to as a “fourth region 132D”) corresponding to a second object 146. The fifth region instruction is an instruction to designate a region 132 (hereinafter also referred to as a “fifth region 132E”) corresponding to the background 148.
[0092] 8 , the initial image 128 is divided into a plurality of regions 132 corresponding to the first region indication, the second region indication, the third region indication, the fourth region indication, and the fifth region indication. Specifically, the initial image 128 is divided into a first region 132A, a second region 132B, a third region 132C, a fourth region 132D, and a fifth region 132E. The first region 132A, the second region 132B, the third region 132C, the fourth region 132D, and the fifth region 132E can be divided because they reflect light of different wavelength bands. The plurality of regions 132 are an example of the "plurality of regions" according to the technology of the present disclosure.
[0093] 9 , when the initial image 128 is divided into a plurality of regions 132, the processor 70 sets the operation mode of the imaging device 10 to a color setting mode in which a user or the like can set a color for each region 132 through the reception device 22. When the operation mode of the imaging device 10 is set to the color setting mode, a color palette 134 from which a plurality of colors can be selected is displayed on the display 24.
[0094] With the color palette 134 displayed on the display 24, the user or the like issues a color setting instruction to the reception device 22 to set a color for each area 132. The color setting instruction includes an area designation instruction that designates an area 132 for which a color is to be set among the multiple areas 132, and a color designation instruction that designates a color to be set for the area 132 corresponding to the area designation instruction. The color setting instruction is issued to, for example, a touch panel (not shown) included in the reception device 22.
[0095] When the reception device 22 receives a color setting instruction, it outputs color setting instruction data indicating the color setting instruction to the processor 70. The color setting instruction data includes area information corresponding to the area designation instruction and color information corresponding to the color designation instruction. The color information is an example of "color information" according to the technology of the present disclosure. The color setting determination unit 96 determines whether color setting instruction data has been input to the processor 70.
[0096] When the color setting determination unit 96 determines that color setting instruction data has been input to the processor 70, the gain setting unit 98 sets a gain G for the multiple spectral images 124 in accordance with the color setting instruction data so that the area 132 specified by the area designation instruction is set to the color specified by the color designation instruction.
[0097] Here, a method for deriving the gain G will be described. The pixel value y of each image pixel included in the pseudo color image 130 (see FIG. 10) generated based on the plurality of spectral images 124 is expressed by equation (8).
[0098]
[0099] However, y R1 is the red luminance value of the first region 132A, and y R2 is the brightness value of red in the second region 132B, and y R3 is the luminance value of red in the third region 132C, and y R4 is the brightness value of red in the fourth region 132D, and y R5 is the luminance value of red in the fifth region 132E.
[0100] y G1 is the green luminance value of the first region 132A, and y G2 is the luminance value of green in the second region 132B, and y G3 is the green luminance value of the third region 132C, and y G4 is the green luminance value of the fourth region 132D, and y G5 is the luminance value of green in the fifth region 132E.
[0101] y B1 is the luminance value of blue in the first region 132A, and y B2 is the luminance value of blue in the second region 132B, and y B3 is the blue luminance value of the third region 132C, and y B4 is the blue luminance value of the fourth region 132D, and y B5 is the luminance value of blue in the fifth region 132E.
[0102] The average luminance value x of each region 132 is expressed by equation (9).
[0103] However, x λ1-1 is the first transmission wavelength band λ of the image pixels included in the first region 132A. 1 is the average brightness value of the light. λ1-2 is the first transmission wavelength band λ of the image pixels included in the second region 132B. 1 is the average brightness value of the light. λ1-3 is the first transmission wavelength band λ of the image pixels included in the third region 132C. 1 is the average brightness value of the light. λ1-4 is the first transmission wavelength band λ of the image pixels included in the fourth region 132D. 1 is the average brightness value of the light. λ1-5 is the first transmission wavelength band λ of the image pixels included in the fifth region 132E. 1 is the average brightness value of the light.
[0104] x λ2-1 is the second transmission wavelength band λ of the image pixels included in the first region 132A. 2 is the average brightness value of the light. λ2-2 is the second transmission wavelength band λ of the image pixels included in the second region 132B. 2 is the average brightness value of the light. λ2-3 is the second transmission wavelength band λ of the image pixels included in the third region 132C. 2 is the average brightness value of the light. λ2-4 is the second transmission wavelength band λ of the image pixels included in the fourth region 132D. 2 is the average brightness value of the light. λ2-5 is the second transmission wavelength band λ of the image pixels included in the fifth region 132E. 2 is the average brightness value of the light.
[0105] x λ3-1 is the third transmission wavelength band λ of the image pixels included in the first region 132A. 3 is the average brightness value of the light. λ3-2 is the third transmission wavelength band λ of the image pixels included in the second region 132B. 3 is the average brightness value of the light. λ3-3 is the third transmission wavelength band λ of the image pixels included in the third region 132C. 3 is the average brightness value of the light.λ3-4 is the third transmission wavelength band λ of the image pixels included in the fourth region 132D. 3 is the average brightness value of the light. λ3-5 is the third transmission wavelength band λ of the image pixels included in the fifth region 132E. 3 is the average brightness value of the light.
[0106] x λ4-1 is the fourth transmission wavelength band λ of the image pixels included in the first region 132A. 4 is the average brightness value of the light. λ4-2 is the fourth transmission wavelength band λ of the image pixels included in the second region 132B. 4 is the average brightness value of the light. λ4-3 is the fourth transmission wavelength band λ of the image pixels included in the third region 132C. 4 is the average brightness value of the light. λ4-4 is the fourth transmission wavelength band λ of the image pixels included in the fourth region 132D. 4 is the average brightness value of the light. λ4-5 is the fourth transmission wavelength band λ of the image pixel included in the fifth region 132E. 4 is the average brightness value of the light.
[0107] x λ5-1 is the fifth transmission wavelength band λ of the image pixels included in the first region 132A. 5 is the average brightness value of the light. λ5-2 is the fifth transmission wavelength band λ of the image pixels included in the second region 132B. 5 is the average brightness value of the light. λ5-3 is the fifth transmission wavelength band λ of the image pixels included in the third region 132C. 5 is the average brightness value of the light. λ5-4 is the fifth transmission wavelength band λ of the image pixels included in the fourth region 132D. 5 is the average brightness value of the light. λ5-5 is the fifth transmission wavelength band λ of the image pixels included in the fifth region 132E. 5 is the average brightness value of the light.
[0108] x λ6-1 is the sixth transmission wavelength band λ of the image pixels included in the first region 132A. 6is the average brightness value of the light. λ6-2 is the sixth transmission wavelength band λ of the image pixels included in the second region 132B. 6 is the average brightness value of the light. λ6-3 is the sixth transmission wavelength band λ of the image pixels included in the third region 132C. 6 is the average brightness value of the light. λ6-4 is the sixth transmission wavelength band λ of the image pixels included in the fourth region 132D. 6 is the average brightness value of the light. λ6-5 is the sixth transmission wavelength band λ of the image pixels included in the fifth region 132E. 6 is the average brightness value of the light.
[0109] x λ7-1 is the seventh transmission wavelength band λ of the image pixels included in the first region 132A. 7 is the average brightness value of the light. λ7-2 is the seventh transmission wavelength band λ of the image pixels included in the second region 132B. 7 is the average brightness value of the light. λ7-3 is the seventh transmission wavelength band λ of the image pixels included in the third region 132C. 7 is the average brightness value of the light. λ7-4 is the seventh transmission wavelength band λ of the image pixel included in the fourth region 132D. 7 is the average brightness value of the light. λ7-5 is the seventh transmission wavelength band λ of the image pixel included in the fifth region 132E. 7 is the average brightness value of the light.
[0110] x λ8-1 is the eighth transmission wavelength band λ of the image pixels included in the first region 132A. 8 is the average brightness value of the light. λ8-2 is the eighth transmission wavelength band λ of the image pixels included in the second region 132B. 8 is the average brightness value of the light. λ8-3 is the eighth transmission wavelength band λ of the image pixel included in the third region 132C. 8 is the average brightness value of the light. λ8-4 is the eighth transmission wavelength band λ of the image pixels included in the fourth region 132D. 8 is the average brightness value of the light.λ8-5 is the eighth transmission wavelength band λ of the image pixel included in the fifth region 132E. 8 is the average brightness value of the light.
[0111] The pixel value y of each image pixel included in the pseudo color image 130 is expressed by equation (10) using the gain G.
[0112] The gain G is the first gain G set for the R channel. R1 ~ 8th Gain G R8 and the first gain G set for the G channel G1 ~ 8th Gain G G8 and the first gain G set for the B channel. B1 ~ 8th Gain G B8 The first gain G R1 ~ 8th Gain G R8 is the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 The first gain G G1 ~ 8th Gain G G8 is the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 The first gain G B1 ~ 8th Gain G B8 is the first transmission wavelength band λ 1 ~ 8th transmission wavelength band λ 8 The gain G is expressed by the following equation (11).
[0113]
[0114] The pixel value y of each image pixel included in the pseudo color image 130 is set based on the area 132 and color specified by the area specification instruction and color specification instruction. The color of the area 132 not specified by the area specification instruction and color specification instruction is set based on the initial image 128.
[0115] Then, from equations (8) to (11), the first gain G R1 ~ 8th Gain G R8 and the first gain G G1 ~ 8th Gain GG8 and the first gain G B1 ~ 8th Gain G B8 is derived.
[0116] The gain G is derived by the above calculation method when the number of regions 132 divided by the region division instruction is equal to or less than the number of transmission wavelength bands λ. When the number of regions 132 is greater than the number of transmission wavelength bands λ, the gain G may be derived as an approximate solution by performing interpolation processing such as least squares sum. The gain G is set uniformly for multiple image pixels included in each spectral image 124, but may also be set individually for each image pixel included in each spectral image 124.
[0117] Table 2 shows an example of the gain G derived in accordance with a color setting instruction for setting the second region 132B to blue (hereinafter referred to as a "first color setting instruction") As shown in Table 2, the gain G derived in accordance with the first color setting instruction includes a negative gain in addition to a positive gain.
[0118]
[0119] 10 shows an example in which a pseudo color image 130 is generated based on the gain G set by the gain setting unit 98. The pseudo color image generating unit 100 performs an allocation process for allocating a plurality of different spectral images 124 to the R channel, G channel, and B channel, thereby generating an R channel image 126A, a G channel image 126B, and a B channel image 126C for each of the R channel, G channel, and B channel.
[0120] The R channel, G channel, and B channel are examples of "different channels" according to the technology of the present disclosure. The allocation process is an example of an "allocation process" according to the technology of the present disclosure. The R channel image 126A, the G channel image 126B, and the B channel image 126C are examples of "channel images" according to the technology of the present disclosure. The pseudocolor image 130 is an example of a "first pseudocolor image" according to the technology of the present disclosure.
[0121] In the allocation process by the pseudo color image generation unit 100, the pseudo color image generation unit 100 performs arithmetic processing on the multiple spectral images 124. In the arithmetic processing, the calculation is performed using the gain G set by the gain setting unit 98 described above. The calculation is a product-sum operation including the multiple spectral images 124.
[0122] That is, as shown in equation (5), the image pixel component X assigned to the R channel among the image pixels included in the plurality of spectral images 124 is R is the first gain G set for the R channel R1 ~ 8th Gain G R8 and the transmission wavelength band λ 1 ~λ 8 Brightness value X λ1 ~X λ8 It is calculated by summing the products of
[0123] Similarly, as shown in equation (6), the image pixel component X assigned to the G channel of each image pixel included in the multiple spectral images 124 is G is the first gain G set for the G channel G1 ~ 8th Gain G G8 and the transmission wavelength band λ 1 ~λ 8 Brightness value X λ1 ~X λ8 It is calculated by summing the products of
[0124] Furthermore, as shown in equation (7), the image pixel component X assigned to the B channel among the image pixels included in the plurality of spectral images 124 is B is the first gain G set for the B channel B1 ~ 8th Gain G B8 and the transmission wavelength band λ 1 ~λ 8 Brightness value X λ1 ~X λ8 It is calculated by summing the products of
[0125] As shown in Table 2, since the gain G includes a negative gain (i.e., a negative value) in addition to a positive gain, the calculation process includes a subtraction. In this way, the image pixel component X R , X G , and X B By calculating the above, the multiple spectral images 124 are assigned to the R channel, the G channel, and the B channel. The pseudo color image generating unit 100 then generates a pseudo color image 130 by combining the generated R channel image 126A, G channel image 126B, and B channel image 126C.
[0126] 11 , the pseudo-color image output unit 102 outputs pseudo-color image data representing the pseudo-color image 130 generated by the pseudo-color image generation unit 100 to the display 24. The pseudo-color image data is an example of "data for displaying a first pseudo-color image" according to the technology of the present disclosure. The display 24 displays the pseudo-color image 130 represented by the pseudo-color image data.
[0127] As an example, in the example shown in Fig. 11 , a pseudo color image 130 in which the second region 132B is set to blue is displayed on the display 24. Note that in the example shown in Fig. 11 , by setting the color of the second region 132B to blue, the color of the second region 132B is saturated at the fingertips of the hand 140. In other words, the upper limit of the range of the pseudo color image 130 exceeds the representational range of the display 24. The representational range of the display 24 refers to the range of brightness that can be displayed on the display 24. The range of the pseudo color image 130 refers to the range of brightness of the image pixels included in the pseudo color image 130. The display 24 is an example of a "display medium" and a "display device" according to the technology of the present disclosure.
[0128] As an example, FIG. 12 shows an example in which a pseudo-color image 130 is generated in accordance with a color setting instruction (hereinafter referred to as a "second color setting instruction") that sets the second region 132B to light blue instead of blue.
[0129] For example, if the color of second region 132B is set to blue as described above and the color of second region 132B becomes saturated, the color of second region 132B may be changed from blue to light blue. In the example shown in Fig. 12, the color of second region 132B is changed from blue to light blue, thereby preventing the color of second region 132B from becoming saturated.
[0130] Table 3 shows an example of the gain G derived in accordance with the second color setting instruction. As shown in Table 3, the gain G derived in accordance with the second color setting instruction includes negative gains in addition to positive gains. As shown in Table 3, since the gain G includes negative gains (i.e., negative values) in addition to positive gains, calculations including subtraction are performed in the calculation process (see FIG. 10 ).
[0131]
[0132] 12, a pseudo color image 130 is generated in which the second region 132B is set to light blue. When the color of the second region 132B is set to light blue, the gain G is set to a value that causes the range of the pseudo color image 130 to fall within the display area of the display 24 that displays the pseudo color image 130. This prevents the color of the second region 132B from becoming saturated.
[0133] For example, if the color of any one of the multiple areas 132 is specified as a saturated color, the gain setting unit 98 (see Figure 9) may adjust the gain G so that the range of the pseudo-color image 130 falls within the representation area of the display 24.
[0134] As an example, Figure 13 shows an example in which a pseudo-color image 130 is generated in accordance with a color setting instruction (hereinafter referred to as a "third color setting instruction") that sets the first area 132A to yellow, which is easily visible compared to light blue, or to red, which is a complementary color to light blue, while leaving the second area 132B set to light blue.
[0135] Table 4 shows an example of the gain G derived in accordance with the third color setting instruction. As shown in Table 4, the gain G derived in accordance with the third color setting instruction includes a negative gain in addition to a positive gain. As shown in Table 4, since the gain G includes a negative gain (i.e., a negative value) in addition to a positive gain, a calculation including subtraction is performed in the calculation process (see FIG. 10 ).
[0136]
[0137] 13 , a pseudocolor image 130 is generated in which the second region 132B is set to light blue and the first region 132A is set to yellow or red. Note that if the same color is set in different regions 132, the colors of the different regions 132 may become saturated, so it is preferable to set different colors in the different regions 132. In the example shown in FIG. 13 , the second region 132B corresponding to the blood vessels 142 is set to light blue, and the first region 132A corresponding to the portion of the hand 140 other than the blood vessels 142 is set to yellow or red, thereby emphasizing the blood vessels 142 relative to the portion of the hand 140 other than the blood vessels 142.
[0138] 14 shows an example in which the pseudo-color image 130 is generated in accordance with a color setting instruction (hereinafter referred to as a "fourth color setting instruction") that sets the second region 132B to light blue, the first region 132A to yellow or red, and the third region 132C to white. Note that the colors of the fourth region 132D and the fifth region 132E remain the same as those of the fourth region 132D and the fifth region 132E in the initial image 128.
[0139] Table 5 shows an example of the gain G derived in accordance with the fourth color setting instruction. As shown in Table 5, the gain G derived in accordance with the fourth color setting instruction includes a negative gain in addition to a positive gain. As shown in Table 5, since the gain G includes a negative gain (i.e., a negative value) in addition to a positive gain, a calculation including subtraction is performed in the calculation process (see FIG. 10 ).
[0140]
[0141] 14, a pseudocolor image 130 is generated in which the second region 132B is set to light blue, the first region 132A is set to yellow or red, and the third region 132C is set to white. In the example shown in FIG. 14, the third region 132C, which corresponds to the background 148, is set to white, thereby emphasizing the hand 140 and blood vessels 142 against the background 148.
[0142] The pseudo-color image 130 is an image in which the distinctiveness of wavelength information has been increased compared to, for example, a pseudo-color image in which the calculations only involve addition (hereinafter referred to as a "comparison pseudo-color image"). An example of an image in which the distinctiveness of wavelength information has been increased is an image that can be distinguished from the comparison pseudo-color image by differences in color and / or brightness, etc.
[0143] Here, color difference refers to, for example, a complementary relationship between colors on a color wheel. For example, if one of two colors related on the color wheel is used in the comparison pseudo color image, the other of the two colors related on the color wheel is used in the pseudo color image 130. Furthermore, the color used in the comparison pseudo color may be set for the comparison pseudo color by utilizing the complementary relationship between the three primary colors of light and the three primary colors of color. Furthermore, with regard to brightness difference, for example, if the comparison pseudo color image is expressed within a brightness range of "150 to 255," the pseudo color image 130 may be expressed within a brightness range of "0 to 50."
[0144] The comparison pseudo color image is an example of a "second pseudo color image" according to the technology of the present disclosure. The gain is an example of a "coefficient of the first spectral image" according to the technology of the present disclosure. The wavelength information is an example of "second wavelength information" according to the technology of the present disclosure.
[0145] Next, the operation of the imaging device 10 according to this embodiment will be described with reference to Fig. 15. Fig. 15 shows an example of the flow of image display processing according to this embodiment.
[0146] 15, first, in step ST10, the output value acquisition unit 82 acquires the output value Y of each physical pixel 58 based on the imaging data 120 input from the image sensor 14 to the processor 70 (see FIG. 4). After the processing of step ST10 is executed, the image display processing proceeds to step ST12.
[0147] In step ST12, the interference removal processor 84 performs interference removal processing on the output value Y of each physical pixel 58 acquired in step ST10 (see FIG. 4 ). As a result, the captured image 122 represented by the imaging data 120 is separated into spectral images 124 for each transmission wavelength band λ of the multiple filters 44. After the processing of step ST12 is executed, the image display processing proceeds to step ST14.
[0148] In step ST14, the alignment processing unit 86 performs alignment processing on the multiple spectral images 124 generated in step ST12 (see FIG. 5). After the processing in step ST14 is executed, the image display processing proceeds to step ST16.
[0149] In step ST16, the initial image generation unit 88 performs an allocation process for allocating the R, G, and B channels to the multiple spectral images 124 that have been aligned in step ST14, thereby generating an R channel image 126A, a G channel image 126B, and a B channel image 126C for each of the R, G, and B channels (see FIG. 6 ). After the process of step ST16 is executed, the image display process proceeds to step ST18.
[0150] In step ST18, the initial image generation unit 88 generates an initial image 128 by combining the R channel image 126A, the G channel image 126B, and the B channel image 126C generated in step ST16 (see FIG. 6). After the processing of step ST18 is executed, the image display processing proceeds to step ST20.
[0151] In step ST20, the initial image output unit 90 outputs the initial image data representing the initial image 128 generated in step ST18 to the display 24 (see FIG. 7). As a result, the initial image 128 is displayed on the display 24. After the processing of step ST20 is executed, the image display processing proceeds to step ST22.
[0152] In step ST22, the area setting determination unit 92 determines whether area division instruction data has been input to the processor 70 (see FIG. 8). If area division instruction data has not been input to the processor 70, the determination is negative, and the image display process proceeds to step ST36. If area division instruction data has been input to the processor 70, the determination is positive, and the image display process proceeds to step ST24.
[0153] In step ST24, the region setting unit 94 divides the initial image 128 into a plurality of regions 132 in accordance with the region division instruction data (see FIG. 8). After the process of step ST24 is executed, the image display process proceeds to step ST26.
[0154] In step ST26, the color setting determination unit 96 determines whether color setting instruction data has been input to the processor 70 (see FIG. 9). If color setting instruction data has not been input to the processor 70, the determination is negative, and the image display process proceeds to step ST36. If color setting instruction data has been input to the processor 70, the determination is positive, and the image display process proceeds to step ST28.
[0155] In step ST28, the gain setting unit 98 sets the gain G for the plurality of spectral images 124 in accordance with the color setting instruction data so that the region 132 specified by the region specification instruction is set to the color specified by the color specification instruction (see FIG. 9 ). After the processing of step ST28 is executed, the image display processing proceeds to step ST30.
[0156] In step ST30, the pseudo-color image generating unit 100 performs arithmetic processing on the multiple spectral images 124 based on the gain G set in step ST28, thereby generating an R channel image 126A, a G channel image 126B, and a B channel image 126C for each of the R, G, and B channels (see FIG. 10 ). As a result, the multiple spectral images 124 are assigned to the R, G, and B channels. After the processing of step ST30 is performed, the image display processing proceeds to step ST32.
[0157] In step ST32, the pseudo-color image generating unit 100 generates a pseudo-color image 130 by combining the R channel image 126A, the G channel image 126B, and the B channel image 126C generated in step ST30 (see FIG. 10). After the processing of step ST32 is executed, the image display processing proceeds to step ST34.
[0158] In step ST34, the pseudo-color image output unit 102 outputs the pseudo-color image data representing the pseudo-color image 130 generated in step ST32 to the display 24 (see FIG. 11 ). This causes the pseudo-color image 130 to be displayed on the display 24. After the processing of step ST34 is executed, the image display processing proceeds to step ST36.
[0159] In step ST36, the processor 70 determines whether a condition for terminating the image display process (i.e., a termination condition) is met. An example of the termination condition is a condition in which a user or the like issues an instruction to terminate the image display process to the imaging device 10. If the termination condition is not met in step ST36, the determination is negative, and the image display process proceeds to step ST26. If the termination condition is met in step ST36, the determination is positive, and the image display process ends. Note that the image display processing method described above as the operation of the imaging device 10 is an example of an "information processing method" according to the technology of the present disclosure.
[0160] As described above, in the imaging device 10 according to this embodiment, the processor 70 performs an assignment process for assigning different spectral images 124 to different channels, thereby generating a channel image 126 for each channel, and generating a pseudo-color image 130 based on the multiple channel images 126 (see FIG. 10 ). The assignment process includes generating the channel images 126 based on operations including subtraction for the multiple spectral images 124. Therefore, compared to, for example, generating the channel images 126 based on operations including only addition for the multiple different spectral images 124, color adjustment with a high degree of freedom can be achieved when pseudo-coloring the multiple spectral images 124. As a result, it is possible to generate a pseudo-color image 130 that is visually distinguishable from a comparison pseudo-color image generated by combining channel images 126 generated based on operations including only addition.
[0161] The operation is a multiply-and-accumulate operation involving the multiple spectral images 124, and the subtraction is realized by including negative values in the gains of the multiple spectral images 124 in the multiply-and-accumulate operation. Therefore, the color of the pseudocolor image 130 can be adjusted by adjusting the value of the gain including a negative value and / or the number of gains including a negative value.
[0162] The gain G is set to a value that causes the range of the pseudo color image 130 to fall within the display area of the display 24 that displays the pseudo color image 130. Therefore, it is possible to prevent the colors set for the pseudo color image 130 from becoming saturated.
[0163] The plurality of spectral images 124 includes polarization information and wavelength information, and the number of the plurality of spectral images 124 is equal to or greater than the number of channels, which allows for greater flexibility in adjusting the color of the pseudocolor image 130 compared to, for example, a case where the number of the plurality of spectral images 124 is smaller than the number of channels.
[0164] The processor 70 performs alignment processing on the multiple spectral images 124 obtained by capturing images using the image sensor 14 having multiple polarizers 46, and subtraction is performed on the aligned multiple spectral images 124. Here, the multiple spectral images 124 are images obtained by capturing images using the image sensor 14 having multiple polarizers 46. For this reason, for example, misregistration occurs in the multiple spectral images 124. Therefore, by performing alignment processing on the multiple spectral images 124, it is possible to ensure the image quality of the initial image 128 (see FIG. 6 ) and the pseudo-color image 130 (see FIG. 10 ) compared to when alignment processing is not performed.
[0165] The pseudo color image 130 is an image in which the discrimination of wavelength information is enhanced compared to a comparison pseudo color image in which the calculation involves only addition, so that a user or the like can visually distinguish the pseudo color image 130 from the comparison pseudo color image.
[0166] The processor 70 divides the image generated based on the plurality of spectral images 124 into a plurality of regions 132, and performs calculations based on color information set for the plurality of regions 132. Therefore, a user or the like can set a color for each of the plurality of regions 132 that he or she specifies.
[0167] The different channels are the three primary color channels, and therefore a pseudocolor image 130 can be generated based on the three primary color channels.
[0168] The processor 70 outputs pseudo-color image data for displaying the pseudo-color image 130 on the display 24. Therefore, the user or the like can confirm the subject image included in the pseudo-color image 130 displayed on the display 24 in a color different from the original color of the subject image.
[0169] In the above embodiment, the multiple spectral images 124 are assigned to the R channel, G channel, and B channel corresponding to the three primary colors of color, but the multiple spectral images 124 may also be assigned to channels corresponding to the three primary colors of light.
[0170] Furthermore, in the above embodiment, the calculation including subtraction is performed on all the spectral images 124 , but the calculation including subtraction may be performed on some of the spectral images 124 out of the plurality of spectral images 124 .
[0171] Furthermore, in the above embodiment, a plurality of spectral images 124 including polarization information and wavelength information are generated, but a plurality of spectral images 124 including only either polarization information or wavelength information may also be generated.
[0172] Furthermore, in the above embodiment, the initial image 128 and the pseudo-color image 130 are displayed on the display 24 provided in the imaging device 10, but the initial image 128 and the pseudo-color image 130 may also be displayed on a display medium or display device provided in an external device other than the imaging device 10.
[0173] Furthermore, in the above embodiment, as an example, a multispectral image has been described in which a multispectral image is generated based on light dispersed into eight transmission wavelength bands λ. However, the eight transmission wavelength bands λ are merely an example, and any number of transmission wavelength bands λ may be used.
[0174] Furthermore, the technology according to the above embodiment may be applied to devices of types other than the imaging device 10 (hereinafter referred to as "other devices").
[0175] Furthermore, in the above embodiment, processor 70 is exemplified, but instead of processor 70, or together with processor 70, at least one other CPU, at least one GPU, and / or at least one TPU may be used.
[0176] In the above embodiment, the image display program 80 is stored in the NVM 72. However, the technology of the present disclosure is not limited to this. For example, the image display program 80 may be stored in a portable, non-transitory, computer-readable storage medium (hereinafter simply referred to as a "non-transitory storage medium") such as an SSD or a USB memory. The image display program 80 stored in the non-transitory storage medium may be installed in the computer 20 of the imaging device 10.
[0177] In addition, the image display program 80 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the image display program 80 may be downloaded and installed on the computer 20 in response to a request from the imaging device 10.
[0178] Furthermore, it is not necessary to store the entire image display program 80 in a storage device such as another computer or server device connected to the imaging device 10, or in the NVM 72; only a portion of the image display program 80 may be stored.
[0179] Furthermore, although the imaging device 10 has a built-in computer 20 , the technology of the present disclosure is not limited to this. For example, the computer 20 may be provided outside the imaging device 10 .
[0180] Furthermore, in the above embodiment, the computer 20 including the processor 70, the NVM 72, and the RAM 74 is exemplified, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 20. Furthermore, instead of the computer 20, a combination of a hardware configuration and a software configuration may be used.
[0181] Furthermore, the following various processors can be used as hardware resources for executing the various processes described in the above embodiments. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing various processes by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, and ASICs, which are processors with a circuit configuration designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute various processes.
[0182] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.
[0183] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes various processes. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute various processes, on a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.
[0184] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The above-described gaze detection process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.
[0185] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.
[0186] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."
[0187] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.
Claims
1. A processor is provided. The processor, performing an assignment process for assigning different spectral images to different channels to generate a channel image for each of the channels; generating a first pseudocolor image based on the plurality of channel images; the allocation process includes a process of generating the channel image based on an operation including subtraction on a first spectral image of the plurality of spectral images; the operation is a multiply-and-accumulate operation involving the first spectral image, The subtraction is realized by including a negative value in the coefficient of the first spectral image in the multiplication and accumulation operation. Information processing device.
2. The coefficient is set to a value such that the range of the first pseudo-color image falls within the representation area of a display medium on which the first pseudo-color image is displayed. The information processing device according to claim 1 .
3. The plurality of spectral images include polarization information and / or first wavelength information, The number of the plurality of spectral images is equal to or greater than the number of the channels. The information processing device according to claim 1 .
4. The plurality of spectral images are images obtained by capturing images using an image sensor having a polarizer, The processor performs a registration process on the first spectral image; The subtraction is performed on the first spectral image that has been subjected to the registration process. The information processing device according to claim 1 .
5. The first pseudo-color image is an image in which the discrimination of second wavelength information is enhanced compared to a second pseudo-color image in which the calculation includes only addition. The information processing device according to claim 1 .
6. The processor divides an image generated based on the plurality of spectral images into a plurality of regions; The calculation is performed based on color information set for the plurality of regions. The information processing device according to claim 1 .
7. The different channels are channels of three primary colors. The information processing device according to claim 1 .
8. The processor outputs data for displaying the first pseudo-color image on a display device. The information processing device according to claim 1 .
9. Performing an assignment process for assigning different spectral images to different channels to generate a channel image for each of the channels; and generating a first pseudocolor image based on a plurality of said channel images; Equipped with the allocation process includes a process of generating the channel image based on an operation including subtraction on a first spectral image of the plurality of spectral images; the operation is a multiply-and-accumulate operation involving the first spectral image, The subtraction is realized by including a negative value in the coefficient of the first spectral image in the multiplication and accumulation operation. Information processing methods.
10. A program for causing a computer to execute a specific process, comprising: The specific processing is performing an allocation process for allocating different spectral images to different channels to generate a channel image for each of the channels; and generating a first pseudocolor image based on a plurality of said channel images; Including, the allocation process includes a process of generating the channel image based on an operation including subtraction on a first spectral image of the plurality of spectral images; the operation is a multiply-and-accumulate operation involving the first spectral image, The subtraction is realized by including a negative value in the coefficient of the first spectral image in the multiplication and accumulation operation. program.
