Information processing device, information processing method, and program

JPWO2023188512A5Pending Publication Date: 2024-12-09
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024511192
Authority / Receiving Office
JP · JP
Patent Type
Applications
Filing Date
2024-09-02
Publication Date
2024-12-09

AI Technical Summary

Technical Problem

Current information processing devices face challenges in reducing processor load while maintaining information quantity, particularly in handling multispectral image data without compromising resolution or increasing computational burden.

Method used

The implementation of a processor that performs efficient first and second processing stages on multispectral image data, including a reduction process that thins out information, uses an arithmetic process based on a matrix condition number, and selectively performs display and storage processing to minimize information processing load.

Benefits of technology

This approach reduces the processor load by thinning out polarization and wavelength information, allowing for parallel display and storage processing while maintaining image quality and reducing the amount of information processed compared to traditional methods.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

An information processing device according to the present invention comprises a processor. The processor performs first processing and second processing related to multispectral image data on output data from a sensor. The first processing includes reduction processing in which an amount of information of the output data is reduced to a greater extent than in the second processing.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing method, and program

[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program.

[0002] Japanese Patent Laid-Open Publication No. 2012-044519 discloses a digital camera including pixels constituting a color sensor for four or more colors having different spectral sensitivity characteristics. The digital camera includes a multispectral image sensor that can operate in either a first signal readout mode in which an image signal having a first amount of information is read out in a first signal readout time, or a second signal readout mode in which an image signal having a second amount of information less than the first amount is read out in a second readout time shorter than the first readout time. In the first signal readout mode, the multispectral image sensor outputs image signals having a number of colors corresponding to all four or more colors read out from the pixels constituting the color sensor for the four or more colors, while in the second signal readout mode, the multispectral image sensor outputs image signals having a number of colors fewer than in the first signal readout mode and forming a color gamut narrower than the color gamut reproducible by the image signals read out in the first signal readout mode.

[0003] Japanese Patent Application Laid-Open No. 2021-136501 discloses an imaging device including an imaging means, a calculation means, an image generation means, and a control means. The imaging means includes polarization pixels that detect polarization information in a plurality of different directions. The calculation means calculates polarization information for the polarization pixels by performing a first polarization calculation or a second polarization calculation that has a smaller calculation load than the first polarization calculation on the video signal output from the polarization pixels. The image generation means generates a polarization-processed image using the polarization information calculated by the calculation means. The control means switches between the first polarization calculation and the second polarization calculation performed by the calculation means depending on the result of detecting a predetermined timing, mode, or predetermined state.

[0004] JP 2008-525158 A discloses an apparatus for acquiring shock data for use in hyperspectral or multispectral imaging, which includes at least one of a detector, a wavelength selector with or without a filter, a power source, and image acquisition hardware.

[0005] One embodiment of the technology disclosed herein provides an information processing device, an information processing method, and a program that can reduce the load on a processor compared to when the first processing is performed without reducing the amount of information in the output data.

[0006] A first aspect of the technology of the present disclosure is an information processing device that includes a processor, wherein the processor performs first and second processing related to multispectral image data on output data from a sensor, and the first processing includes a reduction processing that reduces the amount of information in the output data more than the second processing.

[0007] A second aspect of the technology of the present disclosure is the information processing device according to the first aspect, wherein the reduction process includes a thinning process of thinning out information from the output data.

[0008] A third aspect of the technology of the present disclosure is an information processing device in which, in the information processing device of the first or second aspect, the sensor is a first image sensor and the output data is imaging data output from the first image sensor.

[0009] A fourth aspect of the technology of the present disclosure is the information processing device according to the third aspect, wherein the imaging data is data output from the first image sensor without reducing the electrical signal.

[0010] A fifth aspect of the technique of the present disclosure is the information processing device according to the third or fourth aspect, wherein the information includes polarization information included in the imaging data.

[0011] A sixth aspect of the technology of the present disclosure is an information processing device according to any one of the third to fifth aspects, wherein the information includes wavelength information included in the imaging data.

[0012] A seventh aspect of the technology of the present disclosure is an information processing device according to any one of the first to sixth aspects, wherein the reduction processing includes an arithmetic processing using a first matrix, and the arithmetic processing is performed based on the condition number of the first matrix.

[0013] An eighth aspect of the technology of the present disclosure is an information processing device according to the seventh aspect, wherein the processor generates a third matrix by deleting components corresponding to information from a second matrix indicating the relationship between the output data and the multispectral image data, and the first matrix is ​​generated based on an inverse matrix of the third matrix.

[0014] A ninth aspect of the technology of the present disclosure is an information processing device relating to any one of the first to eighth aspects, wherein the first process is a display process that generates image data for display based on output data, and the second process is a storage process that generates image data for storage based on the output data.

[0015] A tenth aspect of the technology of the present disclosure is the information processing device according to the ninth aspect, wherein the display process and the saving process are performed in parallel.

[0016] An eleventh aspect of the technology of the present disclosure is the information processing device according to the ninth aspect, in which the display process and the storage process are selectively performed.

[0017] A twelfth aspect of the technology of the present disclosure is an information processing device according to any one of the first to eleventh aspects, wherein the first process and / or the second process is a process of generating multispectral image data based on output data.

[0018] A thirteenth aspect of the technology of the present disclosure is an information processing device relating to any one of the first to twelfth aspects, in which the sensor is a second image sensor, the processor selectively performs a third process and a fourth process which are read processes from the second image sensor, and the third process thins out more pixels from the second image sensor than the fourth process.

[0019] A fourteenth aspect of the technology of the present disclosure is an information processing device according to the thirteenth aspect, wherein the second image sensor has a plurality of first pixel blocks, each of which is formed by a plurality of first pixels, and a plurality of types of polarizers are assigned to the plurality of first pixels, and the third processing includes processing for selecting, in each first pixel block, a first pixel corresponding to each polarizer from the plurality of first pixels.

[0020] A fifteenth aspect of the technology of the present disclosure is an information processing device according to the thirteenth aspect, wherein the second image sensor has a plurality of second pixel blocks, each of which is formed by a plurality of second pixels, and a plurality of types of polarizers are assigned to the plurality of second pixels, and the third processing includes a process of selecting, from the plurality of second pixels, second pixels corresponding to different types of polarizers in adjacent second pixel blocks.

[0021] A sixteenth aspect of the technology of the present disclosure is an information processing device according to the thirteenth aspect, wherein the second image sensor has a plurality of third pixel blocks, each of which is formed by a plurality of third pixels, and a plurality of types of polarizers are assigned to the plurality of third pixels, and the third processing includes a process of selecting pixel lines from a plurality of rows of pixel lines formed by the plurality of third pixel blocks, with a row spacing therebetween.

[0022] A 17th aspect of the technology of the present disclosure is an information processing device according to the 13th aspect, wherein the second image sensor has a plurality of fourth pixel blocks, each fourth pixel block is formed by a plurality of fourth pixels, and a plurality of types of polarizers are assigned to the plurality of fourth pixels, and the third processing includes processing of selecting a fourth pixel block at intervals from the plurality of fourth pixel blocks.

[0023] An eighteenth aspect of the technology of the present disclosure is an information processing method comprising performing first processing and second processing related to multispectral image data on output data from a sensor, wherein the first processing includes a reduction processing that reduces the amount of information in the output data more than the second processing.

[0024] A nineteenth aspect of the technology of the present disclosure is a program for causing a computer to execute a specific process, the specific process comprising performing a first process and a second process related to multispectral image data on output data from a sensor, the first process including a reduction process that reduces the amount of information in the output data more than the second process.

[0025] 1 is a perspective view showing an example of an imaging device according to an embodiment. FIG. 2 is an exploded side view showing an example of a lens device and an imaging device body according to an embodiment. FIG. 3 is a block diagram showing an example of a hardware configuration of an imaging device according to an embodiment. FIG. 4 is an exploded perspective view showing an example of a photoelectric conversion element according to an embodiment. FIG. 5 is a block diagram showing an example of a functional configuration for realizing image processing according to an embodiment. FIG. 6 is a block diagram showing an example of an operation of a second output value acquisition unit and a second interference removal processing unit according to an embodiment. FIG. 7 is a block diagram showing an example of an operation of a second interference removal processing unit and a second image data generation unit according to an embodiment. FIG. 8 is a block diagram showing an example of a functional configuration for realizing interference removal matrix generation processing according to an embodiment. FIG. 9 is a block diagram showing an example of an operation of a wavelength setting unit, a polarization component removing unit, an inverse matrix derivation unit, a wavelength component removing unit, a condition number derivation unit, and an interference removal matrix determining unit according to an embodiment. FIG. 10 is a block diagram showing an example of an operation of a first output value acquisition unit and a first interference removal processing unit according to an embodiment. FIG. 11 is a block diagram showing an example of an operation of a first interference removal processing unit and a second image data generation unit according to an embodiment. FIG. 12 is a flowchart showing an example of the flow of a display processing according to an embodiment. FIG. 13 is a flowchart showing an example of the flow of a storage processing according to an embodiment. FIG. 14 is a flowchart showing an example of the flow of an interference removal matrix generation processing according to an embodiment. FIG. 10 is an explanatory diagram showing a third modified example of a read process according to the embodiment; FIG. 11 is an explanatory diagram showing a fourth modified example of a read process according to the embodiment; FIG. 12 is an explanatory diagram showing a fifth modified example of a read process according to the embodiment; FIG. 13 is an explanatory diagram showing a sixth modified example of a read process according to the embodiment; and FIG. 14 is an explanatory diagram showing a seventh modified example of a read process according to the embodiment.

[0026] Hereinafter, exemplary embodiments of an information processing device, an information processing method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] I / F is an abbreviation for "Interface". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". NVM is an abbreviation for "Non-Volatile Memory". RAM is an abbreviation for "Random Access Memory". CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". HDD is an abbreviation for "Hard Disk Drive". LiDAR is an abbreviation for "Light Detection and Ranging". TPU is an abbreviation for "Tensor processing unit". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". ASIC is an abbreviation for "Application Specific Integrated Circuit." FPGA is an abbreviation for "Field-Programmable Gate Array." PLD is an abbreviation for "Programmable Logic Device." SoC is an abbreviation for "System-on-a-chip." IC is an abbreviation for "Integrated Circuit."

[0029] In the description of this specification, "same" refers to the same in the sense of including, in addition to being completely the same, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, "orthogonal" refers to the same in the sense of including, in addition to being completely orthogonal, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. In the description of this specification, "straight line" refers to the same in the sense of including, in addition to being completely straight, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure.

[0030] 1 , as an example, an imaging device 10 is a multispectral camera capable of outputting multispectral images, and includes a lens device 12 and an imaging device body 14. The imaging device 10 is an example of an “information processing device” according to the technology of the present disclosure.

[0031] 2, the lens device 12 has a first mount 16, and the imaging device body 14 has a second mount 18. The first mount 16 and the second mount 18 are each a C-mount and are formed in an annular shape. The first mount 16 is provided coaxially with the optical axis OA of the lens device 12. A screw 20 is formed on the outer peripheral surface of the first mount 16, and a screw hole 22 is formed on the inside of the second mount 18.

[0032] The first mount 16 is attached to the second mount 18 by threading the screws 20 into the screw holes 22. The lens device 12 is attached to the imaging device body 14 by attaching the first mount 16 to the second mount 18. Alternatively, the screw holes 22 may be formed on the inside of the first mount 16, and the screws 20 may be formed on the outer circumferential surface of the second mount 18.

[0033] 3, the lens device 12 includes an optical system 24. The imaging device body 14 includes an image sensor 26, a control driver 28, an input / output I / F 30, a computer 32, a reception device 34, and a display 36.

[0034] The optical system 24 has a first lens 38, a pupil division filter 40, and a second lens 42. The first lens 38, the pupil division filter 40, and the second lens 42 are arranged in this order along the optical axis OA from the subject 4 side to the image sensor 26 side. The first lens 38 transmits light obtained by light emitted from the light source 2 being reflected by the subject 4 (hereinafter referred to as "subject light") through the pupil division filter 40. The second lens 42 forms an image of the subject light that has transmitted through the pupil division filter 40 on a light receiving surface 48A of a photoelectric conversion element 48 provided in the image sensor 26.

[0035] The pupil division filter 40 has a filter 44A, a filter 44B, a filter 44C, a polarizer 46A, a polarizer 46B, and a polarizer 46C. For convenience, in Fig. 3, the filters 44A, 44B, and 44C are shown as being linearly arranged in a direction perpendicular to the optical axis OA, but the filters 44A, 44B, and 44C are actually arranged in a direction around the optical axis OA.

[0036] The filter 44A has a first transmission wavelength band λ 1 and the filter 44B has a second transmission wavelength band λ 2 and the filter 44C has a third transmission wavelength band λ 3 The first transmission wavelength band λ 1 , second transmission wavelength band λ 2 , and the third transmission wavelength band λ 3 are different wavelength bands. 1 , second transmission wavelength band λ 2 , and the third transmission wavelength band λ 3 may be set to any band. Hereinafter, when there is no need to distinguish between the filters 44A, 44B, and 44C, the filters 44A, 44B, and 44C will be referred to as "filters 44." In addition, the first transmission wavelength band λ 1 , second transmission wavelength band λ 2 , and the third transmission wavelength band λ 3 When there is no need to distinguish between the first transmission wavelength band λ 1, second transmission wavelength band λ 2 , and the third transmission wavelength band λ 3 is referred to as the "transmission wavelength band λ."

[0037] Polarizers 46A, 46B, and 46C are superimposed on filters 44A, 44B, and 44C, respectively. Polarizer 46A is a polarizer whose transmission axis angle is set to 0°. Polarizer 46B is a polarizer whose transmission axis angle is set to 60°. Polarizer 46C is a polarizer whose transmission axis angle is set to 120°. Hereinafter, when there is no need to distinguish between polarizers 46A, 46B, and 46C, polarizers 46A, 46B, and 46C will each be referred to as "polarizer 46."

[0038] 3, the number of filters 44 is three, but there may be any number of filters 44. Also, in the example shown in FIG. 3, the number of polarizers 46 is three, but there may be any number of polarizers 46.

[0039] 3 , the image sensor 26 includes a photoelectric conversion element 48 and a signal processing circuit 50. The image sensor 26 is, for example, a CMOS image sensor. In the present embodiment, a CMOS image sensor is used as the image sensor 26, but the technology of the present disclosure is not limited thereto. For example, the technology of the present disclosure can be applied even if the image sensor 26 is another type of image sensor, such as a CCD image sensor. The image sensor 26 is an example of a "sensor" and a "first image sensor" according to the technology of the present disclosure.

[0040] As an example, Fig. 3 shows a schematic configuration of the photoelectric conversion element 48. Also, as an example, Fig. 4 shows a specific configuration of a portion of the photoelectric conversion element 48. The photoelectric conversion element 48 has a pixel layer 52, a polarizing filter layer 54, and a spectral filter layer 56.

[0041] The pixel layer 52 has a plurality of pixels 58. The plurality of pixels 58 are arranged in a matrix and form the light receiving surface 48A of the photoelectric conversion element 48. Each pixel 58 is a physical pixel having a photodiode (not shown), which photoelectrically converts received light and outputs an electrical signal according to the amount of received light.

[0042] Hereinafter, the pixels 58 provided in the photoelectric conversion element 48 will be referred to as "physical pixels 58" to distinguish them from pixels that form a multispectral image. Furthermore, the pixels that form a multispectral image will be referred to as "image pixels." The physical pixels 58 are an example of a "pixel" according to the technology of the present disclosure.

[0043] The photoelectric conversion element 48 outputs the electrical signals output from the plurality of physical pixels 58 to the signal processing circuit 50 as imaging data 120. The signal processing circuit 50 digitizes the analog imaging data 120 input from the photoelectric conversion element 48. The imaging data 120 is, for example, data output from all the physical pixels 58 of the photoelectric conversion element 48.

[0044] All physical pixels 58 refer to all physical pixels 58 that are capable of outputting an electrical signal. Specifically, all physical pixels 58 refer to the remaining physical pixels 58 excluding invalid pixels among the plurality of physical pixels 58 included in the entire region of the light-receiving surface 48A, or the plurality of physical pixels 58 included in the valid region of the light-receiving surface 48A. In other words, the imaging data 120 is data that is output without performing thinning processing on the physical pixels 58 for all physical pixels 58 that are capable of outputting an electrical signal, and is data that is output from the photoelectric conversion element 48 without reducing the electrical signal. The imaging data 120 is an example of "output data" and "imaging data" according to the technology of the present disclosure.

[0045] The plurality of physical pixels 58 form a plurality of pixel blocks 60. Each pixel block 60 is formed by two vertical and two horizontal rows, for a total of four physical pixels 58. For convenience, in Fig. 3, the four physical pixels 58 forming each pixel block 60 are shown as being linearly arranged in a direction perpendicular to the optical axis OA, but as shown in Fig. 4 as an example, the four physical pixels 58 are arranged adjacent to each other in the vertical and horizontal directions of the photoelectric conversion element 48.

[0046] The polarizing filter layer 54 includes a polarizer 62A, a polarizer 62B, a polarizer 62C, and a polarizer 62D. The polarizer 62A has a transmission axis angle θ 1 (Hereinafter, "first polarization angle θ 1 The polarizer 62B has a transmission axis angle θ 2 (Hereinafter, "second polarization angle θ 2 The polarizer 62C has a transmission axis angle θ 3 (Hereinafter, “third polarization angle θ 3 The polarizer 62D has a transmission axis angle θ 4 (Hereinafter, "4th polarization angle θ 4 ") was set at 135°.

[0047] Hereinafter, when there is no need to distinguish between the polarizers 62A, 62B, 62C, and 62D, the polarizers 62A, 62B, 62C, and 62D will be referred to as "polarizers 62." In addition, the first polarization angle θ 1 , second polarization angle θ 2 , third polarization angle θ 3 , and the fourth polarization angle θ 4 When there is no need to distinguish between the first polarization angle θ 1 , second polarization angle θ 2 , third polarization angle θ 3 , and the fourth polarization angle θ 4 is referred to as the "polarization angle θ."

[0048] The spectral filter layer 56 has a B filter 64A, a G filter 64B, and an R filter 64C. The B filter 64A is a blue bandpass filter that transmits the most light in the blue wavelength band among the multiple wavelength bands of light. The G filter 64B is a green bandpass filter that transmits the most light in the green wavelength band among the multiple wavelength bands of light. The R filter 64C is a red bandpass filter that transmits the most light in the red wavelength band among the multiple wavelength bands of light. The B filter 64A, G filter 64B, and R filter 64C are assigned to each pixel block 60.

[0049] For convenience, Fig. 3 shows the B filter 64A, G filter 64B, and R filter 64C arranged in a line along a direction perpendicular to the optical axis OA, but as an example, as shown in Fig. 4, the B filter 64A, G filter 64B, and R filter 64C are arranged in a matrix in a predetermined pattern. In the example shown in Fig. 4, the B filter 64A, G filter 64B, and R filter 64C are arranged in a matrix in a Bayer array, which is an example of a predetermined pattern. Note that the predetermined pattern may be an RGB stripe array, an R / G checkerboard array, an X-Trans (registered trademark) array, a honeycomb array, or the like, in addition to the Bayer array.

[0050] Hereinafter, when there is no need to distinguish between the B filter 64A, the G filter 64B, and the R filter 64C, they will each be referred to as the "filter 64."

[0051] As an example, as shown in FIG. 3, the input / output I / F 30 is connected to a signal processing circuit 50, a control driver 28, a computer 32, a reception device 34, and a display 36.

[0052] The computer 32 includes a processor 70, an NVM 72, and a RAM 74. The processor 70 is an example of a "processor" according to the technology of the present disclosure. The processor 70 controls the entire imaging device 10. The processor 70 is, for example, an arithmetic processing device including a CPU and a GPU. The GPU operates under the control of the CPU and is responsible for executing image-related processing. Here, an arithmetic processing device including a CPU and a GPU is given as an example of the processor 70, but this is merely an example. The processor 70 may be one or more CPUs that integrate GPU functionality, or one or more CPUs that do not integrate GPU functionality. The processor 70, the NVM 72, and the RAM 74 are connected via a bus 76, which is connected to the input / output I / F 30.

[0053] The NVM 72 is a non-transitory storage medium that stores various parameters and programs. For example, the NVM 72 is a flash memory (e.g., an EEPROM). However, this is merely an example, and an HDD or the like may also be used as the NVM 72 in addition to the flash memory. The RAM 74 temporarily stores various information and is used as a work memory.

[0054] The processor 70 reads out necessary programs from the NVM 72 and executes the read programs in the RAM 74. The processor 70 controls the control driver 28 and the signal processing circuit 50 in accordance with the programs executed in the RAM 74. The control driver 28 controls the photoelectric conversion element 48 under the control of the processor 70.

[0055] The reception device 34 has, for example, a release button, a touch panel, and hard keys (all not shown), and receives instructions from a user, etc. The display 36 is, for example, a liquid crystal display, and displays various images including multispectral images.

[0056] As an example, as shown in FIG. 5 , an image processing program 80 is stored in the NVM 72. The image processing program 80 is an example of a "program" according to the technology of the present disclosure. The processor 70 reads the image processing program 80 from the NVM 72 and executes the read image processing program 80 on the RAM 74. The processor 70 performs image processing in accordance with the image processing program 80 executed on the RAM 74. The image processing is an example of a "specific process" according to the technology of the present disclosure. The image processing includes a display process and a storage process.

[0057] The display process and the saving process are processes performed on the imaging data 120. Specifically, the display process is a process of generating display image data 122 (i.e., live view image data) at a default frame rate based on the imaging data 120. The default frame rate is, for example, 30 fps or 60 fps. The saving process is a process of generating saving image data 124 (i.e., actual exposure image data) based on the imaging data 120 when an imaging instruction from a user or the like is accepted by the acceptance device 34 (see FIG. 3 ). The display image data 122 is an example of "display image data" according to the technology of the present disclosure, and the saving image data 124 is an example of "saving image data" according to the technology of the present disclosure.

[0058] The display image data 122 and the storage image data 124 are both multispectral image data, and the display process and the storage process are both processes related to the multispectral image data. When an image capture instruction from a user or the like is accepted by the acceptance device 34, the storage process is performed in parallel with the display process. For example, the display process and the storage process are performed for the same frame. The display process is an example of a "first process" according to the technology of the present disclosure, and the storage process is an example of a "second process" according to the technology of the present disclosure.

[0059] The display process is realized by the processor 70 operating as a first output value acquisition unit 82 , a first interference removal processing unit 84 , and a first image data generation unit 86 in accordance with the image processing program 80 .

[0060] The saving process is realized by the processor 70 operating as a second output value acquisition unit 92 , a second interference removal processing unit 94 , and a second image data generation unit 96 in accordance with the image processing program 80 .

[0061] First, before explaining the display process, the saving process will be explained.

[0062] 6 as an example, in the saving process, the second output value acquisition unit 92 acquires the output values ​​Y of all the physical pixels 58 based on the imaging data 120 input from the image sensor 26 to the processor 70. The output value Y of each physical pixel 58 corresponds to the luminance value of each pixel included in the captured image represented by the imaging data 120.

[0063] Here, the output value Y of each physical pixel 58 is a value including interference (i.e., crosstalk). 1 , second transmission wavelength band λ 2 , and the third transmission wavelength band λ 3 Since light of each transmission wavelength band λ is incident, the output value Y is 1 a value according to the amount of light, and a second transmission wavelength band λ 2 and the third transmission wavelength band λ 3 The value is a mixture of values ​​according to the amount of light.

[0064] To acquire the storage image 134 (see FIG. 7 ), which is a multispectral image for storage, the processor 70 needs to perform a process of separating and extracting values ​​corresponding to each transmission wavelength band λ from the output value Y for each physical pixel 58, that is, an interference removal process, which is a process of removing interference, on the output value Y. Therefore, in this embodiment, the second interference removal processing unit 94 performs the interference removal process on the output values ​​Y of all physical pixels 58 acquired by the second output value acquisition unit 92.

[0065] Here, the interference removal process for obtaining the storage image 134 will be described. The output value Y of each physical pixel 58 includes the luminance values ​​of red, green, and blue as components of the output value Y. The output value Y of each physical pixel 58 is expressed by equation (1). R is the red luminance value of the output value Y, and Y G is the luminance value of green in the output value Y, and Y B is the luminance value of blue in the output value Y.

[0066] The pixel value X of each image pixel forming the storage image 134 is within the first transmission wavelength band λ 1the brightness value of the light, the second transmission wavelength band λ 2 The luminance value of the light, and the third transmission wavelength band λ 3 The pixel value X of each image pixel is expressed by the following equation (2): λ1 is the first transmission wavelength band λ of the pixel value X 1 is the brightness value of the light, and the brightness value X λ2 is the second transmission wavelength band λ of the pixel value X 2 is the brightness value of the light, and the brightness value X λ3 is the third transmission wavelength band λ of the pixel value X 3 is the brightness value of the light.

[0067] When the interference matrix is ​​A, the output value Y of each physical pixel 58 is expressed by equation (3).

[0068] The interference matrix A is a matrix defined based on the spectrum of the subject light, the spectral transmittance of the first lens 38 , the spectral transmittance of the second lens 42 , the spectral transmittance of the plurality of filters 44 , and the spectral sensitivity of the image sensor 26 .

[0069] The interference cancellation matrix, which is the generalized inverse matrix of the interference matrix A, is defined as A + In this case, the pixel value X of each image pixel is expressed by equation (4).

[0070] Interference cancellation matrix A + Similarly to the interference matrix A, the interference cancellation matrix A is a matrix defined based on the spectrum of the subject light, the spectral transmittance of the first lens 38, the spectral transmittance of the second lens 42, the spectral transmittance of the plurality of filters 44, and the spectral sensitivity of the image sensor 26. + is set by a user or the like and stored in advance in the NVM 72.

[0071] The second interference cancellation processor 94 uses the interference cancellation matrix A stored in the NVM 72 + and the output values ​​Y of all the physical pixels 58 acquired by the second output value acquisition unit 92 are acquired, and the acquired interference cancellation matrix A + and the output values ​​Y of all the physical pixels 58, the pixel value X of each image pixel is output according to equation (4).

[0072] Here, as described above, the pixel value X of each image pixel is the first transmission wavelength band λ 1 The brightness value of the light X λ1 , second transmission wavelength band λ 2 The brightness value of the light X λ2 , and the third transmission wavelength band λ 3 The brightness value of the light X λ3 as a component of the pixel value X. 1 The brightness value of the light X λ1 is represented by the first image data in the imaging data 120. 2 The brightness value of the light X λ2 is represented by the second image data in the imaging data 120. 3 The brightness value of the light X λ3 is represented by the third image data of the imaging data 120.

[0073] In this way, the second interference removal processing unit 94 performs the interference removal processing, and the imaging data 120 is filtered out in the first transmission wavelength band λ 1 The brightness value of the light X λ1 and a first image data representing a second transmission wavelength band λ 2 The brightness value of the light X λ2 and a third transmission wavelength band λ 3 The brightness value of the light X λ3 That is, the imaging data 120 is separated into image data for each of the transmission wavelength bands of the plurality of filters 44.

[0074] 7 , the second image data generation unit 96 generates storage image data 124 by combining the first image data, the second image data, and the third image data generated by the second interference removal processing unit 94 through the interference removal processing. The storage image data 124 is image data representing a storage image 134, which is a multispectral image. The storage image data 124 is stored in the NVM 72.

[0075] In this way, in the storage process, the interference cancellation matrix A is applied to the output values ​​Y of all the physical pixels 58. +A multispectral image is obtained by performing interference removal processing using the above method.

[0076] Here, it is conceivable that a multispectral image can be obtained by performing the same processing as the storage processing in the display processing that is performed in parallel with the storage processing. That is, it is conceivable that a multispectral image can be obtained in the display processing by performing interference removal processing on the output values ​​Y of all physical pixels 58. However, if interference removal processing is performed on the output values ​​Y of all physical pixels 58 in the display processing that is performed in parallel with the storage processing, the load on the processor 70 increases.

[0077] Therefore, in order to suppress an increase in the load on the processor 70, it is conceivable to perform a thinning process on all of the physical pixels 58, that is, to perform a read process on a number of physical pixels 58 that is less than the total number of physical pixels 58. However, when performing a thinning process on all of the physical pixels 58, a storage process performed in parallel with a display process will result in a multispectral image having a lower resolution than a multispectral image obtained by performing an interference removal process on the output values ​​Y of all of the physical pixels 58. Therefore, in order to avoid a decrease in the resolution of the multispectral image in the storage process, it becomes necessary to perform the storage process separately from the display process.

[0078] In this embodiment, in consideration of the above circumstances, in order to suppress an increase in the load on the processor 70 and to enable the display process to be performed in parallel with the storage process, the display process is performed as follows.

[0079] First, the interference cancellation matrix B used in the display process + This article explains:

[0080] 8, an interference cancellation matrix generating program 100 is stored in the NVM 72. The processor 70 reads out the interference cancellation matrix generating program 100 from the NVM 72 and executes the read interference cancellation matrix generating program 100 on the RAM 74. The processor 70 generates an interference cancellation matrix B + The interference cancellation matrix generation process is performed to generate the matrix.

[0081] The interference cancellation matrix generation process is realized by the processor 70 operating as a wavelength setting unit 102, a polarization component removal unit 104, an inverse matrix derivation unit 106, a wavelength component removal unit 108, a condition number derivation unit 110, and an interference cancellation matrix determination unit 112 in accordance with the interference cancellation matrix generation program 100.

[0082] 9 as an example, a user or the like issues an instruction to select a wavelength to be used to generate a display image 132 (see FIG. 11 ) from a plurality of wavelengths (hereinafter referred to as a “wavelength selection instruction”) to the reception device 34. When the reception device 34 receives the wavelength selection instruction, it outputs a wavelength selection instruction signal indicating the wavelength selection instruction to the processor 70.

[0083] The wavelength setting unit 102 determines the first transmission wavelength band λ based on the wavelength selection instruction signal input to the processor 70. 1 , second transmission wavelength band λ 2 , and the third transmission wavelength band λ 3 The transmission wavelength band λ used to generate the display image 132 is set from the above.

[0084] The NVM 72 stores in advance an interference matrix A obtained by a user or the like. As an example, the interference matrix A is a matrix with four rows and three columns corresponding to the four types of polarizers 62 and three types of filters 44 (see FIG. 3) described above. That is, the interference matrix A is a matrix with four rows and three columns corresponding to the four types of polarizers 62 and three types of filters 44 (see FIG. 3) described above. 11 ~a 13 , component a 21 ~a 23 , component a 31 ~a 33 and component a 41 ~a 43 The row direction of the interference matrix indicates the polarization angle θ of the polarizer 62, and the column direction of the interference matrix indicates the transmission wavelength band λ of the filter 44.

[0085] In the interference matrix A, the first row represents the polarization angle θ of the polarizer 62A. 1 The second row shows the polarization angle θ of the polarizer 62B. 2 The third row shows the polarization angle θ of the polarizer 62C. 3The fourth row shows the polarization angle θ of the polarizer 62D. 4 In the interference matrix A, the first column indicates the first transmission wavelength band λ of the filter 44A. 1 The second column shows the components related to the second transmission wavelength band λ of the filter 44B. 2 The third column shows the components related to the third transmission wavelength band λ of the filter 44C. 3 The components related to are shown below.

[0086] The polarization component removing unit 104 generates a first interference matrix A based on the interference matrix A stored in the NVM 72. 1 , the second interference matrix A 2 , the third interference matrix A 3 , and the fourth interference matrix A 4 The first interference matrix A 1 is the polarization angle θ of the polarizer 62A 1 The second interference matrix A is obtained by deleting the components related to 2 is the polarization angle θ of the polarizer 62B 2 A third interference matrix A is a matrix obtained by deleting the components related to 3 is the polarization angle θ of the polarizer 62C 3 The fourth interference matrix A is obtained by deleting the components related to 4 is the polarization angle θ of the polarizer 62D 4 is a matrix obtained by deleting the components related to from interference matrix A.

[0087] The inverse matrix deriving unit 106 derives the first interference matrix A 1 A first inverse matrix A, which is the inverse matrix of 1 + and the second interference matrix A 2 A second inverse matrix A 2 + and the third interference matrix A 3 A third inverse matrix A, which is the inverse matrix of 3 + and the fourth interference matrix A 4 A fourth inverse matrix A 4 + The first inverse matrix A 1 + ~Fourth inverse matrix A 4+ are matrices with three rows and three columns. 1 + is component b 11 ~b 13 , component b 21 ~b 23 and component b 31 ~b 33 The inverse matrix A 2 + is component c 11 ~c 13 , component c 21 ~c 23 and component c 31 ~c 33 The inverse matrix A 3 + is the component d 11 ~d 13 , component d 21 ~d 23 , and component d 31 ~d 33 The inverse matrix A 4 + is component e 11 ~e 13 , component e 21 ~e 23 , and component e 31 ~e 33 The row direction of the interference matrix indicates the transmission wavelength band λ of the filter 44, and the column direction of the interference matrix indicates the polarization angle θ of the polarizer 62.

[0088] The wavelength component removing unit 108 removes the components related to the remaining transmission wavelength band λ excluding the transmission wavelength band λ set by the wavelength setting unit 102 from the first inverse matrix A 1 + , the second inverse matrix A 2 + , the third inverse matrix A 3 + , and the fourth inverse matrix A 4 + In the example shown in FIG. 9, the second transmission wavelength band λ 2 and the third transmission wavelength band λ 3 is set by the wavelength setting unit 102 as the transmission wavelength band used to generate the display image 132. 1+ , the second inverse matrix A 2 + , the third inverse matrix A 3 + , and the fourth inverse matrix A 4 + From the first transmission wavelength band λ 1 The components related to (i.e., the components in the first row) have been deleted.

[0089] First inverse matrix A 1 + to the first transmission wavelength band λ 1 The components related to the first interference cancellation matrix B are deleted. 1 + The second inverse matrix A 2 + to the first transmission wavelength band λ 1 The components related to the second interference cancellation matrix B are deleted. 2 + The third inverse matrix A 3 + to the first transmission wavelength band λ 1 The third interference cancellation matrix B 3 + The fourth inverse matrix A 4 + to the first transmission wavelength band λ 1 The fourth interference cancellation matrix B 4 + The first interference cancellation matrix B 1 + ~Fourth interference cancellation matrix B 4 + are matrices with two rows and three columns.

[0090] The condition number derivation unit 110 calculates the first interference cancellation matrix B 1 + and the condition number of the second interference cancellation matrix B 2 + and the third interference cancellation matrix B 3 + and the fourth interference cancellation matrix B 4 + Derive the condition number of and.

[0091] Note that each condition number fluctuates as the relative angle between each polarizer 46 of the pupil division filter 40 and each polarizer 62 of the image sensor 26 changes in accordance with a change in the mounting angle of the lens device 12 with respect to the image capture device body 14. The mounting angle of the lens device 12 with respect to the image capture device body 14 may vary for each image capture device 10 due to manufacturing errors in the screws 20 and / or the screw holes 22.

[0092] The interference cancellation matrix determination unit 112 determines the first interference cancellation matrix B 1 + , the second interference cancellation matrix B 2 + , the third interference cancellation matrix B 3 + , and the fourth interference cancellation matrix B 4 + The interference cancellation matrix determination unit 112 then extracts the interference cancellation matrix with the smallest condition number from the matrix B. The interference cancellation matrix determination unit 112 then determines the extracted interference cancellation matrix as the interference cancellation matrix B + The interference cancellation matrix B is determined as follows: + is stored in the NVM 72.

[0093] Interference cancellation matrix B + is the difference between interference matrix A and interference cancellation matrix B. + In the process of generating the interference cancellation matrix A, the components related to the polarization angle θ of the polarizer 62 and the components related to the transmission wavelength band λ of the filter 44 are deleted. + has fewer components (i.e., fewer rows) than

[0094] Interference cancellation matrix B + is an example of a "first matrix" according to the technology of the present disclosure. The condition number is an example of a "condition number" according to the technology of the present disclosure. The interference matrix A is an example of a "second matrix" according to the technology of the present disclosure. First interference matrix A 1 , the second interference matrix A 2 , the third interference matrix A 3 , and the fourth interference matrix A 4 is an example of a "third matrix" according to the technique of the present disclosure.

[0095] In the example shown in FIG. 9, the first inverse matrix A 1+ , the second inverse matrix A 2 + , the third inverse matrix A 3 + , and the fourth inverse matrix A 4 + From the first transmission wavelength band λ 1 (i.e., the elements in the first row) are deleted, but this is merely an example. For example, the first inverse matrix A 1 + , the second inverse matrix A 2 + , the third inverse matrix A 3 + , and the fourth inverse matrix A 4 + From the second transmission wavelength band λ 2 (i.e., the components in the second row) may be deleted, and the third transmission wavelength band λ 3 The components related to (i.e., the components in the third row) may be deleted. 1 + , the second inverse matrix A 2 + , the third inverse matrix A 3 + , and the fourth inverse matrix A 4 + The component related to the transmission wavelength band λ does not need to be removed from

[0096] Next, the interference cancellation matrix B generated by the interference cancellation matrix generation process is + The display process using the above will be described.

[0097] As an example, as shown in FIG. 10, in the display process, the first output value acquisition unit 82 acquires the output values ​​Y of all physical pixels 58 based on the imaging data 120 input from the image sensor 26 to the processor 70.

[0098] The first interference elimination processor 84 performs interference elimination processing on the output values ​​Y of all the physical pixels 58 acquired by the first output value acquisition unit 82, in order to acquire a display image 132 (see FIG. 11 ), which is a multispectral image for display. Specifically, the first interference elimination processor 84 uses the interference elimination matrix B +and the output values ​​Y of all the physical pixels 58 acquired by the first output value acquisition unit 82 are acquired, and the acquired interference cancellation matrix B + and the output values ​​Y of all the physical pixels 58, the pixel value X of each image pixel is output according to equation (5).

[0099] In this way, the first interference removal processing unit 84 performs interference removal processing, and the imaging data 120 is separated into image data corresponding to the transmission wavelength band λ set based on a wavelength selection instruction (see FIG. 9) by a user or the like. In the example shown in FIG. 10, the wavelength setting unit 102 (see FIG. 9) sets the second transmission wavelength band λ 2 and the third transmission wavelength band λ 3 In response to the setting of the second transmission wavelength band λ , the imaging data 120 is separated into second image data and third image data. The second image data is the second transmission wavelength band λ , which is set by the wavelength setting unit 102. 2 The brightness value of the light X λ2 The third image data is image data representing the third transmission wavelength band λ 2 set by the wavelength setting unit 102. 3 The brightness value of the light X λ3 The interference removal process in the display process is an example of the "reduction process," "thinning process," and "arithmetic process" according to the techniques of the present disclosure.

[0100] In the display process, the interference cancellation matrix A used in the storage process is + The interference cancellation matrix B has fewer components than +By using the above, the amount of information processed on the imaging data 120 is reduced compared to the storage process. That is, in the interference removal process of the display process, polarization information and wavelength information are thinned out from the information included in the imaging data 120. The polarization information is information corresponding to the component related to the polarization angle θ that was removed by the polarization component removal unit 104 (see FIG. 9 ). The wavelength information is information corresponding to the component related to the transmission wavelength band λ that was removed by the wavelength component removal unit 108 (see FIG. 9 ). In this way, by thinning out the polarization information and wavelength information from the information included in the imaging data 120, the amount of information processed on the imaging data 120 is reduced in the interference removal process of the display process compared to the storage process. The polarization information is an example of "information" and "polarization information" according to the technology of the present disclosure. The wavelength information is an example of "information" and "wavelength information" according to the technology of the present disclosure.

[0101] 11 , the first image data generation unit 86 generates display image data 122, which is multispectral image data, by combining the image data generated by the first interference removal processing unit 84 through interference removal processing. In the example shown in FIG. 11 , the display image data 122 is generated by combining the second image data and the third image data. The display image data 122 is image data that represents a display image 132, which is a multispectral image. The display image data 122 is output to the display 36, and the display image 132 is displayed on the display 36.

[0102] In the examples shown in FIGS. 10 and 11, the second transmission wavelength band λ 2 and the third transmission wavelength band λ 3 is set as the transmission wavelength band λ to be used for generating the display image 132, the imaging data 120 is separated into the second image data and the third image data. 1 and the second transmission wavelength band λ 2 is set as the transmission wavelength band λ to be used for generating the display image 132, the imaging data 120 may be separated into first image data and second image data.1 and the third transmission wavelength band λ 3 is set as the transmission wavelength band λ to be used for generating the display image 132, the imaging data 120 may be separated into first image data and third image data.

[0103] Next, the operation of the imaging device 10 according to this embodiment will be described with reference to FIGS.

[0104] First, an example of the flow of the display processing of the image processing performed by the processor 70 of the imaging device 10 will be described with reference to FIG.

[0105] 12 , first, in step ST10, the first output value acquisition unit 82 acquires the output values ​​Y of all physical pixels 58 based on the imaging data 120 input from the image sensor 26 to the processor 70 (see FIG. 10 ). After the processing of step ST10 is executed, the display processing proceeds to step ST12.

[0106] In step ST12, the first interference cancellation processor 84 applies the interference cancellation matrix B + (See FIG. 10 ). As a result, the imaging data 120 is separated into image data corresponding to the transmission wavelength band λ set based on the wavelength selection instruction (See FIG. 9 ) by the user or the like. After the processing of step ST12 is executed, the display processing proceeds to step ST14.

[0107] In step ST14, the first image data generation unit 86 generates display image data 122, which is multispectral image data, by combining the image data generated by the interference removal process in step ST12 (see FIG. 11 ). The display image data 122 is output to the display 36. As a result, a display image 132 represented by the display image data 122 is displayed on the display 36. After the process of step ST14 is executed, the display process ends.

[0108] Next, an example of the flow of the storage process of the image processing performed by the processor 70 of the imaging device 10 will be described with reference to FIG.

[0109] 13 , first, in step ST20, the second output value acquisition unit 92 acquires the output values ​​Y of all physical pixels 58 based on the imaging data 120 input from the image sensor 26 to the processor 70 (see FIG. 6 ). After the processing of step ST20 is executed, the storage processing proceeds to step ST22.

[0110] In step ST22, the second interference cancellation processor 94 applies an interference cancellation matrix A to the output values ​​Y of all the physical pixels 58 acquired in step ST20. + (See FIG. 6 ). As a result, the imaging data 120 is separated into image data corresponding to a plurality of transmission wavelength bands λ. After the processing of step ST22 is performed, the storage processing proceeds to step ST24.

[0111] In step ST24, the second image data generation unit 96 generates storage image data 124, which is multispectral image data, by combining the image data generated by the interference removal process in step ST22 (see FIG. 7). The storage image data 124 is stored in the NVM 72. After the process of step ST24 is executed, the storage process ends.

[0112] Next, referring to FIG. 14, the interference cancellation matrix B used in the display process is + An example of the flow of the interference cancellation matrix generation process for generating the matrix will be described.

[0113] In the interference cancellation matrix generation process shown in FIG. 14, first, in step ST30, the wavelength setting unit 102 sets the first transmission wavelength band λ based on the wavelength selection instruction signal input to the processor 70. 1 , second transmission wavelength band λ 2 , and the third transmission wavelength band λ 3 The transmission wavelength band λ used to generate the display image 132 is set from (see FIG. 9). After the process of step ST30 is executed, the interference cancellation matrix generation process proceeds to step ST32.

[0114] In step ST32, the polarization component removing unit 104 obtains the polarization angle θ of the polarizer 62A from the interference matrix A stored in the NVM 72. 1 The first interference matrix A is obtained by deleting the components related to 1 (See FIG. 9 ). The polarization component removing unit 104 also generates the polarization angle θ of the polarizer 62B from the interference matrix A stored in the NVM 72. 2 The second interference matrix A is obtained by deleting the components related to 2 (See FIG. 9 ). The polarization component removing unit 104 also generates the polarization angle θ of the polarizer 62C from the interference matrix A stored in the NVM 72. 3 The third interference matrix A is obtained by deleting the components related to 3 Furthermore, the polarization component removing unit 104 generates the polarization angle θ of the polarizer 62D from the interference matrix A stored in the NVM 72. 4 The fourth interference matrix A is obtained by deleting the components related to 4 After the process of step ST32 is executed, the interference cancellation matrix generation process proceeds to step ST34.

[0115] In step ST34, the inverse matrix deriving unit 106 calculates the first interference matrix A generated in step ST32 as 1 , the second interference matrix A 2 , the third interference matrix A 3 , and the fourth interference matrix A 4 , the first inverse matrix A 1 + , the second inverse matrix A 2 + , the third inverse matrix A 3 + , and the fourth inverse matrix A 4 + (See FIG. 9) After the process of step ST34 is executed, the interference cancellation matrix generation process proceeds to step ST36.

[0116] In step ST36, the wavelength component deleting unit 108 subtracts the components related to the remaining transmission wavelength band λ excluding the transmission wavelength band λ set in step ST30 from the first inverse matrix A 1+ , the second inverse matrix A 2 + , the third inverse matrix A 3 + , and the fourth inverse matrix A 4 + (See FIG. 9). 1 + , the second inverse matrix A 2 + , the third inverse matrix A 3 + , and the fourth inverse matrix A 4 + From the first interference cancellation matrix B 1 + , the second interference cancellation matrix B 2 + , the third interference cancellation matrix B 3 + , and the fourth interference cancellation matrix B 4 + After the process of step ST36 is executed, the interference cancellation matrix generation process proceeds to step ST38.

[0117] In step ST38, the condition number deriving unit 110 calculates the first interference cancellation matrix B 1 + and the condition number of the second interference cancellation matrix B 2 + and the third interference cancellation matrix B 3 + and the fourth interference cancellation matrix B 4 + After the process of step ST38 is executed, the interference cancellation matrix generation process proceeds to step ST40.

[0118] In step ST40, the interference cancellation matrix determination unit 112 determines the first interference cancellation matrix B obtained in step ST36 as 1 + , the second interference cancellation matrix B 2 + , the third interference cancellation matrix B 3 + , and the fourth interference cancellation matrix B 4 +The interference cancellation matrix with the smallest condition number derived in step ST38 is then used as the interference cancellation matrix B + The interference cancellation matrix B is determined as follows: + is stored in the NVM 72. After the process of step ST40 is executed, the interference cancellation matrix generation process ends.

[0119] The imaging control method described above as the operation of the imaging device 10 is an example of an "information processing method" according to the technology of the present disclosure.

[0120] As described above, in the imaging device 10 according to this embodiment, the processor 70 performs display processing and storage processing related to multispectral image data on the imaging data 120 from the image sensor 26. The display processing includes an interference removal process that reduces the amount of information in the imaging data 120 more than the storage processing. Therefore, the load on the processor 70 can be reduced compared to, for example, when the display processing is performed on the imaging data 120 without reducing the amount of information.

[0121] The interference removal process in the display process includes a thinning process that thins out information from the imaging data 120. Therefore, the interference removal process in the display process can reduce the amount of information processed on the imaging data 120 more than the interference removal process in the storage process.

[0122] The information to be thinned out from the imaging data 120 includes polarization information contained in the imaging data 120. Therefore, the interference cancellation matrix B used in the interference cancellation process in the display process is + The number of rows of the interference cancellation matrix A + The number of lines can be less than the number of lines in the

[0123] The information to be thinned out from the imaging data 120 includes wavelength information contained in the imaging data 120. Therefore, this also results in the interference cancellation matrix B + The number of rows of the interference cancellation matrix A + The number of lines can be less than the number of lines in the

[0124] The interference cancellation process in the display process is performed using the interference cancellation matrix B + Therefore, the interference cancellation matrix B + In comparison with the case where interference removal processing is performed regardless of the number of conditions, a pixel value X with high accuracy can be obtained.

[0125] The processor 70 removes components corresponding to polarization information from the interference matrix A that indicates the relationship between the imaging data 120 and the multispectral image data, thereby obtaining a first interference matrix A 1 ~ Fourth Interference Matrix A 4 and generate an interference cancellation matrix B + is the first interference matrix A 1 ~ Fourth Interference Matrix A 4 A first inverse matrix A, which is the inverse matrix of 1 + ~Fourth inverse matrix A 4 + Therefore, the components corresponding to the polarization information are not removed from the interference matrix A, and the interference cancellation matrix B + Compared to when the interference cancellation matrix B + The number of lines can be reduced.

[0126] The image data 120 is data output from all the physical pixels 58 of the image sensor 26. Therefore, the display process and the storage process can be performed in parallel.

[0127] The display and storage processes are performed in parallel, so that a display image 132 and a storage image 134 can be obtained from the same frame.

[0128] The display process and the storage process are processes for generating multispectral image data based on the imaging data 120. Therefore, it is possible to obtain a display image 132 and a storage image 134, which are multispectral images.

[0129] In the above embodiment, in the display process, the interference cancellation matrix A used in the storage process is + The interference cancellation matrix B has fewer components than +By using the above, polarization information corresponding to the component related to the polarization angle θ deleted by the polarization component deletion unit 104 (see FIG. 9 ) and wavelength information corresponding to the component related to the transmission wavelength band λ deleted by the wavelength component deletion unit 108 (see FIG. 9 ) are thinned out from the information included in the imaging data 120. As a result, the amount of information processed for the imaging data 120 in the display process is reduced compared to the storage process.

[0130] However, the display process may be performed as follows. In the example shown in FIG. 15, the display process uses the interference cancellation matrix A + The interference removal process is performed on the output values ​​Y of all the physical pixels 58 using the first transmission wavelength band λ 1 a first image 140 corresponding to the second transmission wavelength band λ 2 and a second image 142 corresponding to a third transmission wavelength band λ 3 A third image 144 corresponding to the first image 140, the second image 142, and the third image 144 is obtained. Subsequently, image pixels are thinned out for the first image 140, the second image 142, and the third image 144. For example, the image pixels are thinned out at equal intervals. Then, the first image 140, the second image 142, and the third image 144 formed by the extracted image pixels are combined to generate a multispectral image.

[0131] 15, in the display process, image pixels are thinned out for the first image 140, the second image 142, and the third image 144, so that the amount of information processed for the captured image data 120 can be reduced compared to the storage process. + Therefore, the interference removal process can be shared between the display process and the storage process.

[0132] Although not shown, in the display process, the interference cancellation matrix A used in the storage process is + The interference cancellation matrix B has fewer components than +The interference removal process may be performed on the output values ​​Y of all physical pixels 58 using the above equation, and image pixels may be thinned out from the first image 140, the second image 142, and the third image 144 obtained by the interference removal process. Then, the first image 140, the second image 142, and the third image 144 formed by the extracted image pixels may be combined to generate a multispectral image. Even in such an example, the display process can reduce the amount of information processed on the imaging data 120 compared to the storage process.

[0133] Furthermore, in the above embodiment, when an image capture instruction from a user or the like is accepted by the acceptance device 34, the display process and the storage process are performed in parallel. However, if an image capture instruction is not accepted by the acceptance device 34, the display process may be repeatedly performed, and if an image capture instruction is accepted by the acceptance device 34, the storage process may be performed without the display process. That is, the display process and the storage process may be performed selectively. Specifically, the display process and the storage process may be performed for different frames. In this case, too, the load on the processor 70 can be reduced compared to, for example, when the display process is performed without reducing the amount of information on the image data 120.

[0134] In the above embodiment, the output values ​​Y of all physical pixels 58 are acquired in the display process and the storage process. That is, in the display process and the storage process, a read process is performed on all physical pixels 58. However, for example, when the display process and the storage process are selectively performed, the display process may thin out some physical pixels 58 from the photoelectric conversion elements 48 of the image sensor 26, as shown in FIGS. 16 to 22 , for example.

[0135] 16, a plurality of physical pixels 58 form a plurality of pixel blocks 60. Each pixel block 60 is formed by a total of nine physical pixels 58, three vertically and three horizontally. In each pixel block 60, three types of polarizers 62 (i.e., polarizer 62A, polarizer 62B, and polarizer 62C) are assigned to the nine physical pixels 58. The polarizer 62A has a first polarization angle θ 1is set to 0°, and polarizer 62B is set to a second polarization angle θ 2 The polarizer 62C is a polarizer with a third polarization angle θ 3 is the polarizer set at 120°.

[0136] The polarizers 62 in each row are arranged in the horizontal direction of the photoelectric conversion element 48 in the order of polarizer 62A, polarizer 62B, and polarizer 62C. The multiple polarizers 62 are arranged such that the first polarizer 62 in each row alternates between polarizer 62A, polarizer 62C, and polarizer 62B in each row. As a result, the multiple polarizers 62A are arranged in a series diagonally across the photoelectric conversion element 48. Similarly, the multiple polarizers 62B are arranged in a series diagonally across the photoelectric conversion element 48, and the multiple polarizers 62C are also arranged in a series diagonally across the photoelectric conversion element 48. In the example shown in FIG. 16 , the B filter 64A, G filter 64B, and R filter 64C (see FIG. 4 ) are arranged in the same manner as in FIG. 4 .

[0137] In the display process, three physical pixels 58 (i.e., physical pixel 58A, physical pixel 58B, and physical pixel 58C) arranged at the same position in each pixel block 60 are selected from the multiple physical pixels 58. Specifically, in each pixel block 60, one physical pixel 58A corresponding to polarizer 62A, one physical pixel 58B corresponding to polarizer 62B, and one physical pixel 58C corresponding to polarizer 62C are selected from the nine physical pixels 58. Physical pixel 58A is adjacent to physical pixel 58B and physical pixel 58C in the vertical and horizontal directions of the photoelectric conversion element 48. In the display process, the output value Y of the selected physical pixels 58 (i.e., the physical pixels 58 excluding the thinned physical pixels 58) is read out. The pixel block 60 is an example of a "first pixel block" according to the technology of the present disclosure. The physical pixels 58 are an example of a "first pixel" according to the technology of the present disclosure.

[0138] In the example shown in FIG. 17 , three physical pixels 58 are selected from the nine physical pixels 58 in each pixel block 60, differently from the example shown in FIG. 16 . That is, in the example shown in FIG. 17 , physical pixels 58B and 58C, which are positioned away from physical pixel 58A, are selected. As a result, physical pixels 58A, 58B, and 58C, which are arranged diagonally in the photoelectric conversion element 48, are selected. Then, in the display process, the output value Y of the selected physical pixels 58 (i.e., the physical pixels 58 excluding the thinned physical pixels 58), is read out. The pixel block 60 is an example of a "first pixel block" according to the technology of the present disclosure. The physical pixels 58 are an example of a "first pixel" according to the technology of the present disclosure.

[0139] In the example shown in FIG. 18 , the selected physical pixels 58 are different from those in the example shown in FIG. 16 . That is, in the example shown in FIG. 18 , physical pixels 58 corresponding to different types of polarizers 62 are selected in adjacent pixel blocks 60. Specifically, physical pixels 58 are selected from each pixel block 60 such that as the pixel block 60 is shifted in the horizontal direction of the photoelectric conversion element 48, the selected physical pixels 58 are shifted in the vertical direction of the photoelectric conversion element 48, and as the pixel block 60 is shifted in the vertical direction of the photoelectric conversion element 48, the selected physical pixels 58 are shifted in the horizontal direction of the photoelectric conversion element 48. Then, in the display process, the output value Y of the selected physical pixels 58 (i.e., the physical pixels 58 excluding the thinned physical pixels 58) is read out. The pixel block 60 is an example of a "second pixel block" according to the technology of the present disclosure. The physical pixels 58 are an example of a "second pixel" according to the technology of the present disclosure.

[0140] In the example shown in FIG. 19 , the physical pixels 58 selected are different from those in the example shown in FIG. 16 . That is, in the example shown in FIG. 19 , a plurality of pixel lines 150 are formed by a plurality of pixel blocks 60, and pixel lines 150 are selected one by one from the plurality of pixel lines 150, with a row gap between each pixel line 150. Then, in the display process, the output values ​​Y of the physical pixels 58 included in the selected pixel lines 150 (i.e., the pixel lines 150 excluding the thinned pixel lines 150) are read out. The pixel blocks 60 are an example of a "third pixel block" according to the technology of the present disclosure. The pixel lines 150 are an example of a "pixel line" according to the technology of the present disclosure. The physical pixels 58 are an example of a "third pixel" according to the technology of the present disclosure.

[0141] 20 differs from the example shown in FIG. 16 in the arrangement of the polarizers 62A, 62B, and 62C. That is, in the example shown in FIG. 20 , the polarizers 62 in each row are arranged in the horizontal direction of the photoelectric conversion element 48 in the order of polarizer 62A, polarizer 62B, and polarizer 62C. The first polarizer 62 in each row is the same polarizer 62A. As a result, the multiple polarizers 62A are arranged in a row in the vertical direction of the photoelectric conversion element 48. Similarly, the multiple polarizers 62B are arranged in a row in the vertical direction of the photoelectric conversion element 48, and the multiple polarizers 62C are also arranged in a row in the vertical direction of the photoelectric conversion element 48.

[0142] Furthermore, the arrangement of the B filters 64A, G filters 64B, and R filters 64C in the example shown in Fig. 20 is different from the example shown in Fig. 16. That is, in the example shown in Fig. 20, the B filters 64A, G filters 64B, and R filters 64C are provided on a row-by-row basis, and the B filters 64A, G filters 64B, and R filters 64C are arranged in the vertical direction of the photoelectric conversion element 48 in the order of B filters 64A, G filters 64B, and R filters 64C.

[0143] Furthermore, in the example shown in Fig. 20 , the physical pixels 58 selected are different from those in the example shown in Fig. 16 . That is, in the example shown in Fig. 20 , pixel blocks 60 are selected from a plurality of pixel blocks 60 at intervals of one pixel block 60 in the vertical and horizontal directions of the photoelectric conversion element 48. Then, in the display process, the output values ​​Y of the physical pixels 58 included in the selected pixel blocks 60 (i.e., the pixel blocks 60 excluding the thinned-out pixel blocks 60) are read out. The pixel blocks 60 are an example of a "fourth pixel block" according to the technology of the present disclosure. The physical pixels 58 are an example of a "fourth pixel" according to the technology of the present disclosure.

[0144] In the example shown in Fig. 21, the arrangement of the B filter 64A, G filter 64B, and R filter 64C is different from the example shown in Fig. 16. That is, in the example shown in Fig. 21, the filters 64 in each row are arranged in the order of B filter 64A, G filter 64B, and R filter 64C in the horizontal direction of the photoelectric conversion element 48. Furthermore, the multiple filters 44 are arranged such that the first filter 64 in each row alternates between B filter 64A, R filter 64C, and G filter 64B in that order.

[0145] 21 , similar to the example shown in FIG. 20 , pixel blocks 60 are selected from the plurality of pixel blocks 60 at intervals of one pixel block 60 in the vertical and horizontal directions of the photoelectric conversion element 48. Then, in the display process, the output values ​​Y of the physical pixels 58 included in the selected pixel blocks 60 (i.e., the pixel blocks 60 excluding the thinned-out pixel blocks 60) are read out. The pixel blocks 60 are an example of a "fourth pixel block" according to the technology of the present disclosure. The physical pixels 58 are an example of a "fourth pixel" according to the technology of the present disclosure.

[0146] In the example shown in Figure 22, a polarization sub-block is formed by two vertically and two horizontally, a total of four physical pixels 58. In each polarization sub-block, three types of polarizers 62 (i.e., polarizer 62A, polarizer 62B, and polarizer 62C) are assigned to the four physical pixels 58. A filter block is formed by four polarization sub-blocks, two vertically and two horizontally. A pixel block 60 is formed by four filter blocks, two vertically and two horizontally. A B filter 64A, a G filter 64B, and an R filter 64C are assigned to the four filter blocks.

[0147] 22 , pixel blocks 60 are selected from the plurality of pixel blocks 60 at intervals of one pixel block 60 in the vertical and horizontal directions of the photoelectric conversion element 48. Then, in the display process, the output values ​​Y of the physical pixels 58 included in the selected pixel blocks 60 (i.e., the pixel blocks 60 excluding the thinned-out pixel blocks 60) are read out. The pixel blocks 60 are an example of a "fourth pixel block" according to the technology of the present disclosure. The physical pixels 58 are an example of a "fourth pixel" according to the technology of the present disclosure.

[0148] In the examples shown in Figures 16 to 22, the display process involves a greater amount of thinning of physical pixels 58 on the image sensor 26 than the storage process, so the load on the processor 70 in the display process can be reduced compared to, for example, when the amount of thinning of physical pixels 58 is the same in the display process and the storage process.

[0149] In the examples shown in Figures 16 to 22, the image sensor 26 is an example of a "second image sensor" according to the technology of the present disclosure, the display process is an example of a "third process" according to the technology of the present disclosure, and the storage process is an example of a "fourth process" according to the technology of the present disclosure.

[0150] Furthermore, in the above embodiment, the display process and the storage process are performed by a common processor 70, but the processor 70 may have a processor for the display process and a processor for the storage process.

[0151] Furthermore, the technology according to the above embodiment may be applied to a type of device other than the imaging device 10 (hereinafter referred to as "other device"). Furthermore, when the technology according to the above embodiment is applied to other device, the other device may include a sensor other than the image sensor 26 (for example, a LiDAR or a distance sensor). Furthermore, in other device, the display process and the storage process may be performed on output data output from a sensor other than the image sensor 26. Furthermore, in other device, first process and second process other than the display process and the storage process may be performed.

[0152] Furthermore, either the first process or the second process does not have to be a process for generating multispectral image data.

[0153] Furthermore, in the above embodiment, the information that is thinned out from the imaging data 120 is polarization information and wavelength information, but it may be information other than polarization information and wavelength information.

[0154] Furthermore, in the above embodiment, by thinning out information from the image data 120, the amount of information processed on the image data 120 in the display process is reduced compared to the storage process, but the amount of information processed on the image data 120 in the display process may be reduced compared to the storage process by processing other than the thinning process.

[0155] In the above embodiment, the interference cancellation matrix B + is generated by the imaging device 10, but may be generated by an external device other than the imaging device 10.

[0156] Furthermore, in the above embodiment, as an example, a multispectral image has been described in which the multispectral image is generated based on light dispersed into three transmission wavelength bands λ, but three transmission wavelength bands λ is merely an example, and the image capturing device 10 may be a multispectral camera that can capture an image of a subject with a wavelength resolution higher than that of a multispectral camera that can capture light dispersed into three transmission wavelength bands λ.

[0157] Furthermore, among the above multiple modified examples, modified examples that can be combined may be combined as appropriate.

[0158] Furthermore, in the above embodiment, processor 70 is exemplified, but instead of processor 70, or together with processor 70, at least one other CPU, at least one GPU, and / or at least one TPU may be used.

[0159] Furthermore, in the above embodiment, an example has been described in which the image processing program 80 and the interference cancellation matrix generation program 100 are stored in the NVM 72, but the technology of the present disclosure is not limited to this. For example, the image processing program 80 and / or the interference cancellation matrix generation program 100 may be stored in a portable, non-transitory, computer-readable storage medium (hereinafter simply referred to as a "non-transitory storage medium") such as an SSD or a USB memory. The image processing program 80 and / or the interference cancellation matrix generation program 100 stored in the non-transitory storage medium may be installed in the computer 32 of the imaging device 10.

[0160] In addition, the image processing program 80 and / or the interference cancellation matrix generation program 100 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the image processing program 80 and / or the interference cancellation matrix generation program 100 may be downloaded and installed on the computer 32 in response to a request from the imaging device 10.

[0161] Furthermore, it is not necessary to store all of the image processing program 80 and / or the interference cancellation matrix generation program 100 in a storage device of another computer or server device connected to the imaging device 10, or in the NVM 72; only part of the image processing program 80 and / or the interference cancellation matrix generation program 100 may be stored therein.

[0162] Furthermore, although the imaging device 10 has a built-in computer 32 , the technology of the present disclosure is not limited to this. For example, the computer 32 may be provided outside the imaging device 10 .

[0163] Furthermore, in the above embodiment, the computer 32 includes the processor 70, the NVM 72, and the RAM 74, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 32. Furthermore, a combination of a hardware configuration and a software configuration may be used instead of the computer 32.

[0164] Furthermore, the following various processors can be used as hardware resources for executing the various processes described in the above embodiments. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing various processes by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, and ASICs, which are processors with a circuit configuration designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute various processes.

[0165] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0166] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes various processes. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute various processes, on a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.

[0167] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The above-described gaze detection process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0168] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0169] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0170] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A processor is provided. The processor performs a first process and a second process related to multispectral image data on the output data from the sensor; the first processing includes a reduction processing for reducing an amount of information for the output data more than the second processing, the first processing is a display processing for generating image data for display based on the output data; The second process is a storage process for generating image data for storage based on the output data.

2. The reduction process includes a thinning process of thinning out the information from the output data. The information processing device according to claim 1 .

3. the sensor is a first image sensor; The output data is imaging data output from the first image sensor. The information processing device according to claim 1 .

4. The imaging data is data output from the first image sensor without reducing the electrical signal. The information processing device according to claim 3 .

5. The information includes polarization information included in the imaging data. The information processing device according to claim 3 .

6. The information includes wavelength information included in the imaging data. The information processing device according to claim 3 .

7. the reduction process includes a calculation process using a first matrix, The arithmetic processing is performed based on the condition number of the first matrix. The information processing device according to claim 1 .

8. the processor generates a third matrix by deleting an element corresponding to the information from a second matrix indicating a relationship between the output data and the multispectral image data; The first matrix is ​​generated based on the inverse of the third matrix. The information processing device according to claim 7.

9. The display process and the storage process are performed in parallel. The information processing device according to claim 1 .

10. The display process and the storage process are selectively performed. The information processing device according to claim 1 .

11. The first process and / or the second process is a process for generating the multispectral image data based on the output data. The information processing device according to claim 1 .

12. The sensor is a second image sensor, the processor selectively performs a third process and a fourth process which are read processes from the second image sensor; The third process has a larger amount of pixel thinning for the second image sensor than the fourth process. The information processing device according to claim 1 .

13. The second image sensor has a plurality of first pixel blocks, Each of the first pixel blocks is formed by a plurality of first pixels, A plurality of types of polarizers are assigned to the plurality of first pixels, The third process includes a process of selecting, in each of the first pixel blocks, a first pixel corresponding to each of the polarizers from the plurality of first pixels. The information processing device according to claim 12.

14. The second image sensor has a plurality of second pixel blocks, Each of the second pixel blocks is formed by a plurality of second pixels, A plurality of types of polarizers are assigned to the plurality of second pixels, The third process includes a process of selecting second pixels corresponding to different types of polarizers in adjacent second pixel blocks from the plurality of second pixels. The information processing device according to claim 12.

15. The second image sensor has a plurality of third pixel blocks, Each of the third pixel blocks is formed by a plurality of third pixels, A plurality of types of polarizers are assigned to the plurality of third pixels, The third process includes a process of selecting pixel lines at intervals from a plurality of pixel lines formed by the plurality of third pixel blocks. The information processing device according to claim 12.

16. The second image sensor has a plurality of fourth pixel blocks, Each of the fourth pixel blocks is formed by a plurality of fourth pixels, A plurality of types of polarizers are assigned to the plurality of fourth pixels, The third process includes a process of selecting fourth pixel blocks at intervals from the plurality of fourth pixel blocks. The information processing device according to claim 12.

17. A method for detecting a multispectral image data by performing a first process and a second process on output data from a sensor, the first processing includes a reduction processing for reducing an amount of information for the output data more than the second processing, the first processing is a display processing for generating image data for display based on the output data; The second process is a storage process for generating image data for storage based on the output data. Information processing methods.

18. A program for causing a computer to execute a specific process, comprising: the identifying process comprises performing a first process and a second process related to multispectral image data on output data from the sensor; the first processing includes a reduction processing for reducing an amount of information for the output data more than the second processing, the first processing is a display processing for generating image data for display based on the output data; The second process is a storage process for generating image data for storage based on the output data. program.