Signal processing device, signal processing method

Abstract

A signal processing device according to the present technology includes: a first brightness control unit for controlling brightness of a first image on the basis of a detection result of the first image; a second image generation unit for generating a second image different from the first image on the basis of the first image, the brightness of which is controlled by the first brightness control unit; and a second brightness control unit for controlling brightness of the second image on the basis of a detection result of the second image. A first control value used for the brightness control by the first brightness control unit is used for the brightness control by the second brightness control unit, or a second control value used for the brightness control by the second brightness control unit is used for brightness control by the first brightness control unit.

Description

Signal processing device, signal processing method

[0001] This technology relates to a signal processing device and a method thereof, and more particularly to a technical field related to image brightness control in a signal processing system in which a second image is generated based on a first image, the brightness of the first image is controlled based on the detection result of the first image, and the brightness of the second image is controlled based on the detection result of the second image.

[0002] One image signal processing system involves generating a second image based on a first image, controlling the brightness of the first image based on its detection result, and controlling the brightness of the second image based on its detection result. For example, the relationship between the first and second images could be that of a RAW image obtained as the output image of an image sensor and an RGB image generated based on that RAW image.

[0003] Regarding related prior art, Patent Document 1 below can be cited. Patent Document 1 below discloses a signal processing device comprising a signal processing unit that performs signal processing on at least one image from among multiple types of images generated based on the output of a light-receiving sensor having pixels having light-receiving elements arranged in two dimensions, and a control unit that controls the signal processing parameters in the signal processing unit so that the multiple types of images take on an appropriate form.

[0004] Japanese Patent Publication No. 2023-111625

[0005] In a signal processing system where a second image is generated based on the first image as described above, and the brightness control of the first and second images is performed based on the detection results of the first and second images, respectively, the brightness control of the first image may not be appropriate for the brightness control of the second image, and as a result, it may not be possible to properly control the brightness of the second image.

[0006] This technology was developed in view of the above-mentioned problems, and aims to ensure that the brightness control of the second image is performed appropriately in a signal processing system in which a second image is generated based on a first image, and the brightness control of the first and second images is performed based on the detection results of the first and second images, respectively.

[0007] The signal processing device according to this technology comprises: a first brightness control unit that controls the brightness of the first image based on the detection result of the first image; a second image generation unit that generates a second image different from the first image based on the first image whose brightness has been controlled by the first brightness control unit; and a second brightness control unit that controls the brightness of the second image based on the detection result of the second image, wherein a first control value used for brightness control by the first brightness control unit is used for brightness control by the second brightness control unit, or a second control value used for brightness control by the second brightness control unit is used for brightness control by the first brightness control unit. According to the above configuration, in a signal processing system in which a second image is generated based on the first image and the brightness control of the first and second images is performed based on the detection result of the first image and the detection result of the second image, respectively, the brightness control by the second brightness control unit is performed as control that reflects the first control value used for brightness control by the first brightness control unit, or the brightness control by the first brightness control unit is performed as control that reflects the second control value used for brightness control by the second brightness control unit. This makes it possible to adjust the brightness control of the second image or the brightness control of the first image so that the brightness control of the second image is performed appropriately.

[0008] This is a block diagram showing an example configuration of a signal processing device as a first example in the first embodiment of this technology. This is a schematic diagram showing an example configuration of the pixel array section of a spectroscopic sensor. This is an explanatory diagram of linear matrix processing for obtaining M wavelength band images. This is an explanatory diagram of the principle by which brightness hunting occurs. This is a diagram for explaining a brightness control method as a first example in the first embodiment. This is a flowchart showing an example of a specific processing procedure for realizing the brightness control method as a first example in the first embodiment. This is a block diagram showing an example configuration of a signal processing device as a second example in the first embodiment. This is a flowchart showing an example of a specific processing procedure for realizing the brightness control method as a second example in the first embodiment. This is an explanatory diagram of an example configuration of a signal processing device in which the second detection section performs detection on the second image before gain adjustment. This is a block diagram showing an example configuration of a signal processing device in which brightness control is performed on an Mch wavelength band image as the first embodiment. This is a block diagram showing an example configuration of a signal processing device as a second embodiment. This is a flowchart showing an example of a specific processing procedure for realizing the brightness control method as the second embodiment. This is a block diagram showing an example configuration of a signal processing device as a third embodiment. This is a schematic diagram showing an example configuration of the pixel array section of a polarization sensor. This figure illustrates the schematic cross-sectional structure of a pixel in a polarization sensor. This is a block diagram illustrating an example of the internal configuration of the polarization image generation unit. This is an explanatory diagram of the image organization process by the image organization unit and the demosaicing process by the demosaicing unit. This is an explanatory diagram of the process of the polarization state estimation unit. This is an explanatory diagram of the various polarization image generation methods by the polarization image generation processing unit. This is a flowchart showing an example of the processing procedure performed by the first brightness control unit and the second brightness control unit in the signal processing device as a third embodiment.

[0009] The embodiments of this technology will be described below in the following order, with reference to the attached drawings. <1. First Embodiment> (1-1. First Example) (1-2. Second Example) (1-3. Various Other Examples) <2. Second Embodiment> <3. Third Embodiment> <4. Modified Examples> <5. Summary of Embodiments> <6. This Technology>

[0010] <1. First Embodiment> (1-1. First Example) Figure 1 is a block diagram showing an example of the configuration of the signal processing device 1 as a first example in the first embodiment. Hereinafter, the embodiments will describe an example in which the signal processing device according to this technology is configured as an imaging device. An imaging device means a device that obtains an image of a subject. In this specification, "imaging" broadly means obtaining image data that captures a subject. The image data referred to here is a general term for data consisting of multiple pixel data, and the pixel data is a broad concept that includes not only data that shows information on the amount of light received from the subject, but also data that shows, for example, the distance to the subject, the polarization information of the subject, and temperature information. In other words, the "image data" (imaging image data) obtained by "imaging" includes data as a grayscale image that shows information on the amount of light received for each pixel, data as a distance image that shows information on the distance to the subject for each pixel, data as a polarization image that shows polarization information of incident light for each pixel, and data as a thermal image that shows temperature information for each pixel.

[0011] In the first and second embodiments, the signal processing device is configured as a multispectral camera in which a spectroscopic sensor is used as a light-receiving sensor for obtaining captured images. A spectroscopic sensor refers to a light-receiving sensor for obtaining multiple wavelength band images, which are wavelength characteristic analysis images of light from a subject. The captured images, as each wavelength band image, belong to the category of grayscale images, that is, images that show information on the amount of light received for each pixel.

[0012] As shown in Figure 1, the signal processing device 1 comprises an imaging optical system 2, a spectral sensor 3, a demosaicing unit 4, a wavelength analysis image generation unit 5, an RGB image generation unit 6, a first detection unit 7, a second detection unit 8, a first brightness control unit 9, a second brightness control unit 10, an image recognition processing unit 11, and a display unit 12.

[0013] The imaging optical system 2 is an optical system for guiding light from the subject to the spectral sensor 3, and is composed of various optical components for imaging, such as a cover lens provided at the front end, various lenses such as a focus lens and a zoom lens, and an aperture. It should be noted that the imaging optical system 2 does not necessarily have a zoom lens; it can also be configured as a fixed-focus optical system. Furthermore, the imaging optical system 2 can also be configured to include a mechanical shutter.

[0014] The spectroscopic sensor 3 includes a pixel array section 31, a pixel driving section 32, a gain adjustment section 33, and an ADC (Analog to Digital Converter) 34. The pixel array section 31 is formed by arranging multiple pixels Pm, each having a photoreceiving element (photoelectric conversion element), in a two-dimensional array, and it is possible to obtain a light-receiving signal for each pixel Pm. Although not shown in the figures, the pixel array section 31 has various control lines for driving the pixels, such as control lines for controlling the start timing of charge accumulation and the reset timing of charge, and row selection lines for selecting the row (horizontal line) to be read out of the light-receiving signal. The pixel driving section 32 can control the exposure period of the pixels Pm and the readout operation of the light-receiving signal by supplying signals to these control lines.

[0015] The readout signal from the pixel array unit 31 is supplied to the gain adjustment unit 33 for analog gain adjustment, and then digitally sampled by the ADC 34. This digital sampling yields an image (image data) as digital data, which shows the light reception intensity for each pixel Pm as a digital value. The image obtained as digital data by the ADC 34 in this way is output as a RAW image from the spectroscopic sensor 3.

[0016] In this example, the spectral sensor 3 is designed to support both still image capture mode and video capture mode. In video capture mode, it captures RAW images at a predetermined frame rate.

[0017] Here, with reference to Figure 2, the RAW image output from the spectral sensor 3 will be explained. Figure 2 is a schematic diagram showing an example of the configuration of the pixel array section 31 in the spectral sensor 3. As shown in the figure, the pixel array section 31 has spectral pixel units Pu formed therein, which consist of multiple pixels Pm that receive light in different wavelength bands, arranged in a predetermined pattern in two dimensions. The pixel array section 31 consists of spectral pixel units Pu arranged in two dimensions. In the example shown in the figure, each spectral pixel unit Pu receives light in a total of 16 wavelength bands from λ1 to λ16 individually at each pixel Pm, in other words, the number of wavelength bands that are received within each spectral pixel unit Pu (hereinafter referred to as "number of receiving channels") is 16. However, this is merely an example for explanatory purposes, and the number of receiving channels in the spectral pixel unit Pu can be set arbitrarily. Hereafter, the number of receiving channels in the spectral pixel unit Pu will be referred to as "N".

[0018] As described above, the pixel array section 31 is configured with a two-dimensional arrangement of spectral pixel units Pu, so the RAW image from the spectral sensor 3 is an image in which the light-receiving wavelength band differs depending on the pixel position.

[0019] In Figure 1, the demosaicing unit 4 and the wavelength analysis image generation unit 5 are provided to generate M wavelength band images from such a RAW image, where M > N. Specifically, the demosaicing unit 4 performs demosaicing on the RAW image from the spectroscopic sensor 3 to obtain N wavelength band images. Then, the wavelength analysis image generation unit 5 generates M wavelength band images from the N wavelength band images by performing linear matrix processing based on the N wavelength band images obtained from the demosaicing process.

[0020] Figure 3 is an explanatory diagram of the linear matrix processing for obtaining M wavelength band images. Based on the N channels of wavelength band images obtained by demosaicing by the demosaicing unit 4, a M channel of wavelength band images can be obtained by performing the matrix operation shown in the figure, according to [Equation 1] below, for each pixel position.

[0021] Note that, in the above description, it is assumed that for the spectroscopic sensor 3, an optical band-pass filter is provided for each pixel Pm to receive light in each wavelength band individually. However, the spectroscopic sensor 3 can also be configured as a sensor that acquires a spectroscopic image with a structure utilizing the principle of diffraction, a sensor that acquires a spectroscopic image by installing a thin film using a photonic crystal on the sensor, a sensor that obtains a spectroscopic image by utilizing the principle of surface plasmon resonance, or the like.

[0022] Here, as applications of the wavelength band image of Mch generated by the wavelength analysis image generation unit 5 through the above linear matrix processing, that is, the wavelength characteristic analysis image of the subject, for example, applications for vegetation analysis of plants such as vegetables and fruit trees can be cited. For example, as an evaluation index for vegetation analysis, the NDVI (Normalized Difference Vegetation Index) based on the received light value (Red) in the red wavelength band and the received light value (NIR) in the near-infrared wavelength band is known. Specifically, NDVI = (NIR - Red) / (NIR + Red).

[0023] Although illustration is omitted, the wavelength band image of Mch generated by the wavelength analysis image generation unit 5 is transmitted to an external computer device such as a personal computer, a smartphone, or a tablet terminal. In the computer device, evaluation values such as the above NDVI are calculated from the wavelength band image of Mch and used for the analysis of the subject.

[0024] By the way, the wavelength band image of Mch generated as described above is an unnatural image visually for humans and is also an image unsuitable for visually capturing the shape and color of the subject. Therefore, in order to obtain an image that is easy for humans to visually capture the subject from the wavelength band image of Mch, in the signal processing device 1 of the present embodiment, the RGB image generation unit 6 is configured to generate an RGB image (see FIG. 1).

[0025] Specifically, the RGB image generation unit 6 performs matrix operations shown in the following [Equation 2] for each pixel position on each wavelength band image for N channels obtained by the demosaicing process of the demosaicing unit 4, thereby generating images of each wavelength band of R, G, and B as an RGB image. <L

[0026] Here, the RGB image generation unit 6 is capable of performing gain adjustment on the RGB image generated by the matrix operation described above. This gain adjustment differs from the gain adjustment performed by the aforementioned gain adjustment unit 33, as it is a gain adjustment in the digital domain. In this sense, this gain adjustment can be called digital gain adjustment.

[0027] The display unit 12 is composed of a display device capable of displaying images, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, and displays various types of information. Specifically, the display unit 12 in this embodiment displays RGB images generated by the RGB image generation unit 6.

[0028] The image recognition processing unit 11 is configured with a programmable computing device such as a DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array), and performs image recognition processing using an AI (Artificial Intelligence) model. Image recognition processing here refers to the process of recognizing the content of an image. Examples of image recognition processing include object detection processing, which detects the region where an object exists; object recognition processing, which recognizes what kind of object is depicted in the image; and semantic segmentation processing. Object detection processing here includes not only the detection of the region where an object exists, but also the recognition of what kind of object it is, such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

[0029] In the signal processing device 1 of this embodiment, the image recognition processing unit 11 performs image recognition processing on the RGB image generated by the RGB image generation unit 6.

[0030] Furthermore, the information indicating the results of the image recognition processing by the image recognition processing unit 11 may be output to an external device of the signal processing device 1. Alternatively, the information indicating the results of the image recognition processing may be displayed on the display unit 12.

[0031] In the signal processing device 1, a first detection unit 7, a second detection unit 8, a first brightness control unit 9, and a second brightness control unit 10 are provided as components for controlling the brightness of the image.

[0032] The first detection unit 7 detects the RAW image output from the spectral sensor 3 and outputs the detected value to the first brightness control unit 9. The first brightness control unit 9 controls the exposure of the captured image based on the detected value of the RAW image input from the first detection unit 7. In this case, the detection by the first detection unit 7 can be rephrased as "photometry" for exposure control. Here, we assume that multi-segment photometry is performed as the photometry by the first detection unit 7, but it is also possible to perform photometry other than multi-segment photometry, such as spot photometry.

[0033] In this example, the first brightness control unit 9 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), and performs various processes for exposure control by software processing. In this example, the parameters for exposure control are shutter speed, aperture value (F-number), and ISO sensitivity. Since the signal processing device 1 in this example is configured as an electronic shutter type imaging device, the shutter speed can be controlled by instructing the pixel drive unit 32 to set the shutter speed. Furthermore, the ISO sensitivity can be controlled by instructing the gain adjustment unit 33 to set a gain value (analog gain value), and the aperture value can be controlled by instructing the drive circuit that drives the aperture actuator in the imaging optical system 2 to set the aperture value.

[0034] The second detection unit 8 performs detection on an image generated based on a RAW image, specifically the RGB image generated by the RGB image generation unit 6 in this example, and outputs the detected value to the second brightness control unit 10. The second brightness control unit 10 controls the gain (digital gain) of the RGB image generated by the RGB image generation unit 6 based on the detected value of the RGB image input from the second detection unit 8. Specifically, the second brightness control unit 10 calculates a target gain value to match the detected value of the RGB image input from the second detection unit 8 to a target brightness value for the RGB image, and controls the digital gain value of the RGB image generation unit 6 based on this target gain value. As a result, the digital gain value of the RGB image is controlled so that the brightness of the RGB image becomes the brightness of a predetermined target value.

[0035] In this example, the second detection unit 8 is configured to obtain a detection value for the RGB image that represents the overall brightness of the image, such as the average value of the luminance value of each pixel. Correspondingly, the second brightness control unit 10 in this example defines a target brightness value that represents the overall brightness of the image, such as the average value of the luminance value of each pixel.

[0036] In this example, the second brightness control unit 10, like the first brightness control unit 9, is configured with a microcomputer that includes a CPU, ROM, RAM, etc., and performs various processes for controlling the digital gain value of the RGB image as described above through software processing.

[0037] In a configuration like the signal processing device 1 described above, where an RGB image is generated based on a RAW image, and exposure control of the RAW image and digital gain adjustment of the RGB image are performed, there is a risk that brightness hunting may occur in the RGB image when the brightness of the subject changes rapidly.

[0038] Figure 4 is an explanatory diagram of the principle by which such hunting occurs. In Figure 4, the "vertical synchronization signal" is a signal that indicates the division of the frame period f of the captured image, and "sensor exposure" is an example of the exposure period for each frame period f of the spectroscopic sensor 3. Furthermore, "second image generation" indicates the generation timing of the RGB image, and "second image detection" and "first image detection" indicate the completion timing of the detection of the RGB image and the detection of the RAW image, respectively. In addition, "second image gain calculation" and "exposure calculation" indicate the completion timing of the gain calculation by the second brightness control unit 10 and the calculation of the exposure value (shutter speed, aperture, and ISO sensitivity in this example) by the first brightness control unit 9, respectively.

[0039] Generally, due to factors such as differences in the complexity of the calculation process, the image detection process and the calculation of exposure and gain values ​​based on the detection results tend to be implemented separately as hardware (HW) processing and software (SW) processing, respectively. When the detection process is implemented as hardware processing and the calculation of exposure and gain values ​​based on the detection results is implemented as software processing, a delay occurs between the time the detection process is performed and the time the exposure and gain values ​​are calculated.

[0040] Specifically, let's consider the gain value used at the RGB image generation timing "A" in the figure. This RGB image generation at timing "A" is based on the RAW image obtained by exposure during frame period f "D". At this RGB image generation timing "A", the gain value calculated in a frame period f prior to the frame period f to which timing "A" belongs can be used, and in the example shown, the gain value calculated at timing "B" will be used. This gain value calculated at timing "B" is based on the detection value obtained at detection timing "E". And this detection at timing "E" is detection of the RGB image generated from the RAW image obtained by exposure during frame period f "C".

[0041] Thus, the gain value applied to the RGB image obtained during frame period f of "D" at the generation timing of the RGB image of "A" is calculated based on the detection result of the RGB image generated during frame period f of "C", two frames prior to D, resulting in a delay between detection and gain application. Due to this delay, if the subject changes rapidly in brightness from frame period f of "C" to frame period f of "D", the gain value calculated based on the dark image of "C" will be adopted as the gain value of the RGB image generated based on the bright image of "D". Conversely, if the subject changes rapidly in darkness from "C" to "D", the gain value calculated based on the bright image of "C" will be adopted as the gain value of the RGB image generated based on the dark image of "D".

[0042] Due to this phenomenon, when the brightness of the subject changes rapidly, there is a risk that excessive brightness fluctuations (hunting) may occur in the RGB image.

[0043] To suppress this brightness hunting, the signal processing device 1 in this example uses the control value used for brightness control (exposure control) by the first brightness control unit 9 for brightness control by the second brightness control unit 10. Specifically, in the signal processing device 1 in this example, the second brightness control unit 10 controls the brightness of the RGB image based on the inter-frame difference value of the exposure value controlled by the exposure control of the first brightness control unit 9. Referring to the example in Figure 4, the gain value calculated at timing "B", which will be used for generating the RGB image at timing "A", is corrected based on the difference in exposure values ​​calculated at timings "H" and "I" shown in Figure 5.

[0044] Here, "J" is the frame period f during which the RAW image detected for calculating the exposure value of "H" was exposed, and "K" is the frame period f during which the RAW image detected for calculating the exposure value of "I" was exposed. In this embodiment, it is assumed that the brightness fluctuation from "J" to "K" continues from "C" to "D". Under this assumption, the hunting of the RGB image brightness is suppressed by applying a correction to the gain value calculated in "B" based on the brightness target value to cancel out the brightness fluctuation from "J" to "K".

[0045] Specifically, the gain value used for generating the RGB image at timing "A" Ｂ This is calculated using the following [Equation 3]. Here, "Err" refers to the difference between the detected value of the RGB image and the brightness target value (detected value - target value), which is calculated at timing "B". Ｈ "SenseExp Ｉ " and " represent the exposure value calculated at the "H" timing and the exposure value calculated at the "I" timing, respectively. Here, the change in exposure value will be treated as a change in the brightness of the subject. For this reason, "SenseExp Ｈ "SenseExp Ｉ The exposure value is normalized to the unit of the gain value of the RGB image, specifically dB (decibels), and used accordingly.

[0046] Here, in [Equation 3], "Gain Ｇ " is the gain value calculated at timing "G", in other words, the gain value used to generate the RGB image that was the target of detection when calculating the gain value of "B". In [Equation 3], this "Gain Ｇ The reason for using this is that the second detection unit 8 is configured to perform detection on the RGB image after gain adjustment.

[0047] By calculating the gain value in "B" according to the above [Equation 3], even when the brightness of the subject varies from "C" to "D", the gain value used in "A" can be corrected considering the variation. That is, it is possible to suppress the variation in the brightness of the RGB image with respect to a rapid change in the brightness of the subject, and it is possible to suppress the hunting of the brightness of the RGB image.

[0048] Here, FIGS. 4 and 5 show "N" under the following conditions １ 」「N ２ 」「M １ 」「M ２ 」 are all examples on the premise that they are set to "1". (1) The exposure value calculated by the first brightness control unit 9 is reflected in the exposure N １ frames after the frame at the calculation timing. (2) The RAW image in which the exposure value calculated by the first brightness control unit 9 is reflected is output to the frame next to the frame to which the exposure start timing reflecting the exposure value belongs. (3) The gain value calculated by the second brightness control unit 10 is reflected in the generation of the RGB image N ２ frames after the frame to which the calculation timing of the gain value belongs. (4) The exposure value calculated by the first brightness control unit 9 is output from the first brightness control unit 9 M １ frames after the frame in which the detection process of the detection value used for the calculation of the exposure value was performed. (5) The gain value calculated by the second brightness control unit 10 is output from the second brightness control unit 10 M ２ frames after the frame in which the detection process of the detection value used for the calculation of the gain value was performed.

[0049] Using the above "N １ 」「N ２ 」「M １ 」「M ２ 」, generalizing [Equation 3], each term in [Equation 3] can be expressed as follows. 「Gain Ｂ 」: The gain value calculated by the second brightness control unit 10 in a certain frame x. 「Gain Ｇ 」: Of frame x "M ２ + N ２"Gain value calculated before the frame "Err": The difference between the detection value input from the second detection unit 8 and the brightness target value in the gain value calculation process in frame x. Note that "SenseExp Ｈ ”, “SensExp. Ｉ Regarding ", Figure 5 shows "SenseExp Ｉ " is the exposure value calculated one frame before frame x, and "SenseExp Ｈ I gave an example where "" is the exposure value calculated one frame before frame x, but "SenseExp Ｉ " is any exposure value calculated in a frame prior to frame x, and "SenseExp Ｈ " is "SenseExp Ｉ The exposure value calculated in a frame prior to the frame in which the value was calculated is sufficient. In this case, the brightness delay period in the gain adjustment of the RGB image is "M ２ +N ２ Since this corresponds to the duration of "frames", "SenseExp Ｈ " is "SenseExp Ｉ The frame in which "" was calculated is "M ２ +N ２ It is preferable to use the exposure value calculated in the frame before the current frame.

[0050] An example of a specific processing procedure for realizing the brightness control method as a first example of the first embodiment described above will be explained with reference to the flowchart in Figure 6. The flowchart in Figure 6 shows an example of the processing procedure that the first brightness control unit 9 and the second brightness control unit 10 should each execute in order to realize the brightness control method as a first example of the first embodiment. In this example, both the first brightness control unit 9 and the second brightness control unit 10 are configured to have a CPU, and the processing shown in Figure 6 is performed by software processing using this CPU. Note that the processing shown in Figure 6 is not limited to being realized by software processing, but can also be realized by hardware processing. The first brightness control unit 9 and the second brightness control unit 10 execute the processing shown in Figure 6 for each frame.

[0051] First, in step S101, the first brightness control unit 9 calculates the inter-frame difference value of the exposure value. That is, in the aforementioned [Equation 3], "SenseExp Ｉ - SenseExp Ｈ The first brightness control unit 9 then calculates the inter-frame difference value in the following step S102, and transmits the calculated inter-frame difference value to the second brightness control unit 10, completing the series of processes shown in Figure 6.

[0052] In response to receiving the inter-frame difference value, the second brightness control unit 10 calculates the gain of the RGB image in step S201 based on the inter-frame difference value and the target brightness value of the RGB image. That is, it calculates the difference between the target brightness value of the RGB image and the detection value by the second detection unit 8 as the error Err, and then calculates the gain of the RGB image based on this error Err, the received inter-frame difference value, and "M ２ +N ２ "Gain value calculated before the frame" Ｇ Using the above (corresponding to ''), the gain value of the RGB image is calculated according to [Equation 3].

[0053] Then, in step S202 following step S201, the second brightness control unit 10 instructs the RGB image generation unit 6 to use the calculated gain, and completes the process shown in the example in Figure 6.

[0054] (1-2. Second Example) Next, a second example of the first embodiment will be described. In the second example, the sum of the analog gain value set in the exposure control and the digital gain value applied to the second image is controlled so as not to exceed a predetermined gain upper limit. With this control, it becomes possible to adjust the brightness control of the second image so that the degree of superposition of the RGB images does not exceed the degree of superposition determined by the above-mentioned gain upper limit, thereby realizing limit control for the image quality of the second image.

[0055] Referring to the block diagram in Figure 7, an example configuration of the signal processing device 1A as a second example in the first embodiment will be described. In the following description, parts that are the same as those already described will be denoted by the same reference numerals and their description will be omitted.

[0056] As can be seen by comparing it with Figure 1 above, the signal processing device 1A differs from the signal processing device 1 in that it is equipped with a first brightness control unit 9A instead of the first brightness control unit 9, and a second brightness control unit 10A instead of the second brightness control unit 10.

[0057] Figure 8 is a flowchart showing an example of the processing procedures performed by the first brightness control unit 9A and the second brightness control unit 10A, respectively. Here, the processing shown in Figure 8 is illustrated as being implemented by software processing by the first brightness control unit 9A and the second brightness control unit 10A, each configured with a CPU; however, the processing shown in Figure 8 may also be implemented by hardware processing. The first brightness control unit 9A and the second brightness control unit 10A execute the processing shown in Figure 8 for each frame.

[0058] In Figure 8, the first brightness control unit 9A performs the process of transmitting the analog gain value obtained from the exposure calculation to the second brightness control unit 10A in step S301. That is, it transmits the gain value to be instructed to the gain adjustment unit 33, calculated in the exposure calculation, as an analog gain value to the second brightness control unit 10A. The first brightness control unit 9A completes the series of processes shown in Figure 8 in accordance with the execution of the process in step S301.

[0059] The second brightness control unit 10A stores the analog gain value received from the first brightness control unit 9A in step S401. This analog gain value can be stored, for example, in RAM within the second brightness control unit 10A.

[0060] In step S402, following step S401, the second brightness control unit 10A calculates a digital gain based on the detection result of the RGB image and the brightness target value. That is, it calculates a digital gain value so that the error between the detection value by the second detection unit 8 and the brightness target value of the RGB image is canceled out. Specifically in this example, the digital gain value is the aforementioned "Gain Ｇ Calculate "-Err".

[0061] In step S403, following step S402, the second brightness control unit 10A determines whether the sum of the analog gain value and the digital gain value exceeds the gain upper limit. That is, it determines whether the sum of the digital gain value calculated in step S402 and the analog gain value stored in step S402 exceeds a preset gain upper limit.

[0062] In step S403, if the second brightness control unit 10A determines that the sum of the analog gain value and the digital gain value does not exceed the upper limit of the gain, the second brightness control unit 10A proceeds to step S202 and instructs the RGB image generation unit 6 to use the calculated gain. That is, it instructs the RGB image generation unit 6 to use the digital gain value calculated in step S402.

[0063] On the other hand, if in step S403 it is determined that the sum of the analog gain value and the analog gain value exceeds the upper limit of the gain, the second brightness control unit 10A proceeds to step S404 to calculate the digital gain value as "upper limit of gain - analog gain value", and then proceeds to step S202 to instruct the RGB image generation unit 6 to use the calculated digital gain value.

[0064] The second brightness control unit 10A completes the process shown in Figure 8, as an example, after executing the process in step S202.

[0065] (1-3. Various Examples) Here, the above shows an example configuration in which the second detection unit 8 performs detection on the RGB image after gain adjustment. However, as shown in the signal processing device 1B in Figure 9, it is also possible to adopt a configuration in which the second detection unit 8 performs detection on the RGB image before gain adjustment. Specifically, in the signal processing device 1B shown in Figure 9, compared to the signal processing device 1, an RGB image generation unit 6B is provided in place of the RGB image generation unit 6, a gain adjustment unit 13 is added, and a second brightness control unit 10B is provided in place of the second brightness control unit 10. The RGB image generation unit 6B differs from the RGB image generation unit 6 in that it does not have a function to adjust the digital gain of the RGB image it generates, and the gain adjustment unit 13 adjusts the digital gain of the RGB image generated by the RGB image generation unit 6B. In this case, the second detection unit 8 performs detection on the RGB image output by the RGB image generation unit 6B.

[0066] The second brightness control unit 10B calculates the digital gain value to be instructed to the gain adjustment unit 13 in the following manner, corresponding to the fact that the gain adjustment unit 13 is separately provided after the RGB image generation unit 6B. That is, when performing the brightness control described in the first example, the "Gain" in [Equation 3] Ｇ The calculation is performed excluding the " " term to calculate the digital gain value to be instructed to the gain adjustment unit 13.

[0067] In the case of brightness control as described in the second example, the signal processing device 1B is provided with a first brightness control unit 9A instead of the first brightness control unit 9, and the second brightness control unit 10B calculates the error Err as the digital gain value calculated in step S402.

[0068] Furthermore, while the above example illustrates the case where brightness control as the first embodiment (brightness control that reflects the first control value in the brightness control of the second image) is performed on the RGB image among the RGB image and Mch wavelength band image generated based on the RAW image as the first image, it is also possible to perform brightness control as the first embodiment on the Mch wavelength band image.

[0069] Figure 10 shows an example configuration of a signal processing device 1C that performs brightness control as a first embodiment, targeting an Mch wavelength band image. Specifically, Figure 10 shows an example configuration corresponding to the case where brightness control is performed as a first example.

[0070] In Figure 10, the signal processing device 1C differs from the signal processing device 1 in that a second detection unit 8C is provided instead of the second detection unit 8, and a second brightness control unit 10C is provided instead of the second brightness control unit 10.

[0071] The second detection unit 8C detects the Mch wavelength band image generated by the wavelength analysis image generation unit 5. The second brightness control unit 10C calculates a digital gain value to instruct the wavelength analysis image generation unit 5 based on the detection value by the second detection unit 8C and the exposure value calculated by the first brightness control unit 9. Specifically, when performing the brightness control in the first example, the digital gain value to instruct the wavelength analysis image generation unit 5 is calculated by performing a calculation similar to [Equation 3] based on the inter-frame difference value of the exposure value calculated by the first brightness control unit 9.

[0072] In the case of brightness control as described in the second example, the signal processing device 1C is provided with a first brightness control unit 9A instead of the first brightness control unit 9. In this case, the second brightness control unit 10C calculates a digital gain value to cancel the error between the detection value by the second detection unit 8C and the brightness target value, and performs brightness control so that the sum of the digital gain value and the analog gain value received from the first brightness control unit 9A does not exceed the gain upper limit.

[0073] Here, Figure 10 shows an example configuration for performing image recognition processing and image display on an Mch wavelength band image. Specifically, the Mch wavelength band image generated by the wavelength analysis image generation unit 5 is input to the image recognition processing unit 11C, which performs image recognition processing using the Mch wavelength band image as the input image, and is also displayed on the display unit 12. As an example of image recognition processing using an Mch wavelength band image as the input image, one could perform semantic segmentation processing to identify classes such as leaves, soil, and roads for each pixel.

[0074] Here, brightness control for the Mch wavelength band image can be performed individually for each single wavelength band image. Furthermore, the brightness control in the first embodiment can be performed not only on the RGB image or the wavelength band image, but on both images.

[0075] Furthermore, although the above example illustrates a case where the first brightness control unit (9, 9A) and the second brightness control unit (10, 10A, 10B, 10C) are configured as separate components, it is also conceivable that the processing of these first and second brightness control units could be implemented using common hardware. In particular, when the processing of each control unit is implemented using software, it is conceivable to adopt a configuration in which a common CPU executes the software processing of each control unit. This point applies not only to the first embodiment but also to the second and third embodiments described below.

[0076] <2. Second Embodiment> In the second embodiment, instead of reflecting the control value (first control value) used for brightness control of the first image on the brightness control side of the second image, as in the first embodiment, the second control value used for brightness control of the second image is reflected on the brightness control side of the first image. Specifically, the following describes an example in which the difference between the target brightness value and the detection value of the RGB image is reflected on the exposure control side.

[0077] Figure 11 is a block diagram showing an example configuration of the signal processing device 1D as a second embodiment. The signal processing device 1D differs from the signal processing device 1 in that the first brightness control unit 9D is provided instead of the first brightness control unit 9, and the second brightness control unit 10D is provided instead of the second brightness control unit 10.

[0078] The first brightness control unit 9D performs exposure control based on a brightness difference value, which indicates the difference between the target brightness value and the detected brightness for the RGB image. In other words, it performs exposure control based on the aforementioned error Err. Specifically, the first brightness control unit 9D performs exposure control so that the brightness difference indicated by the error Err is canceled out.

[0079] The second brightness control unit 10D transmits the calculated error value Err to the first brightness control unit 9D in order to realize the exposure control performed by the first brightness control unit 9D as described above. The second brightness control unit 10D also instructs the RGB image generation unit 6 with the error value Err to perform brightness control for the RGB image (control to match the brightness of the RGB image to a target value).

[0080] Generally, when the contrast of the subject is low, exposure control stabilizes at an exposure that converges the brightness distribution of the RAW image to the high-brightness side, resulting in an average bright RAW image. This tendency is particularly strong when exposure control is based on multi-segment metering, as in this example. If the RGB image is excessively bright, the brightness control of the RGB image is performed by reducing the gain, but this control can cause the tonal range of the RGB image to be lost. On the other hand, generally, when the contrast of the subject is high, exposure control stabilizes at an exposure that converges the brightness distribution of the RAW image to the low-brightness side, resulting in an average dark RAW image. This tendency is particularly strong when exposure control is based on multi-segment metering, as in this example. If the RGB image is excessively dark, the brightness control of the RGB image is performed by increasing the gain, but this control can cause the RGB image to become noisy, worsening the SNR (Signal to Noise Ratio).

[0081] Therefore, in this embodiment, exposure control is performed so that the difference between the target brightness value and the detected brightness of the RGB image is canceled out, as described above. This makes it possible to adjust the exposure control so that image quality degradation due to tonal loss or noise occurs in the RGB image due to the contrast of the subject, thereby preventing such tonal loss and noise. Consequently, the image quality of the RGB image can be improved.

[0082] Figure 12 is a flowchart showing a specific example of processing steps for realizing the brightness control method as the second embodiment described above. The flowchart in Figure 12 shows an example of processing steps that the first brightness control unit 9D and the second brightness control unit 10D should each execute in order to realize the brightness control method as the second embodiment. In this example, both the first brightness control unit 9D and the second brightness control unit 10D are configured to have a CPU, and the processing shown in Figure 12 is performed by software processing using this CPU. Note that the processing shown in Figure 12 is not limited to being realized by software processing, but can also be realized by hardware processing. The first brightness control unit 9D and the second brightness control unit 10D execute the processing shown in Figure 12 for each frame.

[0083] In Figure 12, the second brightness control unit 10D calculates a brightness difference value in step S601, which represents the difference between the target brightness value and the detected brightness for the RGB image. That is, it calculates the aforementioned error Err as the brightness difference value. Then, in the following step S602, the second brightness control unit 10D transmits the brightness difference value to the first brightness control unit 9D, completing the series of processes shown in Figure 12.

[0084] In step S501, the first brightness control unit 9D controls the exposure so that the brightness difference indicated by the brightness difference value is canceled out. Specifically, the first brightness control unit 9D calculates a target exposure value based on the detection value of the first detection unit 7 using an exposure value calculation method corresponding to a predetermined photometric method such as a multi-segment photometric method, and corrects the target exposure value so that the brightness difference indicated by the brightness difference value received from the second brightness control unit 10D is canceled out. Then, exposure control is performed using the corrected exposure value.

[0085] The first brightness control unit 9D completes the series of processes shown in Figure 12 in accordance with the execution of the process in step S501.

[0086] In the second embodiment, an example was shown in which the RGB image (wavelength band image generated by the wavelength analysis image generation unit 5) is used as the brightness control target among the RGB image and wavelength band image generated based on the RAW image as the first image. However, the wavelength band image can also be used as the brightness control target. In this case, the second brightness control unit 10D transmits the brightness difference value calculated for the wavelength band image to the first brightness control unit 9D.

[0087] Although not shown in the illustration, in the second embodiment as well, the wavelength band image can be used as the target for image recognition processing or display by the display unit 12.

[0088] <3. Third Embodiment> The third embodiment uses a polarization sensor as a light-receiving sensor for obtaining an image. The polarization sensor referred to here means a light-receiving sensor for obtaining a polarized image, which is an image that shows polarization information for each pixel.

[0089] Figure 13 is a block diagram showing an example configuration of the signal processing device 1E as a third embodiment. The signal processing device 1E differs from the signal processing device 1 in that a polarization sensor 15 is provided instead of the spectral sensor 3, a polarization image generation unit 16 and a gain adjustment unit 17 are provided instead of the demosaicing unit 4, wavelength analysis image generation unit 5 and RGB image generation unit 6, and a second detection unit 8E and a second brightness control unit 10E are provided instead of the second detection unit 8 and second brightness control unit 10.

[0090] The polarization sensor 15 differs from the spectral sensor 3 in that it has a pixel array section 35 instead of a pixel array section 31.

[0091] Now, with reference to Figures 14 to 19, we will explain how to generate a polarized image using the polarization sensor 15.

[0092] Figure 14 is a schematic diagram showing an example of the configuration of the pixel array section 35 of the polarization sensor 15. The pixel array section 35 is formed by arranging pixels Ps, which have light-receiving elements, specifically photodiodes PD in this example, in a two-dimensional arrangement.

[0093] As shown in the figure, the pixel array section 35 has a polarizing pixel unit PP and a color polarizing pixel unit PC. The polarizing pixel unit PP is a pixel unit in which multiple types of pixels Ps, each selectively receiving light with a different polarization angle, are arranged in a predetermined two-dimensional pattern. Specifically, the polarizing pixel unit PP in this example consists of a total of four pixels arranged in a predetermined two-dimensional pattern: pixels Ps that receive only light with a polarization angle of 90 degrees, pixels Ps that receive only light with a polarization angle of 45 degrees, pixels Ps that receive only light with a polarization angle of 135 degrees, and pixels Ps that receive only light with a polarization angle of 0 degrees (180 degrees).

[0094] A color polarized pixel unit (PC) is a pixel unit composed of multiple types of polarized pixel units (PP), each selectively receiving light of a different color, arranged in a predetermined two-dimensional pattern. Specifically, in this example, the color polarized pixel unit (PC) consists of four polarized pixel units (PP): one that receives only red light (R), two that receive only green light (G), and one that receives only blue light (B), all arranged in a predetermined two-dimensional pattern. In the figure, the polarized pixel unit (PP) that selectively receives red light (R) is marked with a diagonal line sloping downwards to the left, the polarized pixel unit (PP) that selectively receives green light (G) is marked with a vertical line, and the polarized pixel unit (PP) that selectively receives blue light (B) is marked with a diagonal line sloping downwards to the right. In this example, the four polarized pixel units (PP) that receive red, blue, and green light (R, B, and G) are arranged in a Bayer configuration.

[0095] The pixel array section 35 is made up of the above-described color polarized pixel units PC arranged in two dimensions. That is, multiple color polarized pixel units PC are arranged in the vertical direction (column direction) and the horizontal direction (row direction).

[0096] With the pixel array section 35 configured as described above, it becomes possible to generate R, G, and B color images as various polarization images. In other words, it becomes possible to generate color images as various polarization images.

[0097] In the pixel array section 35, each pixel Ps is configured to selectively receive light with a predetermined polarization angle and to selectively receive light of a predetermined color (wavelength band).

[0098] Figure 15 illustrates a schematic cross-sectional structure of a pixel Ps. As shown in the figure, the pixel Ps has a photodiode PD formed as a light-receiving element within a semiconductor substrate 50, a wiring layer 51 formed on one side of the semiconductor substrate 50, and a polarizing filter 52, a color filter 53, and a microlens 54 stacked on the other side of the semiconductor substrate 50. The polarizing filter 52 is configured to have a polarizer that selectively transmits linearly polarized light vibrating in a specific direction (angle). Examples of polarizers include those using a wire grid or those having a crystalline structure such as a photonic crystal.

[0099] The color filter 53 is configured as an optical bandpass filter that selectively transmits light in a predetermined wavelength band. For example, in a pixel Ps that receives red light, an optical bandpass filter that selectively transmits red light is formed as the color filter 53. Similarly, in a pixel Ps that receives green light and a pixel Px that receives blue light, an optical bandpass filter that selectively transmits green light and an optical bandpass filter that selectively transmits blue light are formed as the color filter 53, respectively. Note that the relative positions of the polarizing filter 52 and the color filter 53 may be reversed.

[0100] Figure 16 is a block diagram illustrating an example of the internal configuration of the polarization image generation unit 16. As shown in the figure, the polarization image generation unit 16 includes an image organization unit 16a, a demosaicing unit 16b, a polarization state estimation unit 16c, and a polarization image generation processing unit 16d.

[0101] Image processing by the image processing unit 16a and demosaicing processing by the demosaicing unit 16b result in the acquisition of R, G, and B color images for images obtained by selectively receiving light with a polarization angle of 90 degrees (hereinafter referred to as the "90-degree image"), images obtained by selectively receiving light with a polarization angle of 45 degrees (hereinafter referred to as the "45-degree image"), images obtained by selectively receiving light with a polarization angle of 0 degrees (hereinafter referred to as the "0-degree image"), and images obtained by selectively receiving light with a polarization angle of 135 degrees (hereinafter referred to as the "135-degree image").

[0102] Figure 17 is an explanatory diagram of the image arrangement process by the image arrangement unit 16a and the demosaicing process by the demosaicing unit 16b. In the image arrangement process, the received values ​​(luminance values) of pixels Ps that receive light at the same polarization angle are extracted from the RAW image output from the polarization sensor 15 (see Figure 17A) to arrange polarization angle separated images, which are images for each polarization angle (see Figure 17B). By performing this image arrangement process, in the polarization angle separated images for polarization angles of 90 degrees, 45 degrees, 0 degrees, and 135 degrees, a total of four pixels—one pixel receiving R light, two pixels receiving G light, and one pixel receiving B light—are arranged in a Bayer array for each color polarization pixel unit PC. In other words, a normal demosaicing process can be applied to the Bayer array.

[0103] In the demosaicing section 16b, demosaicing is performed on each polarization angle-separated image. Through this demosaicing process, as shown in Figure 17C, R, G, and B color images are obtained as 90-degree images, 45-degree images, 0-degree images, and 135-degree images, respectively.

[0104] Figure 18 is an explanatory diagram of the processing of the polarization state estimation unit 16c. In Figure 18A, representative 90-degree, 45-degree, 0-degree, and 135-degree images of each color obtained by the demosaicing unit 16b are shown as R images.

[0105] In the polarization state estimation unit 16c, based on the 90-degree, 45-degree, 0-degree, and 135-degree images of each color obtained in the demosaicing unit 16b, a process is performed for each pixel position to fit a sine wave as shown in Figure 18B, for each of the R, G, and B color images. The polarization state of incident light is represented by a sine wave with luminance on the vertical axis and polarization direction (polarization angle) on the horizontal axis. Therefore, for the 90-degree, 45-degree, 0-degree, and 135-degree images of the target color, the polarization state for each pixel position can be estimated by fitting a sine wave based on the received light value (luminance value) for each pixel position.

[0106] Figure 19 is an explanatory diagram of the polarization image generation method performed by the polarization image generation processing unit 16d. The polarization state estimation unit 16c obtains sinusoidal wave information indicating the polarization state for each pixel position, and based on this sinusoidal wave information, it becomes possible to generate various polarization images.

[0107] Possible polarization images include, for example, reflection-enhanced images, reflection-suppressed images, and polarization-degree images. A reflection-enhanced image is obtained by detecting a reflection-enhanced signal at each pixel position. As shown in the figure, the reflection-enhanced signal is detected as the maximum value (Imax) of a sine wave. In other words, a reflection-enhanced image is generated by detecting the maximum value of a sine wave at each pixel position.

[0108] Furthermore, a reflection-suppressed image is obtained by detecting a reflection-suppressed signal at each pixel position. As shown in the figure, the reflection-suppressed signal is detected as the minimum value (Imin) of a sine wave. Therefore, a reflection-suppressed image can be generated by detecting the minimum value of a sine wave at each pixel position.

[0109] A polarization degree image is obtained by detecting a polarization degree signal ρ at each pixel position. Here, the polarization degree signal ρ is calculated using the following [Equation 4]. A polarization degree image can be generated by calculating the polarization degree signal ρ using [Equation 4] with the maximum value (Imax) and minimum value (Imin) of the sine wave for each pixel position.

[0110] Furthermore, various signals indicating the polarization state can be generated from the sine wave information representing the polarization state of the incident light, and the polarization images generated by the polarization image generation processing unit 16d are not limited to the three types exemplified above: reflection-enhanced images, reflection-suppressed images, and polarization degree images. For example, it is also conceivable to generate a polarization component image, which is an image in which the difference between the maximum and minimum values ​​of the sine wave ("Imax - Imin") is detected for each pixel position, or an average image, which is an image in which the average value of the sine wave is calculated for each pixel position.

[0111] In this example, the polarization image generation processing unit 16d generates and outputs a polarization image as a polarization image, which is one of the reflection-enhanced image, reflection-suppressed image, or polarization degree image exemplified above.

[0112] In this example, the polarization image generation unit 16 generates both a polarization image and an RGB image. Specifically, in the polarization image generation unit 16, the demosaicing unit 16b (see Figure 16) generates an RGB image based on the 90-degree, 45-degree, 0-degree, and 135-degree images of each RGB color shown in Figure 17C obtained by demosaicing. Specifically, for example, an RGB image is generated by calculating the average value of the brightness of the 90-degree, 45-degree, 0-degree, and 135-degree images for each pixel position for each R, G, and B color.

[0113] Let's return to the explanation in Figure 13. In this case, the first detection unit 7 performs detection for exposure control using the RAW image as the output image of the polarization sensor 15.

[0114] The gain adjustment unit 17 adjusts the brightness of the polarized image by performing digital gain adjustment on the polarized image generated by the polarized image generation unit 16.

[0115] The polarized image after gain adjustment by the gain adjustment unit 17 is supplied to the display unit 12 and displayed. The polarized image after gain adjustment by the gain adjustment unit 17 is also provided as the input image to the image recognition processing unit 11E. The image recognition processing unit 11E performs image recognition processing using the polarized image as the input image. For example, by performing image recognition processing using the polarized image as a reflection suppression image as the input image, it becomes possible to detect and identify a subject located behind a window in which an object in the foreground is reflected (in other words, a window reflecting light from the object in the foreground).

[0116] The second detection unit 8E performs detection of the polarized image after gain adjustment by the gain adjustment unit 17, and the second brightness control unit 10E controls the digital gain value in the gain adjustment unit 17 based on the detection value obtained by the second detection unit 8E. Specifically, in this example, the second brightness control unit 10E controls the brightness of the polarized image using the same brightness control method as in the first example of the first embodiment described earlier (a method of controlling brightness based on the inter-frame difference value of exposure). However, in the signal processing device 1E, as in the case of Figure 9 above, the digital gain adjustment is performed in the later stage of image generation, so the second brightness control unit 10E, like the second brightness control unit 10B described earlier, uses "Gain" from [Equation 3]. Ｇ The digital gain value of the target image is calculated using the formula excluding the term "".

[0117] Figure 20 is a flowchart showing an example of the processing procedure performed by the first brightness control unit 9 and the second brightness control unit 10E in the signal processing device 1E. Here, the processing shown in Figure 20 is illustrated as being implemented by software processing by the first brightness control unit 9 and the second brightness control unit 10E, each configured with a CPU; however, the processing shown in Figure 20 may also be implemented by hardware processing. The first brightness control unit 9 and the second brightness control unit 10E execute the processing shown in Figure 20 for each frame.

[0118] In Figure 20, the processing by the first brightness control unit 9 (steps S101 and S102) is the same as that described in Figure 6, so a redundant explanation will be avoided.

[0119] In step S801, the second brightness control unit 10E calculates the gain of the polarized image based on the inter-frame difference value received from the first brightness control unit 9 and the target brightness value of the polarized image. That is, it calculates the difference between the target brightness value of the polarized image and the detection value by the second detection unit 8E as the error Err, and uses this error Err and the received inter-frame difference value to calculate the gain according to [Equation 3] (where "Gain" is used). Ｇ (Excluding the section marked with ") Calculate the gain value of the polarized image.

[0120] Then, in step S802 following step S801, the second brightness control unit 10E instructs the gain adjustment unit 17 to use the calculated gain, and completes the process shown in the example in Figure 20.

[0121] In the above example, we have given an example of applying the brightness control method as the first example in the first embodiment when the second image is a polarized image. However, it is also possible to apply the brightness control method as the second example in the first embodiment or the brightness control method as the second embodiment.

[0122] Furthermore, while the above example focused on controlling brightness for polarized images, it is also possible to adopt configurations that control brightness for RGB images, or configurations that control brightness for both polarized and RGB images.

[0123] <4. Modifications> The embodiments are not limited to the specific examples described above, and various modified configurations can be adopted. For example, although the above example illustrates a case where the first brightness control unit of this technology controls the brightness of a RAW image, the image that the first brightness control unit controls the brightness of is not limited to a RAW image. The first brightness control unit only needs to control the brightness of the first image when a second image different from the first image is generated based on the first image. If the first image is an image other than a RAW image, it is possible that the first image is an image generated based on a RAW image.

[0124] Furthermore, in this technology, the second image may be a different type of image from the first image, generated based on the first image. For example, an image that is simply a copy of the first image, or an image in which only the brightness of the first image has been changed, would not be considered a second image.

[0125] Furthermore, in the explanations so far, we have described examples in which only one of the following is performed: the control that reflects the first control value (the control value used for brightness control of the first brightness control unit) to the brightness control side of the second image, as described in the first embodiment, and the control that reflects the second control value (the control value used for brightness control of the second brightness control unit) to the brightness control side of the first image, as described in the second embodiment. However, it is also possible to adopt a configuration that performs both of these types of control.

[0126] Furthermore, while the above example shows that the signal processing device related to this technology is configured as an imaging device equipped with an imaging unit such as a spectroscopic sensor 3 or a polarization sensor 15, it is not essential that the signal processing device related to this technology includes an imaging unit.

[0127] <5. Summary of Embodiments> As described above, the signal processing apparatus as an embodiment (1, 1A to 1E) comprises: a first brightness control unit (9, 9A, 9D) that controls the brightness of the first image based on the detection result of the first image; a second image generation unit (RGB image generation unit 6, 6B, wavelength analysis image generation unit 5, polarization image generation unit 16) that generates a second image different from the first image based on the first image whose brightness has been controlled by the first brightness control unit; and a second brightness control unit (10, 10A, 10B, 10C, 10D, 10E) that controls the brightness of the second image based on the detection result of the second image. The first control value used for brightness control by the first brightness control unit is used for brightness control by the second brightness control unit, or the second control value used for brightness control by the second brightness control unit is used for brightness control by the first brightness control unit. According to the above configuration, in a signal processing system in which a second image is generated based on a first image, and the brightness control of the first and second images is performed based on the detection result of the first image and the detection result of the second image, the brightness control by the second brightness control unit is performed as a control that reflects the first control value used for brightness control by the first brightness control unit, or the brightness control by the first brightness control unit is performed as a control that reflects the second control value used for brightness control by the second brightness control unit. This makes it possible to adjust the brightness control of the second image or the brightness control of the first image so that the brightness control of the second image is performed appropriately. Therefore, in a signal processing system in which a second image is generated based on a first image, and the brightness control of the first and second images is performed based on the detection result of the first image and the detection result of the second image, it is possible to ensure that the brightness control of the second image is performed appropriately.

[0128] Furthermore, in the signal processing apparatus as an embodiment (1, 1A, 1B, 1C, 1E), the first control value used for brightness control by the first brightness control unit is used for brightness control by the second brightness control unit. With the above configuration, the brightness control of the second image is adjusted based on the first control value used for brightness control of the first image, so the first brightness control can be performed as usual without reflecting the second control value. Therefore, in order to ensure that the brightness control of the second image is performed appropriately, the first brightness control can be performed in a manner that achieves its control purpose. Accordingly, in a signal processing system in which a second image is generated based on a first image, and the brightness control of the first and second images is performed based on the detection result of the first image and the detection result of the second image, respectively, it is possible to ensure that both the brightness control of the first image and the brightness control of the second image are performed appropriately.

[0129] Furthermore, in the signal processing apparatus as an embodiment, the first image is a RAW image obtained by imaging, and the first brightness control unit performs exposure control of the captured image as brightness control of the first image. This ensures that the brightness control of the second image is performed appropriately in a signal processing system where the brightness control of the first image is exposure control and the brightness of the second image is affected by the brightness control of the first image.

[0130] Furthermore, in the signal processing device as an embodiment, the second brightness control unit (10, 10B, 10C, 10E) controls the brightness of the second image based on the inter-frame difference value of the exposure value obtained by exposure control. This makes it possible to suppress a decrease in the tracking ability of the brightness control of the second image in response to changes in the brightness of the subject. Therefore, it is possible to suppress hunting of the brightness of the second image.

[0131] Furthermore, in the signal processing device as an embodiment, the second brightness control unit (10A, 10B, 10C, 10E) receives the analog gain value in exposure control as the first control value, and controls the brightness of the second image so that the sum of the analog gain value and the digital gain value obtained from the detection result of the second image and the brightness target value does not exceed a predetermined upper limit. This makes it possible to adjust the brightness control of the second image so that the degree of noise superposition in the second image does not exceed a predetermined degree of superposition defined by the above upper limit. Thus, limit control of the image quality of the second image can be realized.

[0132] Furthermore, in the signal processing apparatus (1D, 1E) as an embodiment, the second control value used for brightness control by the second brightness control unit is used for brightness control by the first brightness control unit. This makes it possible to adjust the brightness control of the first image based on the second control value so that the brightness control of the second image is properly performed, in response to cases where the brightness control of the second image cannot be properly performed due to the brightness control of the first image. Therefore, in a signal processing system in which a second image is generated based on the first image, and the brightness control of the first and second images is performed based on the detection results of the first and second images, respectively, it is possible to ensure that the brightness control of the second image is properly performed.

[0133] Furthermore, in the signal processing apparatus as an embodiment, the first image is a RAW image obtained by imaging, and the first brightness control unit controls the exposure of the captured image as brightness control of the first image, and the first brightness control unit controls the brightness of the first image based on a brightness difference value that indicates the difference between the brightness target value of the second image and the detected brightness. As a result, when image quality degradation due to tonal loss or noise occurs in the second image due to the contrast of the subject, it is possible to adjust the brightness control of the first image so as to suppress such tonal loss or noise-induced image quality degradation. Therefore, the image quality of the second image can be improved.

[0134] Furthermore, in the signal processing device as an embodiment, the first brightness control unit controls the brightness of the first image so that the brightness difference indicated by the brightness difference value is canceled out. This makes it possible to control the brightness of the first image so that when image quality degradation due to tonal loss or noise occurs in the second image due to the difference in contrast of the subject, such tonal loss or noise degradation is prevented. Therefore, the image quality of the second image can be improved.

[0135] Furthermore, the signal processing device as an embodiment includes an image recognition processing unit (11, 11C, 11E) that performs image recognition processing using an AI model on the second image. This makes it possible to appropriately control the brightness of the second image, which is the target image for image recognition processing. Consequently, the accuracy of image recognition can be improved.

[0136] The signal processing method as an embodiment is a signal processing method in which a signal processing device performs a first brightness control process that controls the brightness of the first image based on the detection result of the first image, a second image generation process that generates a second image different from the first image based on the first image whose brightness has been controlled by the first brightness control process, and a second brightness control process that controls the brightness of the second image based on the detection result of the second image, wherein the first control value used for brightness control in the first brightness control process is used for brightness control by the second brightness control process, or the second control value used for brightness control in the second brightness control process is used for brightness control by the first brightness control process. The same operation and effects as the signal processing device as an embodiment described above can be obtained with such a signal processing method.

[0137] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0138] <6. This Technology> This technology can also be configured as follows: (1) A signal processing device comprising: a first brightness control unit that controls the brightness of the first image based on the detection result of the first image; a second image generation unit that generates a second image different from the first image based on the first image whose brightness has been controlled by the first brightness control unit; and a second brightness control unit that controls the brightness of the second image based on the detection result of the second image, wherein a first control value used for brightness control by the first brightness control unit is used for brightness control by the second brightness control unit, or a second control value used for brightness control by the second brightness control unit is used for brightness control by the first brightness control unit. (2) The signal processing device according to (1) above, wherein a first control value used for brightness control by the first brightness control unit is used for brightness control by the second brightness control unit. (3) The signal processing device according to (1) or (2) above, wherein the first image is a RAW image obtained by imaging, and the first brightness control unit controls the exposure of the captured image as brightness control of the first image. (4) The signal processing device according to (3) wherein the second brightness control unit controls the brightness of the second image based on the interframe difference value of the exposure value obtained by the exposure control. (5) The signal processing device according to (3) wherein the second brightness control unit receives the analog gain value in the exposure control as the first control value and controls the brightness of the second image such that the sum of the analog gain value and the digital gain value obtained from the detection result and the brightness target value of the second image does not exceed a predetermined upper limit. (6) The signal processing device according to any one of (1) to (5) wherein the second control value used for brightness control by the second brightness control unit is used for brightness control by the first brightness control unit. (7) The signal processing device according to any one of (1) to (6) wherein the first image is a RAW image obtained by imaging, the first brightness control unit controls the exposure of the captured image as brightness control of the first image, and the first brightness control unit controls the brightness of the first image based on a brightness difference value indicating the difference between the brightness target value and the detected brightness for the second image.(8) The signal processing device according to (7), wherein the first brightness control unit controls the brightness of the first image such that the brightness difference indicated by the brightness difference value is canceled out. (9) The signal processing device according to any one of (1) to (8), further comprising an image recognition processing unit that performs image recognition processing using an AI model on the second image. (10) A signal processing method comprising: a first brightness control process that controls the brightness of the first image based on the detection result of the first image; a second image generation process that generates a second image different from the first image based on the first image whose brightness has been controlled by the first brightness control process; and a second brightness control process that controls the brightness of the second image based on the detection result of the second image, wherein the first control value used for brightness control in the first brightness control process is used for brightness control by the second brightness control process, or the second control value used for brightness control in the second brightness control process is used for brightness control by the first brightness control process.

[0139] 2 Imaging optical system 3 Spectroscopic sensor 31 Pixel array section 32 Pixel driving section 33 Gain adjustment section 34 ADC Pm Pixel Pu Spectroscopic pixel unit 4 Demosaicing section 5 Wavelength analysis image generation section 6, 6B RGB image generation section 7 First detection section 8, 8C Second detection section 9, 9A, 9D First brightness control section 10, 10A, 10B, 10C, 10D, 10E Second brightness control section 11, 11C, 11E Image recognition processing section 12 Display section 13 Gain adjustment section 15 Polarization sensor 35 Pixel array section Ps Pixel PP Polarization pixel unit PC Color polarization pixel unit 50 Semiconductor layer 51 Wiring layer 52 Polarization filter 53 Color filter 54 Microlens 16 Polarization image generation section 16a Image composition section 16b Demosaicing section 16c Polarization state estimation unit 16d Polarization image generation processing unit 17 Gain adjustment unit

Claims

1. A signal processing device comprising: a first brightness control unit that controls the brightness of a first image based on the detection result of the first image; a second image generation unit that generates a second image different from the first image based on the first image whose brightness has been controlled by the first brightness control unit; and a second brightness control unit that controls the brightness of the second image based on the detection result of the second image, wherein a first control value used for brightness control by the first brightness control unit is used for brightness control by the second brightness control unit, or a second control value used for brightness control by the second brightness control unit is used for brightness control by the first brightness control unit.

2. The signal processing device according to claim 1, wherein the first control value used for brightness control by the first brightness control unit is used for brightness control by the second brightness control unit.

3. The signal processing apparatus according to claim 1, wherein the first image is a RAW image obtained by imaging, and the first brightness control unit performs exposure control of the captured image as brightness control of the first image.

4. The signal processing device according to claim 3, wherein the second brightness control unit controls the brightness of the second image based on the interframe difference value of the exposure value obtained by the exposure control.

5. The signal processing device according to claim 3, wherein the second brightness control unit receives the analog gain value in the exposure control as the first control value, and controls the brightness of the second image such that the sum of the analog gain value and the digital gain value obtained from the detection result of the second image and the brightness target value does not exceed a predetermined upper limit.

6. The signal processing device according to claim 1, wherein the second control value used for brightness control by the second brightness control unit is used for brightness control by the first brightness control unit.

7. The signal processing apparatus according to claim 1, wherein the first image is a RAW image obtained by imaging, the first brightness control unit performs exposure control of the captured image as brightness control of the first image, and the first brightness control unit performs brightness control of the first image based on a brightness difference value indicating the difference between the brightness target value and the detected brightness of the second image.

8. The signal processing apparatus according to claim 7, wherein the first brightness control unit controls the brightness of the first image such that the brightness difference indicated by the brightness difference value is canceled out.

9. The signal processing apparatus according to claim 1, further comprising an image recognition processing unit that performs image recognition processing using an AI model on the second image.

10. A signal processing method comprising: a signal processing device performing a first brightness control process that controls the brightness of a first image based on the detection result of a first image; a second image generation process that generates a second image different from the first image based on the first image whose brightness has been controlled by the first brightness control process; and a second brightness control process that controls the brightness of the second image based on the detection result of the second image, wherein the first control value used for brightness control in the first brightness control process is used for brightness control by the second brightness control process, or the second control value used for brightness control in the second brightness control process is used for brightness control by the first brightness control process.

Images