Signal processing device, photoelectric conversion device, photoelectric conversion system, and control method and program for signal processing device

Abstract

To provide a technique advantageous for improving the accuracy of correction for a signal output from an imaging device in a signal processing device.SOLUTION: A signal processing device processing image data output from a photoelectric conversion unit having a light receiving area and a light blocking area, is provide including: a control data generation unit that outputs control data used to generate correction data for correction of the image data by using a learned model generated by machine learning; and a signal processing unit that generates the correction data on the basis of the light blocking image data which is the image data of the light blocking data and the control data, and corrects the light receiving image data which is the image data of the light receiving area of the image data, without applying the learned model according to the correction data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a signal processing device, a photoelectric conversion device, a photoelectric conversion system, a control method for a signal processing device, and a program. [Background technology]

[0002] In an imaging device using an imaging element such as a CMOS image sensor, it is necessary to correct noise caused by dark current generated in each pixel in order to improve image quality. Patent Document 1 shows that an imaging area is divided into multiple blocks, the size and division position of the blocks are changed depending on the imaging conditions, and correction values ​​according to a division pattern stored in advance are used to perform appropriate correction for various imaging conditions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-034315 Summary of the Invention [Problem to be solved by the invention]

[0004] To further improve image quality, it is necessary to more appropriately correct the dark current generated in each pixel of the image sensor and the noise caused by the circuitry for reading out signals from the pixels.

[0005] An object of the present invention is to provide a technique that is advantageous for improving the accuracy of correction of a signal output from an imaging element in a signal processing device. [Means for solving the problem]

[0006] In view of the above-described problems, a signal processing device according to an embodiment of the present invention is a signal processing device that processes image data output from a photoelectric conversion unit having a light receiving region and a light blocking region, and includes a trained model generated by machine learning. By inputting the image data output from the photoelectric conversion unit or image data acquired by previous imaging,a control data generating unit that outputs control data used to generate correction data for correcting the image data; and light-shielded image data that is image data of the light-shielded region among the image data. By averaging each row and column, Generate the correction data At the same time , the received light image data which is the image data of the light receiving region among the image data; By subtracting the correction data from , applying the trained model The received light image data is then a signal processing unit for correcting the The control data is at least one of information regarding the size of an area in the light-blocking image data where the averaging process is performed, information regarding the location of an area in the light-blocking image data where the averaging process is performed, and information regarding the gain of a low-pass filter to be applied to an average value obtained by the averaging process. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique that is advantageous for improving the accuracy of correction of a signal output from an imaging element in a signal processing device. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a photoelectric conversion device including a signal processing device according to an embodiment of the present invention. [Figure 2] 2 is a conceptual diagram of an output signal from a photoelectric conversion unit of the photoelectric conversion device of FIG. 1. [Figure 3] FIG. 3 is a conceptual diagram of OB clamp processing in the signal processing device of the present embodiment. [Figure 4] FIG. 4 is a flow diagram of generating control data used for correction in the signal processing device of the present embodiment. [Figure 5] FIG. 2 is a diagram showing a machine learning model of the signal processing device of the present embodiment. [Figure 6] FIG. 1 is a diagram showing an example of the configuration of a photoelectric conversion device including a signal processing device according to an embodiment of the present invention. [Figure 7] FIG. 1 is a diagram showing an example of the configuration of a photoelectric conversion device including a signal processing device according to an embodiment of the present invention. [Figure 8] FIG. 1 is a diagram showing an example of the configuration of a photoelectric conversion device including a signal processing device according to an embodiment of the present invention. [Figure 9] FIG. 4 is a flow diagram of generating control data used for correction in the signal processing device of the present embodiment. [Figure 10] FIG. 2 is a diagram showing an example of the arrangement of the photoelectric conversion device shown in FIG. 1. [Figure 11]FIG. 2 is a diagram showing an example of the configuration of an imaging device incorporating the photoelectric conversion device of FIG. 1. [Figure 12] FIG. 2 is a diagram showing an example of the configuration of a photoelectric conversion system including the photoelectric conversion device of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] First embodiment A signal processing device according to a first embodiment of the present disclosure will be described with reference to Figures 1 to 5. Figure 1 is a diagram showing an example configuration of a photoelectric conversion device 100 including a signal processing device 150 of this embodiment. The photoelectric conversion device 100 includes the signal processing device 150 and a photoelectric conversion unit 101 that includes a light-receiving region and a light-shielding region, which will be described later, and that outputs image data processed by the signal processing device 150. The photoelectric conversion device 100 may further include a vertical scanning unit 103, a control unit 104, a readout circuit unit 105, an AD conversion unit 106, a memory unit 107, and a horizontal scanning unit 108.

[0011] The signal processing device 150 includes a machine learning unit 151 and a signal processing unit 152. The machine learning unit 151 includes a trained model 160 generated by machine learning, and a control data generation unit 161 that outputs control data used when generating correction data for correcting the image data in the signal processing unit 152 using the trained model 160 based on the image data. The signal processing unit 152 generates correction data based on the image data and the control data generated by the control data generation unit 161, and corrects the image data output from the photoelectric conversion unit 101 in accordance with the correction data. In this embodiment, an example is shown in which the signal processing device 150 is mounted in the photoelectric conversion device 100, but this is not limiting, and the signal processing device 150 may be separate from the photoelectric conversion device 100.

[0012] The photoelectric conversion unit 101 is configured with (m+1) × (n+1) pixels, with m+1 pixels 102 arranged in the row direction (horizontal direction in FIG. 1 ) and n+1 pixels arranged in the column direction (vertical direction in FIG. 1 ). The vertical scanning unit 103 is connected to the m+1 pixels 102 arranged in each row via row selection lines 110 (V(n)) and selects a row from which to read out signals. In the selected row, signals output from the m+1 pixels 102 included in the selected row are simultaneously read out to the readout circuit unit 105 via vertical output lines 111 (H(m)). The readout circuit unit 105 may include an amplifier or the like to amplify the signals output from the pixels 102. The signals output from the readout circuit unit 105 are converted from analog to digital (AD conversion) by the AD conversion unit 106 and temporarily stored in the memory unit 107. Thereafter, the digital signals addressed by the horizontal scanning unit 108 are sequentially read out to the signal processing unit 152 of the signal processing device 150, where they are subjected to digital signal processing. The control unit 104 may include, for example, a timing generator, and may generate control signals for each component of the photoelectric conversion device 100. The control unit 104 may also acquire setting information such as imaging conditions for imaging by the photoelectric conversion device 100 through communication with an external device, and may supply control signals according to the conditions to each component included in the photoelectric conversion device 100. The control unit 104 controls the vertical scanning unit 103, the readout circuit unit 105, the AD conversion unit 106, the memory unit 107, the horizontal scanning unit 108, and the signal processing unit 152 based on this setting information.

[0013] The signal processing unit 152 performs processing to reduce reset noise generated in switching elements (e.g., MOS transistors) included in each pixel 102 of the photoelectric conversion unit 101. Furthermore, image data output from the photoelectric conversion unit 101 may contain variations (FPN: fixed pattern noise) generated due to dark currents generated by photodiodes included in the pixels 102 and differences in circuit-related factors such as power supply impedance and signal delay. In this specification, the state in which this FPN varies according to a certain rule from row to row or column to column is referred to as shading. After reducing the reset noise component, the signal processing unit 152 performs processing to reduce the FPN component, shading component, and the like, and extract signal components with suppressed noise components from the image data. In this specification, the processing performed by the signal processing unit 152 to reduce the noise component of image data is referred to as OB clamping processing.

[0014] FIG. 2 is a conceptual diagram illustrating a case where a signal output from the photoelectric conversion unit 101 is broken into lines at the end of each row in this embodiment. The photoelectric conversion unit 101 includes a light-receiving region 202 that receives incident light that has passed through an optical system such as a lens, and a light-shielding region 201 (also referred to as an optical black (OB) region) that optically shields the incident light. The light-shielding region 201 is a reference region for determining a so-called black level reference in image data obtained by the photoelectric conversion unit 101. Here, within the light-shielding region 201, as shown in FIG. 2, a region at the top of the screen where all columns of pixels 102 are optically shielded is referred to as a VOB region 203. Furthermore, a region located to the left of the light-receiving region 202 where all rows of pixels 102 are optically shielded is referred to as a HOB region 204.

[0015] In order to reduce the FPN components for each row and column in the OB clamping process, the signal processing unit 152 generates correction data from the light-shielded image data, which is the image data of the light-shielded region 201 among the image data. Next, the signal processing unit 152 corrects the light-receiving image data, which is the image data of the light-receiving region 202 among the image data, in accordance with the correction data. For example, the signal processing unit 152 generates the correction data by averaging the signal levels (signal values) of the light-shielded image data for each row and column. Next, the signal processing unit 152 may generate data for image display by subtracting the corresponding signal levels of the correction data from the signal levels of the light-receiving image data. Here, the region of the light-shielded region 201 from which the average values ​​of the signal levels for each row and column are obtained can be set to any region of the light-shielded region 201. This region is referred to as a clamp value generation region. Furthermore, the average value obtained from the clamp value generation region is referred to as a clamp value.

[0016] 3(a) to 3(c) are conceptual diagrams of OB clamping. FIG. 3(a) shows the photoelectric conversion unit 101 shown in FIG. 2, FIG. 3(b) is a conceptual diagram before OB clamping is performed, and FIG. 3(c) is a conceptual diagram after OB clamping is performed. FIG. 3(b) shows the signal level of the light-shielded region 201 along the vertical axis and the signal level of the light-shielded region 201 along the horizontal axis, and illustrates an example of the average value of the FPN component level and the corresponding signal level. FIG. 3(b) illustrates a conceptual diagram in which the FPN component increases with each row, and shows the clamp value 301 when the entire light-shielded region 201 is used as the clamp value generation region. FIG. 3(c) shows the results of subtracting the clamp value 301 from the signal level of each row. 3(c) shows a state in which signal components in which FPN components and shading components are suppressed are extracted from the image data obtained by the photoelectric conversion unit 101, including the light-receiving area 202, by subtracting the clamp value 301 from the signal level obtained by the light-shielding area 201. In other words, this shows an example in which the OB clamping process by the signal processing unit 152 is performed appropriately.

[0017] However, in OB clamping processing, if the average value of each row of the light-shielded image data of the light-shielded region 201 among the image data is used directly for subtraction, the average value may vary due to the influence of random noise that differs from pixel to pixel, etc. As a result, the shading component may not be reduced appropriately in the OB clamping processing, and the accuracy of the OB clamping processing may decrease.

[0018] Therefore, for example, it is conceivable to use an LPF (Low Pass Filter) on the average value of the darkened image data calculated for each row to make the clamp value resistant to row variations and improve the accuracy of the OB clamping process. For this reason, when the signal processing unit 152 performs the OB clamping process, it is necessary to provide the signal processing unit 152 with control data for performing the OB clamping process. The signal processing unit 152 generates the above-mentioned correction data based on the darkened image data, which is image data of the darkened region 201 among the image data, and the control data. The control data for performing the OB clamping process includes information on the feedback gain of the LPF (the ability to follow shading) as well as information on the size and location of the clamp value generation region in the darkened image data used for the OB clamping process. The control data generation unit 161 arranged in the machine learning unit 151 of the signal processing device 150 generates the control data using the trained model 160. The signal processing unit 152 generates correction data based on the light-shielded image data and the information of these control data, and can generate data for image display, for example, by subtracting the signal level of the correction data from the signal level of the received light image data as described above.

[0019] Next, a process for generating control data will be described with reference to FIG. 4. First, in S401, image data for generating control data is prepared. The image data for generating control data may be image data acquired by previous imaging, or may be image data acquired under equivalent imaging conditions. This image data may be supplied from outside the signal processing device 150 or the photoelectric conversion device 100, or may be stored in a memory 162 arranged in the machine learning unit 151 of the signal processing device 150 as shown in FIG. 1. Furthermore, light-shielded image data, which is data on the light-shielded region 201 among image data acquired in real time, may be used as the image data for generating control data.

[0020] Once the image data for generating the control data is prepared, the process proceeds to steps S402 to S404. In S402, the control data generator 161 determines, as the control data, the size of the area (clamp value generation area) in the light-shielded image data where correction data will be generated, i.e., the average parameter when generating the clamp value. The average parameter when generating the clamp value may be determined, for example, from the amount of noise in the FPN component in the light-shielded area 201. For example, if the amount of noise is large, the clamp value generation area is set large to suppress large variations, and if the amount of noise is small, the size of the clamp value generation area is set small so as not to be unnecessarily affected by blemishes or unevenness present in the light-shielded area 201.

[0021] In step S403, the control data generator 161 determines the location of a region (clamp value generation region) in the light-shielded image data for generating correction data as control data. The location of the clamp value generation region is determined based on the shading shape in the light-shielded region 201 and the location of pixels outputting abnormal signal values. Pixels outputting abnormal signals may be, for example, pixels in the region where the aforementioned blemishes or unevenness occur, or pixels that become defects. Blemishes or unevenness may appear as a collection of pixels that output a signal level that is relatively larger or smaller than other pixels. Pixels that become defects may also be pixels that always output the same signal level, such as white spots or black spots. If the clamp value generation region contains blemishes or unevenness, the clamp value may contain components other than the desired FPN component, potentially making it impossible to reduce the FPN component, resulting in inappropriate OB clamping. Therefore, if blemishes or unevenness exist in the light-shielded image data of the light-shielded region 201, the clamp value generation region may be set to exclude the region where the blemishes or unevenness appear.

[0022] Furthermore, in S404, the control data generator 161 determines the shading tracking capability as control data. For example, the shading tracking capability may be determined from the magnitude of the shading component. For example, if the shading amount is large, the clamp value tracking capability is set large so that the clamp value can track the large shading amount. If the shading amount is small, the clamp value tracking capability is set small so that it is not affected by unintended variations.

[0023] The steps S402 to S404 may be performed in any order. Furthermore, for example, the steps S402 to S404 may be performed simultaneously in parallel. Furthermore, for example, not all steps S402 to S404 may be performed. By performing at least one step and outputting control data from the control data generator 161 to the signal processor 152, the accuracy of the OB clamping process performed by the signal processor 152 may be improved.

[0024] The control data generator 161 outputs control data to the signal processor 152, including at least one of information on the size of the clamp value generation region, information on the location of the clamp value generation region, and information on shading tracking, which were determined in S402 to S404. Next, in S405, the signal processor 152 generates correction data based on the control data determined in S402 to S404 and the light-shielded image data of the light-shielded region 201, and performs OB clamp processing to correct the light-receiving image data of the light-receiving region 202 in accordance with the correction data. This allows the present embodiment to calculate a clamp value suited to the imaging conditions as correction data, enabling appropriate OB clamp processing that can reduce FPN components and shading components. As a result, it is possible to obtain images with good image quality.

[0025] Image data from the photoelectric conversion unit 101 of the photoelectric conversion device 100 may have different FPN components and unevenness for each chip, even under the same imaging conditions, due to scratches that occur during the manufacturing process of the photoelectric conversion unit 101 and the placement location of the chip within the lot. Furthermore, even within the same lot, for example, the thickness of the interlayer film varies between chips, resulting in variations in the wiring capacitance between chips. Differences in the wiring capacitance may also result in variations in the amount and shape of shading between chips. When the chip size of the photoelectric conversion unit 101 is large, such as an image sensor used in a digital single-lens camera, the impact of such variations is even greater. Therefore, it is extremely important for the control data generation unit 161 to tune the trained model 160 used to generate control data before shipment in order to perform OB clamping processing that is suited to the FPN components and unevenness that vary between chips.

[0026] Furthermore, even in the same photoelectric conversion unit 101, the FPN component is a value that can change depending on the imaging conditions, such as the amount of dark current generated depending on the accumulation time when capturing an image, and the amplification factor of the FPN component depending on the ISO setting. Furthermore, even if they are not present at the time of shipment, blemishes and unevenness due to aging may occur in the image output from the photoelectric conversion unit 101. If appropriate OB clamping processing cannot be performed each time taking these effects into account, it will be impossible to reduce the FPN component and shading component, and the accuracy of the OB clamping processing may not be maintained.

[0027] Therefore, in this embodiment, the control data generation unit 161 inputs image data (shaded image data) to the trained model 160 and outputs control data for performing OB clamp processing. That is, the control data generation unit 161 outputs control data for reducing FPN components using a trained model that has learned shading shapes, stains, and unevenness that differ depending on the imaging conditions, lot, and chip.

[0028] In this embodiment, the control data generation unit 161 causes the trained model 160 to output control data for generating the clamp value, rather than outputting the clamp value itself. Furthermore, the signal processing unit 152 generates correction data based on the control data and the light-shielded image data of the light-shielded region 201, and performs OB clamp processing to correct the light-receiving image data of the light-receiving region 202 according to the correction data without applying the trained model 160. This configuration, for example, can reduce the area of ​​the memory 162 compared to when the clamp value itself is stored in the memory 162. Furthermore, when the clamp value itself is stored in the memory 162, it is necessary to access the memory 162 for each frame. However, in this embodiment, when the control data does not need to be changed, such as when continuously capturing images without changing the capturing conditions, there is no need to access the memory 162, and high-frame-rate capturing is possible.

[0029] Furthermore, the machine learning unit 151 may further include a learning unit 163 that updates the trained model 160 through machine learning using image data. By repeatedly capturing images using the photoelectric conversion device 100, the learning unit 163 may, for example, capture statistical characteristics of the image data acquired for the imaging conditions and update the trained model 160. This allows the trained model 160 to adapt to, for example, the occurrence of blemishes or unevenness due to aging. The machine learning performed by the learning unit 163 in this case may be called unsupervised learning.

[0030] FIG. 5 is a schematic diagram of a neural network of a machine learning model in this embodiment. The neural network may include an input layer having multiple nodes, a hidden layer having multiple nodes, and an output layer having one node. Image data is input to each node in the input layer. Each node in the hidden layer is connected to each node in the input layer. Each element of the input value input to each node in the hidden layer is used for calculation at each node in the hidden layer. Each node in the hidden layer calculates a calculated value using, for example, the input value input from each node of the input value, a predetermined weighting coefficient, and a predetermined bias value. Each node in the hidden layer is connected to the output layer and outputs the calculated calculated value to a node in the output layer. The node in the output layer receives the calculated value from each node in the hidden layer. The machine learning model (hidden layer) derives control data from the input image data that can reduce FPN components under various imaging conditions. Performing OB clamping processing based on this derived control data enables highly accurate OB clamping processing that is adaptive not only to FPN components and shading components that vary depending on the imaging conditions and chip, but also to aging. As a result, in this embodiment, it is possible to acquire an image with good image quality.

[0031] Second embodiment A signal processing device according to a second embodiment of the present disclosure will be described with reference to Fig. 6. Fig. 6 is a diagram illustrating an example configuration of a photoelectric conversion device 600 including a signal processing device 150 according to this embodiment. The signal processing device 150 according to this embodiment differs from the first embodiment described above in that, unlike the machine learning unit 151 of the signal processing device 150 shown in Fig. 1, the signal processing device 150 according to this embodiment includes a machine learning unit 651. Compared to the machine learning unit 151, the machine learning unit 651 has an additional signal line for acquiring a signal from outside the photoelectric conversion device 600. Since the configuration other than the machine learning unit 651 may be the same as that of the first embodiment described above, the following description will mainly focus on the differences from the first embodiment described above, and description of the similarities will be omitted as appropriate.

[0032] In the first embodiment described above, it was shown that the control data generation unit 161 can derive control data suitable for each chip or for the imaging conditions from image data using the trained model 160, thereby enabling appropriate OB clamp processing. Also, in the first embodiment, it was explained that the learning unit 163 may perform unsupervised learning. Meanwhile, in this embodiment, the learning unit 163 updates the trained model 160 by performing machine learning using teacher data. Then, similar to the first embodiment described above, the control data generation unit 161 uses the trained model 160 to generate control data used by the signal processing unit 152 in the OB clamp processing.

[0033] More specifically, in this embodiment, data in which FPN components and shading components have been removed from image data is provided from outside the signal processing device 150 (photoelectric conversion device 100). For example, a user provides data in which FPN components and shading components have been removed. The learning unit 163 performs machine learning using the data in which noise has been reduced from image data, provided from outside the signal processing device 150, as training data.

[0034] For example, the learning unit 163 compares the correction data generated by the signal processing unit 152 based on the control data generated by the control data generation unit 161 using the trained model 160 with teacher data in which noise has been reduced from the same image data or image data obtained under the same imaging conditions. This allows the learning unit 163 to perform machine learning so that the control data generation unit 161 can generate control data suited to the chip and imaging conditions using the trained model 160. Based on the comparison result, the learning unit 163 updates the trained model 160. Furthermore, for example, the learning unit 163 may perform machine learning using as teacher data data that is input from outside the signal processing device 150 and corresponds to the control data output by the control data generation unit 161 that can reduce noise in image data.

[0035] The machine learning performed by the learning unit 163 may be performed, for example, before shipping the photoelectric conversion device 100. Furthermore, after shipping the photoelectric conversion device 100, the learning unit 163 may perform machine learning, for example, when starting up or shutting down the photoelectric conversion device 100, or while the battery is being charged. Furthermore, the learning unit 163 may perform machine learning as needed in response to a request from a user or a photoelectric conversion system described below.

[0036] In the first embodiment and the present embodiment described above, the learning unit 163 is arranged in the machine learning unit 651, but this is not limiting. For example, the learning unit 163 may be arranged in the signal processing unit 152, or may be arranged separately from the machine learning unit 151 and the signal processing unit 152.

[0037] In this embodiment, machine learning is performed using training data. By performing OB clamping based on control data generated by the control data generator 161 using the trained model 160, which is updated as needed, it is possible to accurately reduce FPN components and shading components that vary depending on the imaging conditions and chip. Furthermore, highly accurate OB clamping that is also adaptive to aging is possible. As a result, in this embodiment, it is possible to acquire images with good image quality.

[0038] Third embodiment A signal processing device according to a third embodiment of the present disclosure will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating an example configuration of a photoelectric conversion device 700 including a signal processing device 150 according to this embodiment. The signal processing device 150 according to this embodiment differs from the first embodiment described above in that, unlike the machine learning unit 151 of the signal processing device 150 shown in FIG. 1, the signal processing device 150 according to this embodiment includes a machine learning unit 751. Compared to the machine learning unit 151, the machine learning unit 751 has an additional signal line for acquiring a signal from the signal processing unit 152. The machine learning unit 751 also includes an analysis unit 750. Since the configuration of the machine learning unit 751 may be the same as that of the first embodiment described above, the following description will mainly focus on the differences from the first embodiment described above, and descriptions of the similarities will be omitted as appropriate.

[0039] In the second embodiment described above, the learning unit 163 performs machine learning using data received from outside the signal processing device 150 (photoelectric conversion device 100) as training data, and updates the trained model 160. In this embodiment, the signal processing unit 152 performs OB clamping processing using control data generated by the control data generation unit 161, and the result is used as training data for learning. More specifically, the analysis unit 750 analyzes the degree of noise reduction in the correction data generated by the signal processing unit 152 based on the control data generated by the control data generation unit 161. The analysis unit 750 analyzes the correction data generated by the signal processing unit 152, determines whether the accuracy of the OB clamping processing is good or bad (the degree of noise reduction in the correction data), and feeds back the result as training data, and the learning unit 163 performs machine learning. That is, in this embodiment, unlike the second embodiment described above, training data is generated inside the signal processing device 150. The signal line from the signal processing unit 152 to the machine learning unit 751 shown in FIG. 7 is arranged to transfer the corrected image data on which the OB clamping process has been performed by the signal processing unit 152 to the analysis unit 750.

[0040] The machine learning performed by the learning unit 163 in this embodiment may be called reinforcement learning. Machine learning may be performed each time the signal processing unit 152 corrects received-light image data in accordance with correction data to generate corrected image data. Increasing the amount of training data may further improve the accuracy of the OB clamping process.

[0041] An example of a method for analyzing the degree of FPN component removal by OB clamping, performed by the analysis unit 750, is shown below. For the input image data, the analysis unit 750 sets an arbitrary region in the light-shielded image data of the light-shielded region 201. Here, this region is referred to as the FPN shape calculation region. Next, the analysis unit 750 calculates the average signal level for each row and column of the FPN shape calculation region and calculates an approximation curve for the average values. This approximation curve becomes a function that represents the FPN shape. The ideal state in which the FPN component is completely removed is defined as a state in which the slope of the approximation curve is zero. The degree of FPN component reduction can be determined by calculating the correlation between the approximation curve calculated by the analysis unit 750 and a function in which the slope of the approximation curve for the ideal state is set to zero. The degree of noise reduction analyzed by the analysis unit 750 is input to the learning unit 163 and used as training data.

[0042] In the present embodiment, the learning unit 163 and the analysis unit 750 are arranged in the machine learning unit 751, but this is not limiting. For example, the learning unit 163 and the analysis unit 750 may be arranged in the signal processing unit 152, or may be arranged separately from the machine learning unit 151 and the signal processing unit 152, respectively. Furthermore, the learning unit 163 and the analysis unit 750 may be integrated into one unit. When a user uses the photoelectric conversion device 100 under various imaging conditions, a wide range of training data can be obtained, and improvement in learning accuracy in a shorter period of time is expected.

[0043] In this embodiment, training data is generated within the signal processing device 150 and machine learning is performed. By performing OB clamping based on control data generated by the control data generation unit 161 using the appropriately updated trained model 160, it is possible to accurately reduce FPN components and shading components that vary depending on the imaging conditions and chip. Furthermore, highly accurate OB clamping that is also adaptive to aging is possible. As a result, in this embodiment, it is possible to acquire images with good image quality.

[0044] Fourth embodiment A signal processing device according to a fourth embodiment of the present disclosure will be described with reference to Figures 8 and 9. Figure 8 is a diagram showing an example configuration of a photoelectric conversion device 800 including a signal processing device 150 of this embodiment. The signal processing device 150 of this embodiment differs from the first embodiment described above in that, unlike the machine learning unit 151 of the signal processing device 150 shown in Figure 1, the signal processing device 150 of this embodiment includes a machine learning unit 851. Compared to the machine learning unit 151, the machine learning unit 851 further includes an extraction unit 850. Since the configuration other than the machine learning unit 851 may be the same as that of the first embodiment described above, the following description will mainly focus on the differences from the first embodiment described above, and description of the similarities will be omitted as appropriate.

[0045] In the first embodiment described above, image data is input to each node in the input layer of the neural network of the trained model 160. In this embodiment, the machine learning unit 851 is provided with an extraction unit 850, which extracts features from the image data to improve the accuracy of the control data and inputs the extracted features to each node in the input layer of the neural network of the trained model 160. Instead of inputting image data to the neural network of the trained model 160 as in the above-described embodiments, information on the features extracted from the image data is input to the neural network. This improves the accuracy of the control data output by the control data generation unit 161 to the signal processing unit 152 using the trained model 160, and can improve the accuracy of the OB clamping process. Here, it is described that only features are input to the trained model 160; however, both image data and the features of the image data may be input to the neural network of the trained model 160.

[0046] Fig. 9 shows a process for generating control data in this embodiment. Compared to the process shown in Fig. 4, a step S901 for extracting feature quantities from image data is added. The other steps may be the same as the steps shown in Fig. 4, and therefore descriptions thereof will be omitted as appropriate.

[0047] After image data is prepared in S401, the process proceeds to S901, where the extraction unit 850 extracts a feature from the image data to improve the accuracy of the control data. The feature may be, for example, the amount of FPN noise in the light-shielded image data of the light-shielded region 201 to determine the size of the clamp value generation region. Alternatively, the feature may be, for example, the shape of shading in the light-shielded image data of the light-shielded region 201 to determine the location of the clamp value generation region. Alternatively, the feature may be, for example, the magnitude of the shading component in the light-shielded image data of the light-shielded region 201 to determine the shading followability. As described above, the presence of blemishes, unevenness, or scratches may reduce the accuracy of the clamp value. The presence of a step within the clamp value generation region may also reduce the accuracy of the clamp value. Here, a "step" in this embodiment may refer to a state in which the signal value has an offset at a certain row or column boundary. Therefore, the feature may be, for example, information about a pixel 102 that outputs an abnormal signal value in the light-shielded image data of the light-shielded region 201. The extraction unit 850 inputs at least one of these pieces of information into the trained model 160 as a feature.

[0048] Here, an example of a method for deriving the amount of FPN noise in the dark image data of the dark region 201 is shown. Multiple image data acquired under the same imaging conditions are prepared. Each pixel value of these image data is averaged in the time direction. This averaging process makes it possible to remove random noise components. An arbitrary region is set within the dark image data for the image data averaged in the time direction, and the standard deviation within this region is calculated. This standard deviation becomes the amount of FPN noise in the dark image data of the dark region 201.

[0049] The method for deriving the shape of shading and the magnitude of shading components in the light-shielded image data of the light-shielded region 201 is the same as the analysis performed by the analysis unit 750 in the third embodiment described above, and therefore will not be described here.

[0050] Next, an example of a method for deriving the location of pixels 102 that output abnormal signal values, such as blemishes, unevenness, scratches, or the above-mentioned step, in the light-shielded image data of the light-shielded region 201 is shown. The light-shielded image data of the light-shielded region 201 is divided into arbitrary regions. The average signal level within each divided region is obtained. One region is selected from the divided regions, and the average signal level of each region is normalized using the average signal level of the selected region. In this case, a region whose average value is different from that of other regions can be determined to be a region containing pixels 102 that cause blemishes, unevenness, scratches, or step. Furthermore, information about pixels 102 that output abnormal signal values, such as blemishes, unevenness, scratches, or step, known at the time of shipment, may be stored in memory 162 or the like. Here, the information about pixels 102 that output abnormal signal values ​​stored in memory 162 may be position information about pixels 102 that output abnormal signal values ​​or the signal value level of pixels 102 that output abnormal signal values. Furthermore, even after shipment, information about pixels 102 that output abnormal signal values ​​extracted by the extraction unit 850 or the like may be stored in the memory 162. This can reduce the amount of calculation in the extraction unit 850 when extracting feature amounts.

[0051] After the extraction unit 850 extracts the feature amounts, it inputs the feature amounts to the trained model 160. Next, in S402 to S404, the control data generation unit 161 uses the trained model 160 to which the feature amounts have been input to output to the signal processing unit 152 control data to be used when the signal processing unit 152 performs OB clamp processing. Based on this information derived from the image data, the signal processing unit 152 determines each setting value required for OB clamp processing. Next, in S405, the signal processing unit 152 generates correction data based on the control data determined in S402 to S404 and the light-shielded image data of the light-receiving area 201, and performs OB clamp processing to correct the light-receiving image data of the light-receiving area 202 in accordance with the correction data.

[0052] In this embodiment, the learning unit 163 and the extraction unit 850 are arranged in the machine learning unit 851, but this is not limiting. For example, the learning unit 163 and the extraction unit 850 may be arranged in the signal processing unit 152, or may be arranged separately from the machine learning unit 151 and the signal processing unit 152, respectively.

[0053] In this embodiment, feature quantities extracted from image data are input to the neural network of the trained model 160, thereby improving the accuracy with which the control data generator 161 generates control data. This enables the signal processor 152 to perform more appropriate OB clamping. In other words, this embodiment also makes it possible to accurately reduce FPN components and shading components that vary depending on the imaging conditions and chip. Furthermore, it enables highly accurate OB clamping that is also adaptive to aging. As a result, this embodiment makes it possible to acquire images with good image quality.

[0054] The first to fourth embodiments described so far may be combined with each other. For example, the trained model 160 may be updated using data input from outside the signal processing device 150 as training data, and feature quantities of image data may be input to the updated trained model 160. Furthermore, for example, the trained model 160 may be updated using data input from outside the signal processing device 150 and data obtained by analyzing the results of OB clamping processing performed by the signal processing unit 152 as training data. Furthermore, for example, the trained model 160 may be updated using data input from outside the signal processing device 150 and data obtained by analyzing the results of OB clamping processing performed by the signal processing unit 152 as training data, and feature quantities of image data may be input to the updated trained model 160.

[0055] In this way, by combining the above-described embodiments, a trained model 160 can be obtained that is adapted with higher accuracy to FPN components, shading components, and aging changes that vary depending on the imaging conditions and each chip. This makes it possible to realize a signal processing device 150 that can perform OB clamping processing with high accuracy. Furthermore, in the photoelectric conversion devices 100, 600, 700, and 800 incorporating the signal processing device 150 of the present disclosure, it becomes possible to obtain high-quality images.

[0056] Other embodiments The signal processing device 150 that performs the OB clamping process described in each of the above embodiments may be provided inside the photoelectric conversion device 100 as described above. However, this is not limited to this. For example, the signal processing device 150 may be arranged separately from the photoelectric conversion device 100 or the photoelectric conversion unit 101. The signal processing device 150 may be a computer such as a personal computer that includes a processor (for example, a CPU or an MPU) that is arranged separately from the photoelectric conversion device 100. Furthermore, for example, the signal processing device 150 may be a circuit such as an ASIC that realizes the above-mentioned functions.

[0057] FIG. 10 is a diagram showing an example of an arrangement of the blocks of the photoelectric conversion device 100 shown in FIG. 1 on a substrate such as a semiconductor. The photoelectric conversion device 100 may include a substrate 1001 and a substrate 1002 made of a semiconductor such as silicon. Components such as a signal processing device 150, a vertical scanning unit 103, a control unit 104, a readout circuit unit 105, an AD conversion unit 106, a memory unit 107, and a horizontal scanning unit 108 are arranged on the substrate 1001. A photoelectric conversion unit 101 having an array of pixels 102 each having a light-receiving region 202 and a light-shielding region 201 is arranged on the substrate 1002. As shown in FIG. 10, at least a portion of the substrates 1001 and 1002 may be stacked. This configuration allows suitable processes to be selected for the analog unit including the photoelectric conversion unit 101 and the logic unit including the signal processing device 150 when manufacturing the photoelectric conversion device 100. By using a manufacturing process appropriate for each component, it may be possible to obtain good characteristics for each component included in the photoelectric conversion device 100. As a result, a photoelectric conversion device 100 with improved image quality can be obtained.

[0058] 11 shows the configuration of an imaging device 1100 as an example of the photoelectric conversion devices 100, 600, 700, and 800 incorporating the above-described signal processing device 150. The imaging device 1100 includes a signal generation unit 1101, a signal correction unit 1102, a CPU 1103, an external input unit 1104, an optical system 1105, an image display unit 1106, a recording unit 1107, and a drive system 1108.

[0059] The signal correction unit 1102 may be the above-described signal processing device 150. The signal generation unit 1101 may include components such as the photoelectric conversion unit 101 including the above-described pixels 102, the vertical scanning unit 103, the control unit 104, the readout circuit unit 105, the AD conversion unit 106, the memory unit 107, and the horizontal scanning unit 108. Therefore, a configuration including the signal generation unit 1101 and the signal correction unit 1102 may be the above-described photoelectric conversion devices 100, 600, 700, and 800. In this configuration, the signal generation unit 1101 alone may also be referred to as a photoelectric conversion device. In other words, the signal processing device 150 may be arranged separately from the photoelectric conversion device, as described above.

[0060] In response to light incident through an optical system 1105 for making light incident on the photoelectric conversion unit 101 of the signal generation unit 1101, the signal generation unit 1101 performs photoelectric conversion to generate an analog image signal, and outputs image data by performing AD conversion. The output image data is corrected by a signal correction unit 1102 so that it can be output to and stored in a video display unit 1106 and a recording unit 1107. The video display unit 1106 displays an image using the display image data after the correction process. The recording unit 1107 stores the display image data. The CPU 1103 controls the various components of the imaging device 1100 described above. The drive system 1108 is arranged to operate, for example, the focus and aperture of the optical system 1105. The external input unit 1104 can be various buttons or the like through which the user inputs and operates imaging conditions, shutter operation, etc. A touch panel may be arranged as the video display unit 1106, and the video display unit 1106 may function as (part of) the external input unit 1104.

[0061] 11, the imaging device 1100 may include a component, such as a thermometer, for obtaining other environmental information, such as an image obtained by the signal generating unit 1101. For example, the signal processing device 150 may perform machine learning using information, such as temperature information, that cannot be obtained by the signal generating unit 1101 (photoelectric conversion device), to construct the trained model 160. By using various environmental parameters of the environment in which the imaging device 1100 is used, it may be possible to perform OB clamping processing that more appropriately corresponds to the shooting conditions. As a result, the image quality of the image obtained by the imaging device 1100 may be improved.

[0062] 12 is a diagram showing an example configuration of a photoelectric conversion system 1200 including a plurality of photoelectric conversion devices 100, 600, 700, and 800, and a communication unit 1202 for communicating with a server 1201 (e.g., a cloud server) external to the photoelectric conversion devices 100, 600, 700, and 800. The communication unit 1202 may be realized by having a function such as wireless LAN or Bluetooth (registered trademark) for communication between the photoelectric conversion devices 100, 600, 700, and 800. The communication unit 1202 may also be realized by the Internet or the like, which allows a user to upload or download various data that can be acquired by the photoelectric conversion devices 100, 600, 700, and 800 to or from the server 1201. The communication unit 1202 may be in any form as long as it can exchange data between the photoelectric conversion device 100, 600, 700, 800 and the server 1201 disposed outside the photoelectric conversion device 100, 600, 700, 800.

[0063] For example, the trained model 160 of the signal processing device 150 of the photoelectric conversion device 100, 600, 700, or 800 may be shared among the multiple photoelectric conversion devices 100, 600, 700, or 800 via the server 1201. For example, when the photoelectric conversion device 100, 600, 700, or 800 is connected to the server 1201 via the communication unit 1202, the learning unit 163 of the signal processing device 150 may perform machine learning and update the trained model 160.

[0064] Furthermore, for example, the photoelectric conversion device may not include the signal processing device 150 as described above, and the signal processing device 150 may be arranged in the server 1201. In other words, the trained model 160 generated by machine learning may be arranged on the server 1201.

[0065] The photoelectric conversion system 1200 having such a configuration enables the trained model 160 to be updated using information from multiple photoelectric conversion devices (e.g., photoelectric conversion devices 100, 600, 700, 800). In other words, photoelectric conversion devices used by users who capture images under a wide range of imaging conditions are connected to the server 1201 of the photoelectric conversion system 1200. This makes it possible, for example, to acquire a large amount of training data in a short period of time. As a result, the accuracy of OB clamping processing is further improved, enabling even higher image quality for images captured by each photoelectric conversion device.

[0066] The present invention can also be realized by executing the following process: software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and one or more processors (e.g., CPU or MPU) in the computer of the system or device read and execute the programs. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0067] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0068] 150: signal processing device, 152: signal processing unit, 160: trained model, 161: control data generation unit, 201: light-shielding region, 202: light-receiving region

Claims

1. A signal processing device that processes image data output from a photoelectric conversion unit having a light receiving region and a light blocking region, a control data generation unit that outputs control data used to generate correction data for correcting the image data by inputting the image data output from the photoelectric conversion unit or image data acquired by past imaging into a trained model generated by machine learning; and A signal processing unit that generates the correction data by performing averaging processing for each row and column on the light-shielded image data, which is image data of the light-shielded region among the image data, and corrects the light-receiving image data without applying the trained model by subtracting the correction data from the light-receiving image data, which is image data of the light-receiving region among the image data; Including, The signal processing device characterized in that the control data is at least one of information regarding the size of the area in the light-blocking image data where the averaging process is performed, information regarding the location of the area in the light-blocking image data where the averaging process is performed, and information regarding the gain of a low-pass filter to be applied to the average value obtained by the averaging process.

2. an extraction unit that extracts a feature amount from the image data to improve the accuracy of the control data; The signal processing device according to claim 1 , wherein the feature amount is input to the trained model.

3. 3. The signal processing device according to claim 2, wherein the feature amount includes at least one of an amount of fixed pattern noise in the dark image data, a shape of shading in the dark image data, a magnitude of shading components in the dark image data, and information on pixels that output abnormal signal values ​​in the dark image data.

4. The signal processing device according to claim 1 , further comprising a learning unit that updates the trained model by machine learning using the image data.

5. The signal processing device according to claim 4, characterized in that the learning unit updates the trained model by performing machine learning using data in which noise has been reduced from the image data, which is provided from outside the signal processing device, as training data.

6. The signal processing device described in claim 4 or 5, characterized in that the learning unit updates the learned model by performing machine learning using data corresponding to the control data capable of reducing noise in the image data, which is provided from outside the signal processing device, as training data.

7. further including an analysis unit that analyzes the degree of noise reduction in the correction data generated based on the control data; 7. The signal processing device according to claim 4, wherein the learning unit updates the trained model by performing machine learning using the degree of noise reduction in the correction data analyzed by the analysis unit as training data.

8. The signal processing device according to claim 7 , wherein the machine learning is performed every time the signal processing unit generates the correction data.

9. A signal processing device according to any one of claims 1 to 8; a photoelectric conversion unit including a light receiving region and a light blocking region, and outputting image data processed by the signal processing device; A photoelectric conversion device comprising:

10. the photoelectric conversion device includes a first substrate and a second substrate; 10. The photoelectric conversion device according to claim 9, wherein the signal processing device is disposed on the first substrate, and the photoelectric conversion unit is disposed on the second substrate.

11. The photoelectric conversion device according to claim 10 , wherein at least a portion of the first substrate and the second substrate are laminated.

12. The photoelectric conversion device according to any one of claims 9 to 11, a communication unit for communicating with a server external to the photoelectric conversion device; A photoelectric conversion system comprising: A photoelectric conversion system characterized in that the trained model is shared with a trained model of a photoelectric conversion device other than the photoelectric conversion device via the server.

13. A signal processing device according to any one of claims 1 to 8; a plurality of photoelectric conversion devices each including a light receiving region and a light blocking region, and a photoelectric conversion unit that outputs image data processed by the signal processing device; a communication unit that performs communication between the signal processing device and the plurality of photoelectric conversion devices; A photoelectric conversion system comprising:

14. A control method for a signal processing device that processes image data output from a photoelectric conversion unit having a light receiving region and a light blocking region, comprising: a step of inputting the image data output from the photoelectric conversion unit or image data acquired by previous imaging into a trained model generated by machine learning, thereby outputting control data used to correct the image data; generating correction data by performing averaging processing for each row and column on the light-shielded image data, which is image data of the light-shielded region among the image data; a step of correcting the received light image data by subtracting the correction data from the received light image data, which is image data of the light receiving region, among the image data; Including, A control method for a signal processing device, characterized in that the control data is at least one of information regarding the size of the area in the light-blocking image data where the averaging process is performed, information regarding the location of the area in the light-blocking image data where the averaging process is performed, and information regarding the gain of a low-pass filter to be applied to the average value obtained by the averaging process.

15. A program for causing a computer to execute each step of the method for controlling a signal processing device according to claim 14.

Images