Defective pixel repairing method and device, image sensor and storage medium

Abstract

The invention belongs to the technical field of image sensors, and particularly relates to a defective pixel repairing method and device, an image sensor and a storage medium. The method comprises the following steps: acquiring defect pixel data of the image sensor; taking K rows and / or K columns of pixel data taking the defective pixel as the center as input data, and inputting the input data into a defective pixel repairing model obtained by pre-training; predicting the input data by using the defective pixel repair model to obtain a repaired pixel value; replacing the defective pixel by using the repair pixel value so as to repair the defective pixel; wherein the size of a convolution kernel of the defect pixel repair model is K * K, K = 2N * T + 1, N is a positive integer, and T is an arrangement period of pixel units in the image sensor, the image sensor defect pixel repair method guarantees a good repair effect and is small in calculation amount, so that the defect pixel repair model can be directly integrated in the image sensor.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image sensors, and particularly relates to a defective pixel repair method and device, an image sensor, and a storage medium. BACKGROUND

[0002] In an image sensor, some pixels may not be able to normally receive light signals due to defects in the manufacturing process or aging during use, appearing as high-light or dark scattered points on image data. These pixels are called defective pixels or bad pixels. The types of bad pixels can include:

[0003] Hot Pixel: fixedly maintains a high pixel value, generally appearing as a high-light point in the picture; Dead Pixel: fixedly maintains a low pixel value, generally appearing as a dark point in the picture; Noise Pixel: the signal does not conform to the normal variation law with the change of light intensity.

[0004] If there are bad pixels in the image sensor, after interpolation (such as Demosaic) and filtering of the image and other nonlinear processes, the size of the bad pixels will increase (bad pixel diffusion), and due to color repair and crosstalk compensation, the intensity and saturation of the color at the bad pixels will also be significantly improved, so the bad pixels need to be repaired before other modules are executed.

[0005] In some cases, a large number of adjacent bad pixels may appear in the image sensor, even forming a bad row or a bad column. As shown in FIG. 1, taking a bad row in a Bayer array image sensor as an example, at this time, due to the 5x5 neighborhood, the reference pixels in the same row as a certain bad pixel are also bad pixels, resulting in a great damage to the information that can be referred to during repair. Especially when the area where the bad row is located is a horizontal line, it is difficult to estimate the value of the center bad pixel through the pixels in the vertical or diagonal direction. Even if it is a gradient-based method, the gradient in the horizontal direction cannot be calculated in this case, and the repair effect is limited. Figure 1 SUMMARY

[0006] Therefore, the embodiments of the present application provide a defective pixel repair method and device, an image sensor, and a storage medium, which have good pixel repair effect, small calculation amount, and can be integrated in the image sensor.

[0007] A first aspect of the embodiments of the present application provides a defective pixel repair method, comprising:

[0008] Obtaining defective pixel data of an image sensor;

[0009] Inputting K rows and / or K columns of pixel data centered on the defective pixel as input data to a defective pixel repair model trained in advance; ​

[0010] predicting the input data by using the defective pixel repairing model to obtain a repaired pixel value;

[0011] replacing the defective pixel by using the repaired pixel value to repair the defective pixel;

[0012] wherein a size of a convolution kernel of the defective pixel repairing model is KxK, K=2N*T+1, N is a positive integer, and T is an arrangement period of a pixel unit in the image sensor.

[0013] A second aspect of the embodiment of the present application provides a defective pixel repairing apparatus, comprising:

[0014] a collecting module configured to acquire defective pixel data of an image sensor;

[0015] an input module configured to input K rows and / or K columns of pixel data centered on the defective pixel as input data to a defective pixel repairing model trained in advance;

[0016] a predicting module configured to predict the input data by using the defective pixel repairing model to obtain a repaired pixel value;

[0017] a repairing module configured to replace the defective pixel by using the repaired pixel value to repair the defective pixel;

[0018] wherein a size of a convolution kernel of the defective pixel repairing model is KxK, K=2N*T+1, N is a positive integer, and T is an arrangement period of a pixel unit in the image sensor.

[0019] A third aspect of the embodiment of the present application provides an image sensor, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the defective pixel repairing method provided in the first aspect of the embodiment of the present application when executing the computer program.

[0020] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the steps of the defective pixel repairing method provided in the first aspect of the embodiment of the present application.

[0021] The first aspect of the embodiments of the present application provides a defective pixel repairing method, which comprises the following steps: acquiring defective pixel data of an image sensor; inputting K rows and / or K columns of pixel data centered on the defective pixel as input data into a defective pixel repairing model which is obtained by pre-training; predicting the input data by using the defective pixel repairing model to obtain a repaired pixel value; and replacing the defective pixel with the repaired pixel value to repair the defective pixel; wherein the size of the convolution kernel of the defective pixel repairing model is KxK, K=2N*T+1, N is a positive integer, and T is an arrangement period of a pixel unit in the image sensor. In the embodiments of the present application, K rows and / or K columns of pixel data centered on the defective pixel are used as input, so that the model can make full use of the context information around the defective pixel to repair the defective pixel when repairing a bad row or a bad column. The model convolution kernel is KxK, K=2N*T+1, which ensures that the pixels corresponding to the convolution kernel have the same arrangement rule, thereby reducing the difficulty of model repair and ensuring good repair effect, and the calculation amount is small, so that the defective pixel repairing model can be directly integrated into the image sensor.

[0022] It can be understood that the beneficial effects of the second and third aspects described above can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0024] Figure 1 is a schematic diagram of a bad column distribution in a Bayer array image sensor;

[0025] Figure 2 is a flowchart of a defective pixel repairing method provided by an embodiment of the present application;

[0026] Figure 3 is a flowchart of a defective pixel repairing method provided by another embodiment of the present application;

[0027] Figure 4 is a schematic diagram of a convolution kernel in a Bayer array image sensor provided by an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of a repaired pixel value provided by an embodiment of the present application;

[0029] Figure 6is a flowchart of a defect pixel repair method provided by another embodiment of the present application;

[0030] Figure 7 is a flowchart of a defect pixel repair method provided by another embodiment of the present application;

[0031] Figure 8 is a structural diagram of a knowledge distillation device provided by an embodiment of the present application;

[0032] Figure 9 is a structural diagram of an image sensor provided by an embodiment of the present application. DETAILED DESCRIPTION

[0033] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0034] It should be understood that the term “includes” when used in the present application and the appended claims indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0035] It should also be understood that the term “and / or” when used in the present application and the appended claims indicates that one or more of the associated listed items can be present and / or can be combined with any other one or more of the associated listed items.

[0036] As used in the present application and the appended claims, the term “if’ can be construed to mean “when” or “once” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [a described condition or event] is detected” can be construed to mean “once it is determined” or “in response to determining” or “once [the described condition or event] is detected” or “in response to detecting [the described condition or event],” depending on the context.

[0037] In addition, in the description of the present application and the appended claims, the terms “first,” “second,” “third,” etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0038] Reference within the specification of this application to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in additional embodiments," and so on, in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily referring to some, but not all, embodiments. The terms "including," "comprising," "having," and variations thereof, mean "including but not limited to," unless expressly specified otherwise.

[0039] The defect pixel repair method provided by the embodiments of the present application comprises the following steps: acquiring defect pixel data of an image sensor; inputting K rows and / or K columns of pixel data centered on the defect pixel as input data into a defect pixel repair model trained in advance; predicting the input data by using the defect pixel repair model to obtain a repair pixel value; and replacing the defect pixel with the repair pixel value to repair the defect pixel. The size of the convolution kernel of the defect pixel repair model is KxK, and K=2N*T+1, where N is a positive integer and T is the arrangement period of a pixel unit in the image sensor. In the embodiments of the present application, K rows and / or K columns of pixel data centered on the defect pixel are used as input, so that the model can make full use of the context information around the defect pixel to repair the defect pixel when repairing a bad row or a bad column. The model convolution kernel is KxK, K=2N*T+1, which is beneficial to making the pixels corresponding to the convolution kernel have the same arrangement rule, thereby reducing the difficulty of model repair and achieving good repair effect, and the calculation amount is small, so that the defect pixel repair model can be integrated in the image sensor directly. Of course, the present application is also applicable to bad pixel repair.

[0040] In applications, the image sensor can be any one of a CMOS image sensor (Complementary Metal-Oxide-Semiconductor Image Sensor), a CCD sensor (Charge-Coupled Device), a 3D ToF (Time of Flight) sensor, a thermal image sensor, a backside illumination (BSI) sensor, and a stacked sensor. The arrangement mode of the pixel units in the image sensor can be any one of a Bayer pattern, an X-Trans pattern, an RGBW pattern, an RGGB pattern, and a Quad Bayer pattern.

[0041] Specifically, the application designs an image sensor defect repair algorithm, which can repair the obvious defects such as bad rows and bad columns in the image sensor to a certain extent. In addition, the most common defect in the sensor is the bad pixel, which can be regarded as a special case of bad row and bad column, that is, the row and column numbers are both 1. Therefore, as long as KxK pixels centered on the bad pixel are read out, the algorithm proposed in the application can also be used for repair. For the bad pixel, since the pixels in the same row and column can be used as a reference, a smaller K value can be taken in the subsequent optimization and repair, and the repair effect can be optimized by increasing the channel number C.

[0042] The arrangement period of the pixel unit in the image sensor can be 2x2, 4x4, 6x6, etc. for the above various arrangement modes, so that T can be any one of 2, 4, 6, etc. For example, in the Bayer array, every 2x2 pixels form a basic pixel unit, and one pixel unit includes 2 G pixels, 1 B pixel and 1 R pixel. For B and R pixels, there is only one pixel with the same color channel in every 2 rows or 2 columns, so T is selected as 2, which is the arrangement period of the pixel unit in the image sensor.

[0043] As shown in Figure 2 The defect pixel repair method provided by the embodiment of the application includes the following steps S101-S104:

[0044] Step S101, acquire the defect pixel data of the image sensor.

[0045] In applications, the defective pixel data of the image sensor can be obtained by the image sensor itself or from external devices used for detecting the image sensor. For example, in some implementations, the bad rows (or bad columns, etc.) can be calibrated when the sensor is shipped, or some specific algorithm can be used to identify new bad rows (or bad columns, etc.) generated during the use of the sensor, and the positions of the bad rows (or bad columns, etc.) can be written into the memory. The defective pixel data at least includes the positions of the bad pixels, which record the specific coordinates of each bad pixel on the image sensor, i.e., the row and column positions of the pixel. In addition, the defective pixel data can also include the types of the bad pixels, which can be divided into static bad pixels and dynamic bad pixels. The static bad pixels are generated during the manufacturing process of the sensor and have fixed positions. The dynamic bad pixels can appear or disappear due to changes in environmental temperature, light intensity, etc. The defective pixel data can also include the characteristics of the bad pixels, such as hot pixels, dead pixels, and noise pixels. The defective pixel data can be pre-stored in the non-volatile memory (such as OTP-One Time Programmable memory) of the image sensor. Thus, when the image sensor finishes sensing light and reads out the data, the defective pixel data and the K rows and / or K columns of pixel data centered on the defective pixel can be read from the non-volatile memory to the buffer of the image sensor, so as to obtain the input data of K rows or K columns or KxK matrix, in order to call the defective pixel repair model to repair the defective pixel.

[0046] In step S102, the K rows and / or K columns of pixel data centered on the defective pixel are input as input data to the defective pixel repair model pre-trained.

[0047] In applications, after obtaining the defective pixel data, the coordinates of the defective pixel in the defective pixel data are located to the defective pixel, the bad row, the bad column, etc., and K rows or K columns of pixel data above and below the defective pixel are selected as the input data of the defective pixel repair model. K=2N*T+1, N is a positive integer, and T is the arrangement period of the pixel unit in the image sensor. When the image sensor adopts a Bayer array, the period T is 2, and when N is 1, K=5, i.e., in addition to the row or column where the defective pixel is located, 2 rows or 2 columns of pixel data above and below or left and right of the defective pixel are also taken as the input data, which is used to provide sufficient context information for the defective pixel repair model to repair and to realize accurate prediction. For example, in the bad column repair, when the sensor reads out the data, K (convolution kernel size) pixels in each row centered on the bad column are read into the buffer at the same time, so as to obtain the input data of HxK (H is the height of the original image).

[0048] In an application, for two-dimensional image data, when a convolutional neural network is used for processing, for Raw data in an image sensor, due to the use of different color filters by each pixel, the pixel values are different in space, if a 3x3 convolution kernel is used to calculate the data by sliding, the corresponding data of each convolution kernel may be different arrangement, in addition, due to the existence of bad lines, when the convolution kernel slides through the bad lines, more different corresponding relationships will be generated, which will greatly increase the learning difficulty of the network with limited parameter amount, and limit the repair effect. Taking bad line repair as an example, for K-line data input of the bad line repair application, a KxK size convolution kernel is used, the bad line is taken as the center, and the calculation is performed by sliding in the horizontal direction. For a row of pixels, the convolution kernel can only encounter 2 different arrangements, in addition, for different rows, B pixels and R pixels can also be regarded as one arrangement, so that the repair effect can be improved.

[0049] Step S103, predicting the input data by using the defective pixel repair model to obtain a repaired pixel value.

[0050] In an application, the defective pixel repair model is a model for realizing pixel repair obtained by pre-training, the size of the convolution kernel of the model is KxK, K=2N*T+1, N is a positive integer, and T is the arrangement period of the pixel unit in the image sensor. The model can output corresponding predicted data, i.e., a repaired pixel value, based on the input defective pixel data. The repaired pixel value is predicted based on the normal pixel data around the defective pixel.

[0051] Step S104, replacing the defective pixel with the repaired pixel value to repair the defective pixel.

[0052] In an application, after the image sensor defective pixel repair model outputs the repaired pixel value, the original defective pixel is replaced with the repaired pixel value, thereby realizing the repair of the defective pixel of the image sensor.

[0053] The method uses an ultra-light convolutional neural network to repair the obvious defects such as bad lines and bad columns in the image sensor to a certain extent by using a small amount of calculation. It has the advantages of good repair effect, simple algorithm, easy development and application, flexible form, can adapt to various defect problems, and small amount of calculation. Due to the simplicity and small amount of calculation of the algorithm, the algorithm can be realized by simple hardware and integrated into an image sensor of advanced technology to improve the quality and service life of the sensor.

[0054] In one embodiment, step S101 comprises:

[0055] At least one of the defective pixel data obtained in the factory calibration stage and the defective pixel data obtained in the use stage is acquired.

[0056] In an application, after the manufacture of a sensor chip is completed, the pixel array thereof is fixed, and each pixel unit contains a photodiode and related signal processing circuit. However, during the manufacture of the image sensor, due to impure materials, process deviation, electrostatic discharge, pollution or other random events in the manufacturing process, pixel-level or row-column-level damage may occur, that is, there are defective pixels. By factory stage calibration inspection and recording of any congenital defects of the sensor during the manufacturing and assembly process, it can be ensured that the sensor can provide standard-compliant performance when first used, and at the same time, initial defective pixel (including bad row, bad column and bad point, etc.) information is provided for later image processing and compensation. In addition, even after factory stage calibration, when the sensor leaves the production line and is formally put into use, it may still cause acquired bad row and other defective pixels due to wear, environmental factors (such as temperature, humidity), voltage fluctuation or physical damage, etc. The identification of acquired defective pixels can be continuously performed throughout the life cycle of the sensor, such as at each boot or regular maintenance, by using a specific algorithm to identify newly acquired defective pixels. In the embodiments of the present application, whether the defective pixel data obtained in the factory stage calibration or the defective pixel data obtained in the use stage, can be written into the memory for storage, so that the defective pixels can be repaired when the image sensor reads out data. The above-mentioned method for obtaining defective pixels can be realized by using the operation of the prior art, of course, other ways suitable for image sensors can also be used to obtain defective pixels affecting the imaging quality for later repair.

[0057] In one embodiment, step S102 comprises:

[0058] When the defective pixel forms a bad row, K rows of pixel data centered on the bad row are input as input data to the defective pixel repair model trained in advance;

[0059] When the defective pixel forms a bad column, K columns of pixel data centered on the bad column are input as input data to the defective pixel repair model trained in advance;

[0060] When the defective pixel forms a bad point, KxK matrix pixel data centered on the bad point are input as input data to the defective pixel repair model trained in advance.

[0061] In applications, defective pixels may exist in any of the following forms: bad spots, bad rows, or bad columns. To achieve accurate prediction in the defective pixel repair model, when defective pixels form bad rows, the input data covers K rows of pixel data centered on the bad row; when defective pixels form bad columns, the input data covers K columns of pixel data centered on the bad column; and when defective pixels form bad spots, the input data covers a K×K matrix of pixel data centered on the bad spot. All of these methods can achieve good repair results. Here, bad rows or bad columns are not limited to the case where the entire row or column consists entirely of defective pixels. They may also include normal pixels and some continuous bad spots within an entire row or column of pixels, or bad spots that, while not continuous, are distributed in rows or columns, forming localized bad rows or bad columns. For the case of localized bad rows or bad columns, the data input method used in this step when defective pixels form bad rows or bad columns can be adopted.

[0062] In one embodiment, such as Figure 3 As shown, step S103 includes the following steps S301 to S303:

[0063] Step S301: Use C convolutional kernels of size K×K to extract features from the input data to obtain feature maps of C channels, where C is an integer greater than 1.

[0064] In applications, the size of a convolution kernel is typically represented by a two-dimensional pair of integers. In this embodiment, the kernel size is K×K, meaning the kernel size is K rows and K columns. Figure 4 As shown, when the image sensor uses a Bayer array, the minimum size of the convolutional kernel is 5 rows and 5 columns. In one implementation, taking a bad row as an example, when the convolutional kernel slides along the row direction (i.e., the horizontal direction), it will cover the bad row and the valid data of the two rows above and below it. In other cases, it will cover the bad column (or bad pixel) and the valid data of the two columns to its left and right. Specifically, taking a bad row as an example, after processing the input data using one type of convolutional kernel along the horizontal direction, one row of calculation results will be obtained. That is, the feature values ​​extracted from each point in the bad row using this convolutional kernel, i.e., the dot product operation between the convolutional kernel and the valid data, will calculate the pixel estimate value at the corresponding position in the bad row.

[0065] In some applications, since the data of bad rows is not involved in the calculation, the part of the convolution kernel on bad rows can be set to zero, and estimation can be performed using only the effective data from the upstream and downstream rows.

[0066] Specifically, feature extraction is performed using C convolutional kernels of size K×K to obtain feature maps of C channels. When C is selected as multiple, more different convolutional kernels are used to extract different feature values, resulting in C rows (arranged as channels). Based on the above method, each convolutional kernel extracts different features of the input data, thereby achieving diversified feature extraction.

[0067] Step S302, performing nonlinear transformation processing on the feature map.

[0068] In application, the output of the convolutional layer can be nonlinearly transformed using a ReLU function, which sets all negative values to 0 and retains positive values, helping the model to learn more complex image features. In nonlinear transformation, any one of the Sigmoid function, tanh (hyperbolic tangent) function, Leaky ReLU function, Parametric ReLU (PReLU) function, Exponential Linear Units (ELU) function, Swish function, etc. can be used, and the specific choice of the activation function mainly depends on the architecture of the network, task requirements, and characteristics of the data.

[0069] Step S303, fusing the feature map after the nonlinear transformation processing to obtain the repaired pixel value.

[0070] In application, feature fusion can be performed using a 1x1 convolution kernel (such as a 1x1 convolution in the C channel) to obtain 1 row of output. It can compress the feature map of multiple channels to a smaller number of channels while retaining or enhancing important feature information, thereby helping the network better understand and predict the local and global structure of the image. After using the 1x1 convolution kernel in the C channel to perform feature fusion, the output value of the defect pixel repair model, i.e., the repaired pixel value, is obtained. As shown in Figure 5 , the neural network realizes K rows of input and 1 row of output through this step.

[0071] In one embodiment, before step S102, it further includes:

[0072] When the defect pixel is an edge pixel, edge padding is performed on the input data.

[0073] In application, when the convolution kernel slides on the image boundary, part of it will exceed the boundary of the image, resulting in insufficient pixel information for direct calculation. Therefore, when the defect pixel is an edge pixel, edge padding is performed on the input data to ensure that the convolution kernel has enough data for calculation at the image boundary, avoiding information loss and edge effects. The method of edge padding can use mirror padding, i.e., expanding the image boundary by mirroring the pixel values near the edge along the edge, as shown in Figure 4 , the pixel values of the image edge are symmetrically copied along the edge. Zero padding, cyclic padding, etc. can also be used for edge padding.

[0074] In one embodiment, as shown in Figure 6 , before step S102, it further includes steps S201 to S204 as follows:

[0075] Step S201, obtaining a plurality of original data.

[0076] In application, a normal CMOS image sensor can be used to collect sufficient Raw image data in various scenarios. The collected data can be obtained under different lighting conditions, different scenes (indoor, outdoor, high contrast, low contrast, etc.), and different exposure times to ensure that the model can handle various real-world situations.

[0077] Step S202, randomly selecting K rows and / or K columns of pixel data from the original data.

[0078] Step S203, selecting an intermediate row or column from the K rows and / or K columns of pixel data, setting the pixel value to 0, and storing the original pixel value of the intermediate row or column as the true value.

[0079] In application, this step simulates the bad row or bad column that may occur in the sensor, providing a basis for the model to learn how to repair these defects. The stored original pixel value as the true value will be used to calculate the loss function in model training to measure the accuracy of the model prediction.

[0080] Step S204, inputting the K rows and / or K columns of pixel data into the defective pixel repair network for model training until the model converges or reaches the preset training round.

[0081] In application, backpropagation algorithm and gradient descent method can be used to optimize the weight, and learning rate scheduling, regularization, batch normalization and other techniques can be used to speed up convergence and prevent overfitting. Among them, model convergence can be judged by monitoring whether the loss function on the validation set is stable or starts to rise, or the maximum training round or threshold can be set to terminate the training. That is, using backpropagation algorithm, the gradient of the error with respect to the convolution kernel is calculated, that is, the gradient descent method is used to update the parameters, so that the difference between the network output and the true value is smaller. After repeated iterations, the network parameters can converge to a minimum point, and the network model obtained from this can predict similar row pixels to the true value based on the bad row input.

[0082] In one embodiment, as shown in Figure 7 Step S204 includes steps S401 to S407 as follows:

[0083] Step S401, inputting the K rows and / or K columns of pixel data into the defective pixel repair network for forward propagation.

[0084] In an application, when used for repairing a bad row, K rows of pixel data including the bad row are input, a total of 2K+1 rows of data. Similarly, for a bad column, K columns of data including the bad column are input, a total of 2K+1 columns of data. Forward propagation means that after the defective pixel repair model receives the input data, it is calculated in turn through each layer in the defective pixel repair model. Each layer of the defective pixel repair model will transform or filter the data, so as to extract the features of the input data, and finally generate a prediction value for estimating or reconstructing the correct value of the defective pixel. For example, for K row data input for bad row repair application, a KxK size convolution kernel is designed to be used directly, with the bad row as the center, and the calculation is performed along the horizontal direction.

[0085] Step S402, feature extraction is performed on the K rows and / or K columns of pixel data using a KxK size convolution kernel.

[0086] In an application, for example, for K row data input for bad row repair application, a KxK size convolution kernel is used to extract features from the input data, and through the convolution operation sliding along the row direction, information about shape, texture and pattern, etc. (i.e. feature extraction) can be extracted from the surrounding pixels.

[0087] Step S403, nonlinear transformation processing is performed on the extracted features.

[0088] Step S404, the features after the nonlinear transformation processing are fused to obtain the prediction value of the defective pixel repair model.

[0089] In an actual application, K row input data can be sent to a neural network, and after the above-mentioned 2 convolution operations, 1 row of output results is obtained.

[0090] Step S405, the loss between the prediction value and the true value is calculated.

[0091] In an application, a loss function such as mean square error (MSE), mean absolute error (MAE), etc. can be used to calculate the difference between the prediction value and the true value. For example, since the parameters of the convolution kernel are randomly initialized, the output result is also meaningless at the beginning of training, and based on a certain standard that can be set according to actual needs, such as mean absolute error, the difference between the network output and the true row of pixels can be calculated.

[0092] Step S406, the parameters of the defective pixel repair model are updated to minimize the loss, and the step S202 is returned.

[0093] Step S407, the model converges or reaches a preset training round.

[0094] Each raw data can be Raw data, which can be collected by an image sensor under different scenes.

[0095] In applications, the performance of the defective pixel repair model can also be evaluated on an independent verification data set to ensure the generalization ability of the defective pixel repair model. According to the verification result, the learning rate, batch size, regularization coefficient and other hyperparameters are adjusted to optimize the performance of the defective pixel repair model. After the defective pixel repair model is trained, the defective pixel repair model can be integrated into the image sensor to realize the automatic bad row repair function.

[0096] In one embodiment, step S201 further includes:

[0097] Randomly selecting K rows of pixel data from the raw data and performing random flipping processing.

[0098] In applications, for a Bayer array, K rows of pixel data are randomly selected from the raw data, and random transformation such as flipping is implemented to increase the diversity of the data set and improve the generalization ability of the model.

[0099] Numerical normalization processing is performed on the raw data.

[0100] In applications, normalization processing refers to standardizing the numerical range of all data to a uniform scale, such as between 0 and 1, according to sensor bit width information, which facilitates model processing.

[0101] In one embodiment, the value range of K is 5-13, and the value range of C is 2 4 ~ 2 8 .

[0102] In applications, the larger K is, the larger the row or column data cache required, and the larger the calculation amount will also be; the larger C is, the larger the calculation amount will also be. K directly determines the number of pixels of the same color channel that can be referenced by the bad pixel. When K = 5, it is difficult to predict the horizontal texture by limited reference points; when K = 9, the repair effect will be significantly improved. C represents the number of convolution kernels, i.e., the number of extracted features. Under the condition of calculation amount, increasing the number of convolution kernels can better extract features and achieve better prediction results.

[0103] In one embodiment, K rows and / or K columns of pixel data centered on the defective pixel are input as input data to the step of the pre-trained defective pixel repair model, the defective pixel repair model is implemented based on a neural network processing unit, and the neural network processing unit is integrated in the image sensor; or the defective pixel repair model is implemented based on model parameter solidification hardware, and the model parameter solidification hardware is integrated in the image sensor.

[0104] Specifically, the method provided in the present application can process multi-row and / or multi-column image data based on a convolutional neural network, so that in actual deployment, the neural network processing unit (NPU) can be used for efficient calculation and inference, and the NPU is used to process the neural network model. Since the parameter quantity and the calculation quantity of the network are small, the computing power requirement of the NPU is also low, so that a super-light NPU can be used. By integrating the NPU into a CMOS image sensor of an advanced process, the data reading and writing efficiency can be further optimized, and the power consumption can be reduced. In another implementation manner, after the model training is completed, the parameters of the convolution kernel are fixed. If no other AI task needs to be processed, and the repair effect is also determined, the NPU can not be used, but the model parameters can be directly solidified into a hardware IP, and a similar method of traditional image processing is used for processing. In the implementation manner, the solidified hardware can be arranged in the peripheral circuit region of the image sensor.

[0105] The present application also provides a defect pixel repair device for executing the steps in the defect pixel repair method embodiments. The defect pixel repair device can be a virtual appliance in an image sensor, run by a processor in the image sensor, or can be the image sensor itself.

[0106] As shown in FIG. 1, the defect pixel repair device 100 provided in the present application includes: Figure 8

[0107] The acquisition module 101 is configured to acquire defect pixel data of an image sensor.

[0108] The input module 102 is configured to input K rows and / or K columns of pixel data centered on the defect pixel as input data to a defect pixel repair model pre-trained.

[0109] The prediction module 103 is configured to predict the input data by using the defect pixel repair model to obtain a repair pixel value.

[0110] The repair module 104 is configured to replace the defect pixel with the repair pixel value to repair the defect pixel.

[0111] In the present application, the size of the convolution kernel of the defect pixel repair model is KxK, K=2N*T+1, N is a positive integer, and T is the arrangement period of the pixel unit in the image sensor.

[0112] In one embodiment, the acquisition module 101 is further configured to:

[0113] ​Acquire at least one of the defective pixel data calibrated during the manufacturing stage and the defective pixel data identified during the usage stage.

[0114] In one embodiment, the input module 102 is further configured to:

[0115] When the defective pixel forms a bad row, the K rows of pixel data centered on the bad row are used as input data and input into the pre-trained defective pixel repair model.

[0116] When the defective pixel forms a bad column, the K columns of pixel data centered on the bad column are used as input data and input into the pre-trained defective pixel repair model.

[0117] When the defective pixel forms a bad spot, the K×K matrix pixel data centered on the bad spot is used as input data and input into the pre-trained defective pixel repair model.

[0118] In one embodiment, the prediction module 103 is further configured to:

[0119] The input data is processed using C convolutional kernels of size K×K to extract features, resulting in feature maps of C channels.

[0120] The feature map is subjected to nonlinear transformation processing;

[0121] The feature maps after the nonlinear transformation are fused to obtain the repaired pixel values;

[0122] Where C is a positive integer greater than 1.

[0123] In one embodiment, a preprocessing module is further included for:

[0124] When the defective pixel is an edge pixel, edge filling is performed on the defective pixel.

[0125] In one embodiment, a training module is also included, for:

[0126] Acquire raw data, which is collected by a normal image sensor under different scenarios;

[0127] Randomly select K rows and / or K columns of pixel data from the original data;

[0128] Select a middle row or column from the K rows and / or K columns of pixel data, set the pixel value to 0, and store the original pixel value of the middle row or column as the true value.

[0129] The K rows and / or K columns of pixel data are input into the defect pixel repair model for model training until the model converges or reaches the preset training rounds.

[0130] In an embodiment, the training module is further configured to:

[0131] inputting the K rows and / or K columns of pixel data into the defective pixel repair model for forward propagation;

[0132] performing feature extraction on the K rows and / or K columns of pixel data using a KxK size convolution kernel;

[0133] performing nonlinear transformation processing on the extracted features;

[0134] fusing the features after the nonlinear transformation processing to obtain a predicted value of the defective pixel repair model;

[0135] calculating a loss between the predicted value and the true value;

[0136] updating parameters of the defective pixel repair model to minimize the loss;

[0137] returning to the step of randomly selecting K rows and / or K columns of pixel data from the original data until the model converges or a preset number of training rounds is reached.

[0138] In an embodiment, the training module is further configured to:

[0139] randomly selecting K rows of pixel data from the original data and performing random flipping processing;

[0140] performing numerical normalization processing on the original data.

[0141] In applications, the modules in the defective pixel repair device can be software program modules, can be implemented through different logic circuits integrated in a processor, or can be implemented through multiple distributed processors.

[0142] In an embodiment, since the defective pixel repair model has a small number of parameters and a small amount of calculation, the requirement for the computing power of a neural processing unit (NPU) is low, and therefore a super-lightweight neural processing unit can be used. By integrating the neural processing unit into an image sensor of an advanced process, the data read-write efficiency can be further optimized, and the power consumption can be reduced. In addition, the defective pixel repair model can be directly integrated into the existing hardware of the image sensor, and a separate neural processing unit is no longer needed.

[0143] Specifically, the method provided in the present application can process multi-row and / or multi-column image data based on a convolutional neural network, so that in actual deployment, the neural network processing unit (NPU) can be used for efficient calculation and inference, and the NPU is used to process the neural network model. Since the parameter quantity and the calculation quantity of the network are small, the computing power requirement of the NPU is also low, so a super-light NPU can be used. By integrating the NPU into a CMOS image sensor of an advanced process, the data read-write efficiency can be further optimized, and the power consumption can be reduced. In another implementation manner, after the model training is completed, the parameters of the convolution kernel are fixed. If no other AI task needs to be processed, and the repair effect is also determined, the NPU can not be used, but the model parameters can be directly solidified into a hardware IP, and a similar method of traditional image processing is used for processing. In the implementation manner, the solidified hardware can be arranged in the peripheral circuit region of the image sensor.

[0144] As Figure 9 shown, the present application also provides an image sensor 200, which comprises at least one processor 201 (only one processor is shown in the processor), a memory 202, and a computer program 203 stored in the memory 202 and executable on the at least one processor 201, and the processor 201 implements the steps in each of the above method embodiments when executing the computer program 203. Figure 9

[0145] In application, the image sensor can include, but is not limited to, a processor, a memory. Those skilled in the art can understand that Figure 9 is only an example of the image sensor, and does not constitute a limitation on the image sensor, and can include more or fewer components than those shown, or combine certain components, or different components, for example, can also include an input / output device, a network access device, etc.

[0146] In application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0147] ​In the application, the memory can be an internal storage unit of the image sensor, such as a hard disk or a memory of the image sensor, in some embodiments. The memory can also be an external storage device of the image sensor, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the image sensor, in other embodiments. Further, the memory can include both the internal storage unit and the external storage device of the image sensor. The memory is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of computer programs, etc. The memory can also be used to temporarily store data that has been output or will be output.

[0148] It should be noted that the information interaction and execution process between the above devices / units are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought by them can be referred to the method embodiments part, which will not be repeated here.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0150] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in each of the above method embodiments can be implemented.

[0151] The embodiment of the present application provides a computer program product, which, when running on an image sensor, enables the image sensor to execute the steps in each of the above method embodiments.

[0152] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium can at least include any entity or device capable of carrying the computer program code to the device / image sensor, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0153] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0154] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0155] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are only schematic, and the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0156] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0157] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for repairing defective pixels, characterized in that, include: Acquire defective pixel data from the image sensor; The K rows and / or K columns of pixel data centered on the defective pixel are used as input data and fed into the pre-trained defective pixel repair model. The defect pixel repair model is used to predict the input data to obtain the repair pixel value; The defective pixel is replaced using the repair pixel value to repair the defective pixel; The convolution kernel size of the defect pixel repair model is K×K, where K = 2N*T+1, N is a positive integer, and T is the arrangement period of pixel units in the image sensor.

2. The defect pixel repair method as described in claim 1, characterized in that, The acquisition of defective pixel data from the image sensor includes: Acquire at least one of the defective pixel data calibrated during the manufacturing stage and the defective pixel data identified during the usage stage.

3. The defect pixel repair method as described in claim 1, characterized in that, The step of inputting K rows and / or K columns of pixel data centered on the defective pixel into a pre-trained defective pixel repair model includes: When the defective pixel forms a bad row, the K rows of pixel data centered on the bad row are used as input data and input into the pre-trained defective pixel repair model. When the defective pixel forms a bad column, the K columns of pixel data centered on the bad column are used as input data and input into the pre-trained defective pixel repair model. When the defective pixel forms a bad spot, the K×K matrix pixel data centered on the bad spot is used as input data and input into the pre-trained defective pixel repair model.

4. The defect pixel repair method as described in claim 1, characterized in that, The step of using the defective pixel repair model to predict the input data and obtain the repaired pixel value includes: The input data is processed using C convolutional kernels of size K×K to extract features, resulting in feature maps of C channels. The feature map is subjected to nonlinear transformation processing; The feature maps after the nonlinear transformation are fused to obtain the repaired pixel values; Where C is an integer greater than 1.

5. The defect pixel repair method as described in claim 4, characterized in that, The feature maps after the nonlinear transformation are fused to obtain the repaired pixel values, including: The C feature maps after the nonlinear transformation are fused using a 1x1 convolution kernel with C channels to obtain the repaired pixel value output as a row, column, or point. And / or, when using C convolutional kernels of size K×K to extract features from the input data, the value of K ranges from 5 to 13, and the value of C ranges from 2. 4 ~2 8 .

6. The defect pixel repair method as described in claim 1, characterized in that, Before inputting the K rows and / or K columns of pixel data centered on the defective pixel into the pre-trained defective pixel repair model, the process also includes: When the defective pixel is an edge pixel, edge filling is performed on the input data.

7. The defect pixel repair method as described in claim 6, characterized in that, The edge filling method is mirror filling.

8. The defect pixel repair method as described in claim 1, characterized in that, In the step of inputting K rows and / or K columns of pixel data centered on the defective pixel as input data into a pre-trained defective pixel repair model, the defective pixel repair model is implemented based on a neural network processing unit, and the neural network processing unit is integrated into the image sensor; or, the defective pixel repair model is implemented based on hardware with fixed model parameters, and the hardware with fixed model parameters is integrated into the image sensor.

9. The defect pixel repair method according to any one of claims 1 to 8, characterized in that, Before inputting the K rows and / or K columns of pixel data centered on the defective pixel into the pre-trained defective pixel repair model, the process also includes: Obtain the raw data; Randomly select K rows and / or K columns of pixel data from the original data; Select a middle row or column from the K rows and / or K columns of pixel data, set the pixel value to 0, and store the original pixel value of the middle row or column as the true value. The K rows and / or K columns of pixel data are input into the defect pixel repair network for model training until the model converges or reaches the preset training rounds, thus obtaining the defect pixel repair model.

10. The defective pixel repair method as described in claim 9, characterized in that, The step of inputting the K rows and / or K columns of pixel data into the defect pixel repair network for model training until the model converges or reaches a preset number of training rounds includes: The K rows and / or K columns of pixel data are input into the defect pixel repair network for forward propagation; Feature extraction is performed on the K rows and / or K columns of pixel data using a K×K convolution kernel; The extracted features are subjected to nonlinear transformation processing; The features after the nonlinear transformation are fused to obtain the predicted value of the defect pixel repair model; Calculate the loss between the predicted value and the true value; The parameters of the defect pixel repair model are updated with the goal of minimizing the loss. Return to the step of randomly selecting K rows and / or K columns of pixel data from the original data until the model converges or reaches the preset training rounds; The raw data includes imaging data from image sensors in different scenarios.

11. The defective pixel repair method as described in claim 10, characterized in that, After obtaining the raw data, the process also includes: K rows of pixel data are randomly selected from the original data and then randomly flipped. The original data is then subjected to numerical normalization.

12. A defective pixel repair device, characterized in that, include: The acquisition module is used to acquire defective pixel data from the image sensor; The input module is used to input the K rows and / or K columns of pixel data centered on the defective pixel as input data into the pre-trained defective pixel repair model; The prediction module is used to predict the input data using the defect pixel repair model to obtain the repair pixel value; The repair module is used to replace the defective pixel using the repair pixel value to repair the defective pixel. The convolution kernel size of the defect pixel repair model is K×K, where K = 2N*T+1, N is a positive integer, and T is the arrangement period of pixel units in the image sensor.

13. An image sensor, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the defective pixel repair method as described in any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the defective pixel repair method as described in any one of claims 1 to 11.

Images