An image sensor defect correction method, system, and medium
Patent Information
- Application Number
- CN202611517310.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-29
- Publication Date
- 2026-10-30
AI Technical Summary
其利用缺陷像素周围正常像素的灰度值，通过替换、线性插值、中值滤波等方式估算缺陷像素的理想值，而图像传感器上固定位置缺陷，无法有效处理团簇型缺陷，当缺陷区域覆盖多个相邻像素时，缺陷像素的邻域像素本身亦为异常值，基于错误邻域的插值结果同样失真，矫正效果急剧劣化
本发明通过对缺陷识别、矫正系数计算、行稀疏编码及实时矫正有机整合，形成完整的技术闭环，能够有效地处理团簇缺陷、存储资源消耗及过渡区域不平滑等问题，实现图像传感器固定位置缺陷矫正的实时性、精准性和高效性，大大降低成本，具有显著的应用价值。
Smart Images
Figure CN123015290A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of machine vision, and in particular relates to a method, system and medium for correcting defects in image sensors. Background Technology
[0002] As the core photosensitive element of a camera, the image sensor inevitably introduces various defects during its manufacturing and packaging process. Common defect types include: particle contamination on the sensor surface, clusters of dead pixels caused by poor wafer processing, scratches and dust contamination on the filter or sealing glass, and pores. These defects typically manifest as large-area pixel response anomalies at fixed locations on the image sensor, causing the defective area to cover multiple consecutive pixels and continuously appear in the same spatial location, severely affecting image quality.
[0003] For pixel correction of fixed-location defects, existing technologies mainly fall into two categories: The first category is correction methods based on neighborhood replacement / interpolation. These methods utilize the grayscale values of normal pixels surrounding the defective pixel, estimating the ideal value of the defective pixel through replacement, linear interpolation, median filtering, etc. However, fixed-location defects on image sensors cannot effectively handle clustered defects. When the defective region covers multiple adjacent pixels, the neighboring pixels of the defective pixel themselves are also outliers, resulting in distorted interpolation results based on incorrect neighborhoods and a sharp deterioration in correction effectiveness. In practical applications, filter pores, dust contamination, wafer process defects, etc., are often not isolated points but rather exhibit blocky or strip-like distributions, severely limiting the applicability of this type of method. The second category is methods based on global point-by-point correction. This type of method pre-calibrates the response difference of each pixel in the sensor to generate a full-frame correction coefficient matrix. When outputting the image, it performs point-by-point multiplication and division operations on all pixels. Otherwise, it will lead to obvious transition boundaries between defective and non-defective areas. However, point-by-point correction requires a large amount of resources. In order to ensure that the responsivity of each pixel is adjusted, for large-format sensors, the responsivity correction of all pixels needs to be stored, which consumes a lot of memory and costs.
[0004] Therefore, existing technologies lack an image sensor defect correction scheme that can effectively correct various fixed-location defects (especially clustered defects), control storage and computing resource consumption, and ensure a natural transition in the correction area. Thus, this paper provides an image sensor defect correction method, system, and medium. Summary of the Invention
[0005] The purpose of this invention is to overcome the above-mentioned problems in the prior art and to provide a method, system and medium for image sensor defect correction.
[0006] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution: A method for correcting defects in an image sensor, which involves averaging multiple frames of images acquired by the image sensor to obtain a pixel response distribution map, is characterized by comprising: Identify the defective regions in the pixel response distribution map, with each defective region constituting a point set; Based on the position coordinates of pixels in each defect region, calculate the minimum envelope rectangle corresponding to each defect region, and analyze the pixel mean of the neighborhood region of the minimum envelope rectangle to determine the expected gray value of each pixel in the point set. The correction coefficient for each pixel is determined by the ratio between the expected gray value of each pixel and the original gray value of each pixel in the pixel response distribution map. Row sparse coding is performed on the correction coefficients of each point set to generate a correction coefficient tuple corresponding to the scanning line sequence of the image sensor. The correction coefficient tuple consists of the row start coordinates of the point set that overlaps with the scanning line of the image sensor, the width of the point set in the scanning line, and the correction coefficients of the point set in the scanning line. Determine whether there is an overlapping region among the correction coefficient tuples of each point set on the same scan line. If there is an overlapping region, extract the correction coefficients of the same pixel in the two point sets within the overlapping region and calculate the average to update the correction coefficients of each pixel in the overlapping region. Use the correction coefficients corresponding to each pixel in the defect region to correct the pixel response distribution map.
[0007] Furthermore, the gray values corresponding to the non-defective pixels in the relative positions of the neighborhood regions corresponding to each pixel in the minimum envelope rectangle are extracted, and the average value is calculated to obtain the average gray value of each pixel in the minimum envelope rectangle. Based on the mapping relationship between each pixel in each point set and each pixel in the minimum envelope rectangle, the expected gray value of each pixel in the point set is determined.
[0008] Furthermore, the method for identifying defective regions in the pixel response distribution map includes: extracting the gray values of pixels whose gray values differ from those of neighboring pixels by less than a set gray value change threshold; correcting the gray values of each pixel in the pixel response distribution map using surface fitting; and performing thresholding processing on the corrected pixel response distribution map to determine the pixels with defects.
[0009] Furthermore, connected component analysis is performed on the pixel response distribution map after thresholding, and regions with an area greater than a set area threshold are selected as defect regions.
[0010] Furthermore, the method for determining the minimum envelope rectangle includes: extracting the maximum and minimum values of the abscissa and the maximum and minimum values of the ordinate of the same point set; The coordinates of the top left corner of the minimum envelope rectangle are determined based on the minimum values of the x and y coordinates, and the coordinates of the bottom right corner of the minimum envelope rectangle are determined based on the maximum values of the x and y coordinates.
[0011] Furthermore, a method for implementing row sparse coding for the correction coefficients of each point set includes: determining whether each row of the image sensor overlaps with the point set corresponding to all defect regions; if there is overlap, extracting the correction coefficients corresponding to that row of the image; if there is no overlap, not extracting the correction coefficients corresponding to that row of the image.
[0012] Furthermore, the method for determining whether each scan line of the image sensor overlaps with each point set includes: extracting the maximum and minimum values of the ordinate covered by the minimum envelope rectangle corresponding to each point set; if the index value of the scan line is not greater than the maximum value of the ordinate and not less than the minimum value of the ordinate, then the scan line overlaps with the point set; otherwise, the scan line does not overlap with the point set.
[0013] Furthermore, when the same row of images overlaps with the point set corresponding to at least one defect region, the correction coefficients are arranged in ascending order according to the horizontal coordinates corresponding to each correction coefficient.
[0014] Based on the same inventive concept, this invention provides an image sensor defect correction system that averages multiple frames of images acquired by an image sensor to obtain a pixel response distribution map, including: The defect identification module is used to identify defect regions in the pixel response distribution map, and each defect region constitutes a point set; The neighborhood analysis module calculates the minimum envelope rectangle corresponding to each defect region based on the position coordinates of pixels in each defect region, and analyzes the pixel mean of the neighborhood region of the minimum envelope rectangle to determine the expected gray value of each pixel in the point set. The encoding adjustment module is used to determine the correction coefficient of each pixel based on the ratio between the expected gray value of each pixel and the original gray value of each pixel in the pixel response distribution map. Row sparse coding is performed on the correction coefficients of each point set to generate a correction coefficient tuple corresponding to the scanning line sequence of the image sensor. The correction coefficient tuple consists of the row start coordinates of the point set that overlaps with the scanning line of the image sensor, the width of the point set in the scanning line, and the correction coefficients of the point set in the scanning line. The deduplication correction analysis module is used to determine whether there is an overlapping region in the correction coefficient tuples of each point set on the same scan line. If there is an overlapping region, the correction coefficients of the same pixel in the overlapping region in the two point sets are extracted and averaged to update the correction coefficients of each pixel in the overlapping region. The correction coefficients corresponding to each pixel in the defect region are used to correct the pixel response distribution map.
[0015] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the correction method as described in any of the preceding claims.
[0016] The beneficial effects of this invention are: This invention organically integrates defect identification, correction coefficient calculation, row sparse coding, and real-time correction to form a complete technical closed loop. It can effectively handle problems such as cluster defects, storage resource consumption, and uneven transition regions, achieving real-time, accurate, and efficient correction of fixed-position defects in image sensors, greatly reducing costs, and has significant application value.
[0017] This invention determines the minimum envelope rectangle corresponding to the defect region and uses the pixel mean of the neighborhood region of the minimum envelope rectangle to determine the expected gray value corresponding to each abnormal pixel in the minimum envelope rectangle. This effectively avoids the interference of the defective pixels in the neighborhood on the expected gray value, and can obtain accurate and reliable correction coefficients. It provides a correction basis for the original gray value output by the image sensor, realizes effective correction of clustered defects, and breaks through the limitations of defect correction.
[0018] This invention utilizes the sparsity of defect distribution to store only the correction coefficient tuples corresponding to the scan lines that overlap with the defect area. It filters the point sets that overlap with the same row of images and arranges them in ascending order of their horizontal coordinates. This reduces the storage requirement from the full-size matrix to only the effective data corresponding to the defect area, significantly reducing memory usage and hardware costs. This allows the correction coefficients corresponding to the defect area to be deployed efficiently even in resource-constrained systems. By significantly compressing storage overhead through row sparse coding, it balances the correction effect with resource consumption.
[0019] This invention eliminates coefficient conflicts in the column direction between adjacent defective regions by determining whether there are overlapping regions in the correction coefficient tuples on the same scan line and merging the average value of the correction coefficients in the overlapping regions. This avoids response cliffs between the corrected and uncorrected regions, ensures that the same pixel coordinate corresponds to a unique correction coefficient, and makes the corrected image transition naturally and visually continuous at the edge of the defective region. Thus, it ensures that after deduplication and fusion, the defective region can smoothly transition with the non-defective region. Attached Figure Description
[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of the defect correction method in this invention; Figure 2 This is a schematic diagram of the minimum envelope region in this invention; Figure 3 This is a structural block diagram of the defect correction system in this invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] When image sensors have large-area defects in fixed locations, such as surface contamination, poor manufacturing process, clustered dead pixels, filter / sealant contamination, or pores, it is necessary to correct these large-area defects in fixed locations. While ensuring the correction effect, memory and cost consumption can be controlled.
[0023] like Figure 1 As shown, this embodiment first provides an image sensor defect correction method, which averages multiple frames of images acquired by the image sensor to obtain a pixel response distribution map, including: Step 1: Identify the defective regions in the pixel response distribution map, with each defective region constituting a point set.
[0024] The image sensor to be tested is placed under a uniform light source, and the exposure time is adjusted to ensure that the image sensor can accumulate sufficient photocharge so that the output signal amplitude meets the requirements, thereby ensuring that the image acquired by the image sensor has an effective signal-to-noise ratio and dynamic range.
[0025] Under the current light source and exposure time, multiple frames of images are acquired, and the average of the acquired frames is calculated to obtain the gray value of each pixel. Based on the obtained gray values of each pixel, a pixel response distribution map is constructed. By averaging multiple frames of images, random noise can be suppressed.
[0026] Thresholding is applied to the pixel response distribution map to distinguish between abnormal and normal pixels. Adjacent abnormal pixels are marked, and all adjacent abnormal pixels constitute a defect region. Each defect region contains a set of abnormal pixels. The 8-neighborhood criterion is used to classify adjacent abnormal pixels; if two abnormal pixels are adjacent in the horizontal, vertical, or diagonal direction, they belong to the same defect region.
[0027] Step 2: Based on the position coordinates of pixels in each defect region, determine the minimum envelope rectangle corresponding to each defect region, and analyze the pixel mean of the neighborhood region of the minimum envelope rectangle to determine the expected gray value of each pixel in the point set.
[0028] By obtaining the position coordinates of each pixel in the point set corresponding to each defect region, and sequentially filtering the range of the horizontal and vertical coordinates of the pixel position coordinates in the defect region, the minimum envelope rectangle can be determined. The minimum envelope rectangle corresponding to each defect region completely covers the defect region.
[0029] Based on the minimum envelope rectangle corresponding to each defect region, the neighborhood region of the minimum envelope rectangle is extracted. The gray values in the neighborhood region of the minimum envelope rectangle are averaged to obtain the average gray value corresponding to the same relative position in the neighborhood region. This average gray value is then used as the gray value corresponding to the relative position in the minimum envelope rectangle, thereby determining the expected gray value of each pixel in the point set.
[0030] By using the pixel mean of the minimum envelope rectangle and its neighboring regions, the expected gray value of the defective pixel in the minimum envelope rectangle can be determined. This can effectively avoid the interference of the defective pixel itself on the correction coefficient, making the calculation result of the expected gray value accurate and robust, thus providing a reliable reference for subsequent correction.
[0031] Step 3: Determine the correction coefficient for each pixel based on the ratio between the expected grayscale value of each pixel and the original grayscale value of each pixel in the pixel response distribution map.
[0032] Extract the expected grayscale value of each pixel in the point set corresponding to each defect region, and use the ratio between the expected grayscale value of the pixel and the grayscale value of each pixel in the pixel response distribution map as the correction coefficient of the pixel in the point set. The correction coefficient of each pixel may be greater than 1 or less than 1.
[0033] Step 4: Perform row sparse coding on the correction coefficients of each point set to generate a correction coefficient tuple corresponding to the scanning line sequence of the image sensor. The correction coefficient tuple consists of the row start coordinates of the point set that overlaps with the scanning line of the image sensor, the width of the point set in the scanning line, and the correction coefficients of the point set in the scanning line.
[0034] The image sensor scan rows are traversed sequentially from smallest to largest. For the k-th image row, the ordinate range of the point sets corresponding to all defect regions is compared to determine whether the k-th image row overlaps with the point sets corresponding to each defect region.
[0035] Filter the point sets that overlap with the k-th image row, sort them in ascending order according to the starting position of the horizontal coordinate of each point set on the image row, and record the starting position of the horizontal coordinate of each point set with the k-th image row, the number of pixels that overlap with the k-th image row, and the correction coefficient of each overlapping pixel, so as to obtain the correction coefficient tuple between each point set and each scan row of the image sensor.
[0036] When the i-th point set overlaps with the k-th image row, the correction coefficient tuple formed by the i-th point set and the k-th image row is: ,in, W represents the starting position of the x-coordinate where the i-th point set overlaps with the k-th image row. i This represents the number of pixels that overlap between the i-th point set and the k-th image row. Let be the correction coefficients corresponding to the pixels that overlap between the i-th point set and the k-th image row. This represents the endpoint of the horizontal coordinate where the i-th point set overlaps with the k-th image row.
[0037] By performing row sparse coding on the scan lines, correction coefficient tuples are generated. Based on each correction coefficient tuple, it is possible to quickly identify whether there are overlapping areas among the correction coefficient tuples on the same scan line, and achieve sequential arrangement. This greatly compresses the amount of data stored, significantly improves the storage efficiency and addressing speed in hardware implementation, and has real-time performance.
[0038] Step 5: Determine whether there is an overlapping region in the correction coefficient tuples of each point set on the same scan line. If there is an overlapping region, extract the correction coefficients of the same pixel in the two point sets within the overlapping region and calculate the average to update the correction coefficients of each pixel in the overlapping region. Use the correction coefficients corresponding to each pixel in the defect region to correct the pixel response distribution map.
[0039] When there are correction coefficient tuples of at least one point set on the same scan line, it is necessary to analyze whether there is an overlapping region between any two point sets in order to filter out the overlapping region between any two point sets on the same scan line. The correction coefficients of each pixel in the overlapping region in the two point sets are extracted respectively. The correction coefficients of any pixel in the overlapping region in the two point sets are used and averaged to obtain the correction coefficient corresponding to the pixel in the overlapping region of the scan line, so as to solve the problem that the same pixel on the scan line has different correction coefficients in different two point sets.
[0040] By averaging the correction coefficients in the overlapping areas of the same scan line, the problem of inconsistency caused by assigning different correction coefficients to the same pixel in different defect areas is eliminated. This ensures the spatial continuity and visual consistency of the corrected image, effectively avoids blocky distortion or splicing marks in the corrected image, and ultimately achieves a high-quality image correction effect.
[0041] In this embodiment, the method for identifying defective regions in the pixel response distribution map includes: extracting the gray values of pixels whose gray values differ from those of neighboring pixels by less than a set gray value change threshold; correcting the gray values of each pixel in the pixel response distribution map using surface fitting; and performing thresholding processing on the corrected pixel response distribution map to determine the pixels with defects.
[0042] In this embodiment, before constructing the surface mathematical model of the image sensor, the gray values corresponding to pixels whose gray values are less than a set gray value change threshold are selected. Based on the gray values of the selected pixels, a surface fitting method is used to construct the surface mathematical model of the image sensor to fit the macroscopic non-uniformity of the image sensor response.
[0043] The setting of a grayscale value change threshold is used to filter out non-abnormal pixels whose grayscale values are close to those of neighboring pixels. The value of the grayscale value change threshold depends on the upper limit of the allowable difference between the grayscale values of the center pixel and its neighboring pixels.
[0044] A mathematical model for fitting the surface is constructed using a bivariate quadratic polynomial. The gray values corresponding to several non-abnormal pixels on the image sensor are used to solve the mathematical model of the fitted surface using the least squares method. The mathematical model of the surface of the image sensor is obtained by substituting the coordinates of each pixel position into the mathematical model of the surface of the image sensor to obtain the corrected gray values corresponding to each pixel.
[0045] Based on the curve quantity model of the image sensor, the gray values of each pixel are obtained and compared with the set upper and lower limits of gray values. If the gray value of a pixel is greater than the upper limit or less than the lower limit, the pixel is marked as an abnormal pixel. Conversely, if the gray value of a pixel is neither greater than the upper limit nor less than the lower limit, the pixel is marked as a normal pixel.
[0046] In this embodiment, abnormal pixels are marked. If an abnormal pixel is marked as 1, a normal pixel is marked as 0, and vice versa. By marking abnormal pixels and normal pixels differently, abnormal pixels and normal pixels can be distinguished in the pixel response distribution map.
[0047] Since the image sensor acquires multiple frames of images under uniform light source, when the object under test is a standard calibration object, the gray values of each pixel acquired by the image sensor should be completely consistent. However, this application adopts a surface fitting method, which can present the macroscopic response distribution of the image sensor caused by lens vignetting, pixel sensitivity differences, etc.
[0048] For the identification of the aforementioned defective regions, it is not limited to a single defect identification method that combines surface fitting and thresholding. Template matching defect identification can be used, which involves comparing the image acquired by the image sensor with a standard template image pixel by pixel or region by region to identify the defective region. Defect identification based on convolutional neural networks and machine learning methods can also be used.
[0049] In addition, before surface fitting, if there is a large area of uniformity deviation in the image sensor, a flat field correction method can be used to correct the large area of uniformity deviation in order to preserve the sparse defect area.
[0050] Connectivity analysis is performed on the thresholded image to identify each defective region. Specifically, the method used for connectivity analysis includes: determining whether there are abnormal pixels in the neighboring pixels of each abnormal pixel; if there are abnormal pixels, assigning the same label to the abnormal pixel and its neighboring abnormal pixels as the same connected region; otherwise, the abnormal pixel is not considered as the same connected region as its neighboring pixels.
[0051] Image sensors, under uniform lighting, often exhibit large-area and gradual grayscale variations due to lens shadows, chip manufacturing deviations, and other factors. After thresholding, these grayscale variation areas are marked as anomalous pixels. However, the individual pixels within these grayscale variation areas are not actually anomalous. Therefore, large areas are easily treated as defective regions, increasing the workload of correcting anomalous pixels in these areas and disrupting the overall brightness gradient of the image, introducing new brightness distortions. Similarly, for isolated anomalous pixels after thresholding, treating them as defects will lead to inaccurate calculations of the expected grayscale value due to the small minimum envelope rectangle corresponding to the isolated anomalous pixel, and will significantly increase the storage requirements of row sparse coding.
[0052] Based on this, in this embodiment, connected component analysis is performed on the pixel response distribution map after thresholding, and regions whose area meets the set area limit range are selected as defect regions.
[0053] For each connected region, the area of that region is calculated, represented by the total number of pixels contained within it, and compared with a set area limit. Based on the set area limit, an area limit range is determined, which includes an upper area limit and a lower area limit, wherein the upper area limit is greater than the lower area limit. The area limit range includes both the upper and lower area limits.
[0054] After connected component analysis, if the number of pixels in a connected region is within a set area limit, the connected region is determined to be a valid defect region, and the position coordinates and area of each pixel in the connected region are recorded. If the connected region is smaller than the set lower area limit, it is determined to be noise and is not considered a defect region. If the connected region is larger than the upper area limit, it is determined to be a macroscopic anomaly rather than a local defect, and connected regions larger than the upper area limit are excluded to prevent the image sensor from treating the overall non-uniform area as a defect region.
[0055] For each valid defect region, the corresponding connected region is represented as a point set D. i Di = {(x ij , y ij ) |i=1, 2, ..., m,j=1,2,...,n i}, where (x ij , y ij Let be the coordinates of each pixel in the i-th point set, and n be the coordinates of each pixel in the i-th i Let be the total number of pixels corresponding to the i-th point set. Therefore, the sequence of point sets for all real defect regions is D1-D. m .
[0056] By setting area limits, defect areas can be further identified and filtered to facilitate the identification and filtering of effective defect areas. The setting of area limits can be determined according to the application scenario of the image sensor and the type of defect.
[0057] This embodiment discloses a method for determining the minimum envelope rectangle corresponding to each defect region, including: extracting the maximum and minimum values of the abscissa and the maximum and minimum values of the ordinate of the same point set; The coordinates of the top left corner of the minimum envelope rectangle are determined based on the minimum values of the x and y coordinates, and the coordinates of the bottom right corner of the minimum envelope rectangle are determined based on the maximum values of the x and y coordinates.
[0058] Based on the coordinates of the upper left and upper right corners, determine the minimum envelope rectangle corresponding to each defect region.
[0059] Specifically, for each point set (x) ij , y ij )∈D i Calculate separately: x max =max(x ij ),x min =min(x ij ) y max =max(y ij ),y min =min(y ij ) Using (x) min ,y min ) and (x max ,y max The top left and bottom right corners of the minimum envelope rectangle are respectively used as the top left and bottom right corners of the minimum envelope rectangle. The minimum envelope rectangle is determined based on the top left and bottom right corners of the minimum envelope rectangle.
[0060] Based on the determined minimum envelope rectangle, calculate the width W and height H of the minimum envelope rectangle, where the width W = x. max -xmin H=y max -y min .
[0061] like Figure 2 As shown, by filtering the maximum and minimum values of the x and y coordinates of each point set, the boundary of the minimum envelope rectangle is determined, and then the width and height of the minimum envelope rectangle are determined. Based on the width and height of the minimum envelope rectangle, the neighborhood region of the minimum envelope rectangle needs to be filtered. Based on the pixel gray values of the neighborhood region, the expected gray value of each defective pixel in the minimum envelope rectangle is determined.
[0062] In this embodiment, the gray values corresponding to the relative positions of the four neighboring regions corresponding to each pixel in the minimum envelope rectangle are extracted, and the average value is calculated to obtain the average gray value of each pixel in the minimum envelope rectangle. Based on the mapping relationship between each pixel in each point set and each pixel in the minimum envelope rectangle, the expected gray value of each pixel in the point set is determined.
[0063] The above uses the average pixel value corresponding to the relative positions of four neighboring regions as the expected gray value of the corresponding pixel in the point set. However, it is not limited to using the four-neighboring criterion; the average pixel value corresponding to the relative positions of eight neighboring regions can also be used as the expected gray value of the corresponding pixel in the point set.
[0064] The minimum envelope rectangle is denoted as R. 0,0 The four neighborhoods of the minimum envelope rectangle, namely the left, right, top, and bottom neighborhoods, are denoted as R. 0,-1 R 0,1 R -1,0 and R 1,0 The coordinates of the top-left corner of the left, right, top, and bottom neighbors are respectively (x min -W,y min ), (x min +W,y min ), (x min ,y min -H) and (x min ,y min +H).
[0065] The average gray value of each pixel in the minimum envelope rectangle is calculated by averaging the gray values corresponding to the relative positions of the four neighboring regions.
[0066] In this process, it is determined whether the neighborhood regions of the minimum envelope rectangle overlap with each defect region. If one of the neighborhood regions overlaps with at least one defect region, the neighborhood region that overlaps with at least one defect region is removed. The gray values corresponding to the relative positions of the remaining neighborhood regions are averaged to obtain the expected gray values of each pixel in the defect region within the minimum envelope rectangle. The above analysis of the neighborhood regions of the minimum envelope rectangle to determine whether the neighborhood regions overlap with the defect regions is fully applicable to four-neighbor regions and eight-neighbor regions.
[0067] This embodiment is not limited to removing neighboring regions that overlap with defective regions when they overlap. It can also remove the gray values corresponding to the pixel positions that overlap with defective regions in the neighboring regions based on the overlap position between the neighboring regions and defective regions. Then, the average value of the removed pixels or neighboring regions is calculated to obtain the expected gray value corresponding to the defective pixels in the minimum envelope rectangle.
[0068] Since the pixel data of the image sensor is read out serially in a line-by-line scanning manner, and the pixel clock is continuous, the correction coefficients are required to be synchronized with the data stream. If the correction coefficients of each point set are stored independently according to the defect area, when a certain row of pixels is reached, the hardware needs to traverse all point sets to filter out the point sets that overlap with that row, which affects the data processing speed and destroys real-time performance. Based on this, this embodiment provides a method for implementing row sparse coding for the correction coefficients of each point set.
[0069] Specifically, the method of implementing row sparse coding for the correction coefficients of each point set includes: determining whether each row of the image sensor overlaps with the point set corresponding to all defect regions; if there is overlap, extracting the correction coefficients corresponding to that row of the image; if there is no overlap, not extracting the correction coefficients corresponding to that row of the image.
[0070] In this embodiment, the method for determining whether each scan line of the image sensor overlaps with each point set includes: extracting the maximum and minimum values of the ordinate covered by the minimum envelope rectangle corresponding to each point set; if the index value of the scan line is not greater than the maximum value of the ordinate and not less than the minimum value of the ordinate, then the scan line overlaps with the point set; otherwise, the scan line does not overlap with the point set.
[0071] For any row of an image from an image sensor, it is necessary to determine the values of all point sets D1-D1 respectively. m Does it overlap with the row? Label the minimum envelope rectangle and the correction coefficient matrix of the i-th point set, denoted as D respectively. i and C i Taking the k-th row as an example, it is necessary to determine whether the i-th point set overlaps with the k-th row. Extract point set D. iThe top left corner (x) of the corresponding minimum envelope rectangle i min ,y i min ) and the coordinates of the lower right corner (x i max ,y i max When y is satisfied i min ≤k<y i max When the k-th row image overlaps with the point set Di, then the k-th row image overlaps with the point set Di.
[0072] When the same row of images overlaps with the point set corresponding to at least one defect region, the correction coefficients are arranged in ascending order according to the horizontal coordinates corresponding to each correction coefficient.
[0073] Select the point sets that overlap with the k-th row, arrange the correction coefficient matrix of each point set in ascending order of horizontal coordinate, and simultaneously extract the correction coefficients that overlap with the k-th row to form the correction coefficient tuple of the k-th row.
[0074] The correction coefficient tuple in the k-th row is: {x i min W i C i (k,x i min ),...,C i (k,x i max )}, where x i min Representing the point set D i The x-coordinate of the starting position that overlaps with the k-th row, W i Representing the point set D i The width (number of pixels) of the overlap with the k-th row, C i (k,x i min ),...,C i (k,x i max ) represent point sets D respectively i The pixel coordinates (x) corresponding to the pixels overlapping with the k-th row, ordered from smallest to largest x-coordinate. i max Representing the point set D i The x-coordinate of the endpoint position that overlaps with the k-th row.
[0075] For each point set that overlaps with the k-th row, there is a correction coefficient tuple. The correction coefficient tuples in the same row are sorted in ascending order of their starting position x-coordinates.
[0076] By using row sparse coding, overlap determination is performed on all rows of the image sensor in advance, and the determination result is converted into an index table sorted by row. This allows the hardware to directly locate the correction coefficient corresponding to the row by simply by using the row number when reading the image of the k-th row, without traversing the entire point set. This ensures that the hardware has prepared all the coefficients that need to be corrected in the row in advance at the moment the pixel arrives, thus meeting the real-time requirements.
[0077] When an image in the same row overlaps with multiple point sets, it is necessary to determine whether there is overlap between the correction coefficient tuples corresponding to each point set in the image in that row. If there is overlap between two correction coefficient tuples, then the two overlapping correction coefficient tuples are merged.
[0078] Determine the set of points D in the same row of an image i The coordinates of the endpoint x i max With point set D i+1 The starting position coordinates x i+1 min When x i max >x i+1 min At that time, it is necessary to consider the point set D. i Sum of points D i+1 After merging, the merged correction coefficient tuple is: {x i min ,W' i C i (k,x i min ),...,C i+1 (k,x i+1 max )},W' i =x i+1 max -x i min .
[0079] For all rows of images from the image sensor, the above steps are performed sequentially according to the row image number in ascending order. This generates multiple correction coefficient tuples for each row of the image sensor. Each correction coefficient tuple contains the row index, the start and end range of the valid column segment, and the corresponding correction coefficient. During the actual output of image data by the image sensor, the correction coefficients of each pixel in the correction coefficient tuples corresponding to each row of images are used to correct the raw image data output by the image sensor in real time point by point.
[0080] The corrected grayscale value of a pixel is obtained by multiplying the correction coefficient of each pixel with the grayscale value of the corresponding pixel in the original image output by the image sensor in real time.
[0081] When the grayscale value of an abnormal pixel in the original image output by the image sensor in real time is 0 or 255, the grayscale value of the pixel after neighborhood correction can be used for interpolation to obtain the corrected grayscale value of the abnormal pixel.
[0082] After correction, in each frame of the image output by the image sensor, all pixels in the identified defective areas have been replaced with their corresponding expected grayscale values, while the remaining areas retain their original data. In the end, the image output by the image sensor not only retains all the details of the original scene, but also eliminates the poor image quality problem caused by the defects of the image sensor itself, greatly improving timeliness.
[0083] In this embodiment, the image sensor can be a monochrome image sensor, a color image sensor, or other types of image sensors. When the image sensor is a color image sensor, it is only necessary to ensure that the number of pixels in the width and height of the minimum envelope rectangle covering the defect area are both even numbers. Similarly, the number of pixels in the width and height of the neighborhood region of the minimum envelope rectangle are also even numbers.
[0084] like Figure 3 As shown, based on the same inventive concept, this invention provides an image sensor defect correction system that averages multiple frames of images acquired by an image sensor to obtain a pixel response distribution map, including: The defect identification module is used to identify defect regions in the pixel response distribution map, and each defect region constitutes a point set; The neighborhood analysis module calculates the minimum envelope rectangle corresponding to each defect region based on the position coordinates of pixels in each defect region, and analyzes the pixel mean of the four neighborhood regions of the minimum envelope rectangle to determine the expected gray value of each pixel in the point set. The encoding adjustment module is used to determine the correction coefficient of each pixel based on the ratio between the expected gray value of each pixel and the original gray value of each pixel in the pixel response distribution map. Row sparse coding is performed on the correction coefficients of each point set to generate a correction coefficient tuple corresponding to the scanning line sequence of the image sensor. The correction coefficient tuple consists of the row start coordinates of the point set that overlaps with the scanning line of the image sensor, the width of the point set in the scanning line, and the correction coefficients of the point set in the scanning line. The deduplication correction analysis module is used to determine whether there is an overlapping region in the correction coefficient tuples of each point set on the same scan line. If there is an overlapping region, the correction coefficients of the same pixel in the overlapping region in the two point sets are extracted and averaged to update the correction coefficients of each pixel in the overlapping region. The correction coefficients corresponding to each pixel in the defect region are used to correct the pixel response distribution map.
[0085] For detailed operating methods and principles of each module in the orthodontic system, please refer to the orthodontic method described above, which will not be repeated here.
[0086] The present invention also provides a computer-readable storage medium including a computer program that, when executed by a processor, implements the above-described correction method.
[0087] In practical applications, a computer-readable storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0088] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.
[0089] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0090] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0091] In the description of this specification, references to terms such as "an embodiment," "example," and "specific example" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0092] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.
Claims
1. A method for correcting defects in an image sensor, comprising averaging multiple frames of images acquired by the image sensor to obtain a pixel response distribution map, characterized in that, include: Identify the defective regions in the pixel response distribution map, with each defective region constituting a point set; Based on the position coordinates of pixels in each defect region, calculate the minimum envelope rectangle corresponding to each defect region, and analyze the pixel mean of the neighborhood region of the minimum envelope rectangle to determine the expected gray value of each pixel in the point set. The correction coefficient for each pixel is determined by the ratio between the expected gray value of each pixel and the original gray value of each pixel in the pixel response distribution map. Row sparse coding is performed on the correction coefficients of each point set to generate a correction coefficient tuple corresponding to the scanning line sequence of the image sensor. The correction coefficient tuple consists of the row start coordinates of the point set that overlaps with the scanning line of the image sensor, the width of the point set in the scanning line, and the correction coefficients of the point set in the scanning line. Determine whether there is an overlapping region among the correction coefficient tuples of each point set on the same scan line. If there is an overlapping region, extract the correction coefficients of the same pixel in the two point sets within the overlapping region and calculate the average to update the correction coefficients of each pixel in the overlapping region. Use the correction coefficients corresponding to each pixel in the defect region to correct the pixel response distribution map.
2. The image sensor defect correction method according to claim 1, characterized in that, Extract the gray values of non-defective pixels in the relative positions of the neighborhood regions corresponding to each pixel in the minimum envelope rectangle, calculate the average gray value of each pixel in the minimum envelope rectangle, and then determine the expected gray value of each pixel in the point set based on the mapping relationship between each pixel in each point set and each pixel in the minimum envelope rectangle.
3. The image sensor defect correction method according to claim 1, characterized in that, The method for identifying defective regions in the pixel response distribution map includes: extracting the gray values of pixels whose gray values differ from those of neighboring pixels by less than a set gray value change threshold; correcting the gray values of each pixel in the pixel response distribution map using surface fitting; and performing thresholding processing on the corrected pixel response distribution map to determine the pixels with defects.
4. A method for correcting defects in an image sensor according to claim 1 or 2, characterized in that, Connectivity analysis is performed on the pixel response distribution map after thresholding, and regions with an area greater than a set area threshold are selected as defect regions.
5. The image sensor defect correction method according to claim 4, characterized in that, The method for determining the minimum envelope rectangle includes: extracting the maximum and minimum values of the abscissa and the maximum and minimum values of the ordinate of the same point set; The coordinates of the top left corner of the minimum envelope rectangle are determined based on the minimum values of the x and y coordinates, and the coordinates of the bottom right corner of the minimum envelope rectangle are determined based on the maximum values of the x and y coordinates.
6. The image sensor defect correction method according to claim 1, characterized in that, The method of implementing row sparse coding for the correction coefficients of each point set includes: determining whether each row of the image sensor overlaps with the point set corresponding to all defect regions; if there is overlap, extracting the correction coefficients corresponding to that row of the image; if there is no overlap, not extracting the correction coefficients corresponding to that row of the image.
7. The image sensor defect correction method according to claim 6, characterized in that, The method for determining whether each scan line of the image sensor overlaps with each point set includes: extracting the maximum and minimum values of the ordinate covered by the minimum envelope rectangle corresponding to each point set; if the index value of the scan line is not greater than the maximum value of the ordinate and not less than the minimum value of the ordinate, then the scan line overlaps with the point set; otherwise, the scan line does not overlap with the point set.
8. The image sensor defect correction method according to claim 1, characterized in that, When the same row of images overlaps with the point set corresponding to at least one defect region, the correction coefficients are arranged in ascending order according to the horizontal coordinates corresponding to each correction coefficient.
9. An image sensor defect correction system, applied to the image sensor defect correction method according to any one of claims 1-8, wherein the system averages multiple frames of images acquired by the image sensor to obtain a pixel response distribution map, characterized in that, include: The defect identification module is used to identify defect regions in the pixel response distribution map, and each defect region constitutes a point set; The neighborhood analysis module calculates the minimum envelope rectangle corresponding to each defect region based on the position coordinates of pixels in each defect region, and analyzes the pixel mean of the neighborhood region of the minimum envelope rectangle to determine the expected gray value of each pixel in the point set. The encoding adjustment module is used to determine the correction coefficient of each pixel based on the ratio between the expected gray value of each pixel and the original gray value of each pixel in the pixel response distribution map. Row sparse coding is performed on the correction coefficients of each point set to generate a correction coefficient tuple corresponding to the scanning line sequence of the image sensor. The correction coefficient tuple consists of the row start coordinates of the point set that overlaps with the scanning line of the image sensor, the width of the point set in the scanning line, and the correction coefficients of the point set in the scanning line. The deduplication correction analysis module is used to determine whether there is an overlapping region in the correction coefficient tuples of each point set on the same scan line. If there is an overlapping region, the correction coefficients of the same pixel in the overlapping region in the two point sets are extracted and averaged to update the correction coefficients of each pixel in the overlapping region. The correction coefficients corresponding to each pixel in the defect region are used to correct the pixel response distribution map.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the correction method as described in any one of claims 1-8.