CMOS image denoising method, system and computer readable medium based on FPGA brightness self-adaptation

Abstract

The present application relates to the technical field of CMOS image sensor noise reduction, and discloses a CMOS image noise reduction method, system and computer readable medium based on FPGA brightness self-adaption. The method receives and caches the image output by the CMOS image sensor through the FPGA, and performs brightness self-adaptive image filtering on the image by using the high-speed parallel processing capability of the FPGA. First, the image is row-cached by using the cache module in the FPGA, and the number of rows of the cache is the size of the noise reduction template; then, the image data in the template is subjected to mean filtering or median filtering through the ambient illuminance judgment and self-adaptive filtering module, and the processed image is cached after a preset time delay. Finally, the image in the cache is output by using the serial-parallel conversion module in the FPGA, so that real-time filtering processing of the CMOS image is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of CMOS image sensor noise reduction technology, and particularly relates to a CMOS image noise reduction method, system and computer readable medium based on FPGA brightness self-adaptation. BACKGROUND

[0002] With the rapid development of CMOS technology, CMOS image sensors have been widely used in various occasions. However, when the CMOS image sensor converts the optical signal into the electrical signal, various noises are always introduced, which is not conducive to the subsequent image transmission and observation. In order to solve the problem that the image is affected by the observation due to too much noise, the image containing noise must be processed in real time.

[0003] At present, the image noise reduction method for the CMOS image sensor mainly includes median filter, average filter and bilateral filter. The median filter is a nonlinear filter, and the principle is to replace the middle pixel value with the median value of the pixel values in the filter template, which is mainly used to process the salt and pepper noise. The average filter is a linear filter, and the principle is to replace the center pixel value with the average value of the pixel values in the filter template, which is mainly used to process the Gaussian noise. However, the noise distribution and noise degree of the CMOS image sensor are different under different illuminations. If a single filter method is used for real-time noise reduction, the image details may be lost under high illumination, or the noise reduction intensity may be insufficient under low illumination. Although the complex hybrid filter algorithm can solve the problem of processing different noises, the algorithm has high complexity and is difficult to realize in FPGA, which is not conducive to real-time processing and display. SUMMARY

[0004] In view of the problems in the prior art, according to the first aspect of the purpose of the present application, a CMOS image noise reduction method based on FPGA brightness self-adaptive filter is provided, which includes the following steps:

[0005] Step 1, the FPGA internal cache module receives the image output by the CMOS image sensor, and performs row cache processing on the image, wherein the number of cache rows is determined according to the preset filter template size;

[0006] Step 2, the average gray value and the exposure gain of the image are used to estimate the current illumination analog value, and the filter mode is adaptively selected based on the current illumination analog value and the environment illumination state;

[0007] Step 3, the image data in the template is processed according to the selected filter mode, and the filtered image data is obtained;

[0008] Step 4, the FPGA internal cache module stores the image data processed by the filter; and

[0009] Step 5, output the filtered image.

[0010] Thus, the CMOS image noise reduction method provided by the application receives and caches the image output by the CMOS image sensor by using the FPGA, and performs the image filtering on the image by using the high-speed parallel processing capability of the FPGA.

[0011] As an optional example, in the implementation process of the method, first, the image is row-cached by using the cache module in the FPGA, and the number of rows of the cache is the size of the noise reduction template; then the image data in the template is filtered by using the ambient illuminance judgment and adaptive filtering module, and the processed image is cached after a preset time delay; finally, the image in the cache is output by using the serial-parallel conversion module in the FPGA, so that the real-time filtering processing of the CMOS image is realized.

[0012] In the CMOS image noise reduction method based on the FPGA brightness adaptive filtering provided by the application, different noise reduction processing of different intensity and different types is performed on the image according to different ambient illuminance, so that the details of the filtered image are not lost under high brightness, the power consumption of the image processing system is reduced, and the image quality is at a high level.

[0013] Since the noise of the image output by the CMOS sensor generally shows an upward trend as the illuminance decreases, the application performs the pre-processing on the ambient illuminance, calculates the weighted average gray value G ave of the whole image of a single frame, and performs the operation on the calculation result, the real-time exposure time t exp of the CMOS image sensor and the real-time analog gain value v gain , obtains the current analog ambient illuminance B cu by combining the weighted coefficients c exp and c gain . After the analog ambient illuminance B cu is obtained, the salt and pepper noise and the Gaussian noise in the image are further processed: if the ambient illuminance B cu is less than the illuminance threshold B st , it is proved that the noise in the current image is serious, and the mean filtering module needs to be turned on to process the Gaussian noise; otherwise, the mean filtering module needs to be turned off to reduce the power consumption.

[0014] In order to prevent the edge details in the image after the mean filtering from being lost seriously, the edge judgment is performed before the mean filtering, the gradient is calculated to determine whether the center pixel of the template is located in the edge of the image, and the variable gradient threshold G st is determined, and if the calculated center pixel gradient G stIf G > G, it means the center pixel is located at the edge of the image. In this case, mean filtering is not performed, and the original data is output directly. Conversely, if G > G, the center pixel is located at the edge of the image. st If the center pixel is not an edge in the image, then it can be subjected to mean filtering, and the resulting pixel value P after weighted mean filtering can be output. ave To remove Gaussian white noise.

[0015] To prevent excessive detail loss after median filtering, this invention employs an adaptive median filtering method. First, it assesses salt-and-pepper noise and compares the center pixel value C. l C p The average value P of the surrounding 8 pixels ave The size. This invention employs a variable tolerance D. gal and D val The filter intensity can be set, meaning the mean difference between the center pixel and surrounding pixels can be adjusted. Variable tolerance D under high illumination. val Greater than the variable tolerance D under low illumination gal This further reduces the smoothing effect of the high-light, low-noise denoising algorithm on the image. When C l C p Greater than P ave When the sum of tolerances is reached, P is enabled. cen Replace C l C p Conversely, continue outputting the center pixel value C. l C p .

[0016] In a second aspect of the present invention, a CMOS image noise reduction system based on FPGA brightness adaptation is also proposed, comprising:

[0017] One or more processors;

[0018] The memory stores operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the aforementioned flow of the FPGA-based brightness-adaptive CMOS image denoising method.

[0019] In a third aspect of the invention, a computer-readable medium for storing software is also provided, the software comprising instructions executable by one or more computers, the instructions causing the one or more computers to perform operations including the aforementioned flow of the FPGA-based brightness adaptive CMOS image denoising method.

[0020] Compared with the prior art, the CMOS image denoising method based on FPGA brightness self-adaptation provided by the application aims to perform denoising processing on the CMOS image sensor output image under different ambient illuminance, which can not only preserve the details of the image under high brightness, but also effectively suppress the Gaussian white noise and salt and pepper noise when the low-illumination noise increases, thereby ensuring the image quality and reducing the power consumption of hardware.

[0021] It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter. Additionally, all combinations of claimed subject matter are contemplated as part of the inventive subject matter.

[0022] The foregoing and other aspects, embodiments and features of the present teachings are more fully described and understood by reference to the following description taken in conjunction with the accompanying drawings. Other aspects, embodiments and features of the present teachings will be apparent from the description that follows, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical, or nearly identical, component that is illustrated in various figures is represented with a like numeral. For purposes of clarity, not every component is called out in every drawing. Embodiments of various aspects of the present teachings will now be described, by way of example, with reference to the drawings, in which:

[0024] Figure 1 A flow chart of a CMOS image denoising processing method according to an exemplary embodiment of the present application.

[0025] Figure 2 A diagram of pixel position relationships in a 3x3 filter template according to an embodiment of the present application.

[0026] Figure 3 A flow chart of sorting and calculating the median of pixel values in a filter template according to an embodiment of the present application.

[0027] Figure 4 A flow chart of judging whether a center pixel of a template is an edge point according to an embodiment of the present application.

[0028] Figure 5 An effect comparison diagram of image enhancement according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to better understand the technical content of the present application, specific embodiments are described below with reference to the accompanying drawings.

[0030] Aspects of the present disclosure are described herein with reference to the drawings, which are meant to be illustrative only and not limiting in any way. Embodiments of the present disclosure are not necessarily intended to include all aspects of the present disclosure. It should be understood that various concepts and embodiments introduced above and discussed in greater detail below can be implemented in any of numerous ways, as the disclosed concepts and embodiments are not limited to any particular manner of implementation. Additionally, some aspects of the present disclosure can be utilized independently, or in any suitable combination with other aspects of the present disclosure.

[0031] In conjunction Figure 1 The FPGA-based brightness adaptive CMOS image noise reduction method of the exemplary embodiment shown receives and caches the image output by the CMOS image sensor through the FPGA, and uses the high-speed parallel processing capability of the FPGA to perform brightness adaptive image filtering on the image.

[0032] In the implementation process of the method, first, the image is line buffered by the internal cache module of the FPGA, and the number of buffered lines is the size of the noise reduction template; then the image data in the template is subjected to mean filtering or median filtering through the ambient illuminance judgment and adaptive filtering module, and after a predetermined time delay, the processed image is cached. Finally, the image in the cache is output using the internal serial-to-parallel conversion module of the FPGA, realizing real-time filtering processing of the CMOS image.

[0033] Figure 1 The flowchart shown represents an example implementation process, which includes the following steps 1-5:

[0034] Step 1, the internal cache module of the FPGA receives the image output by the CMOS image sensor and performs line buffering processing of the image, wherein the number of buffered lines is determined according to the size of the preset filter template;

[0035] Step 2, estimate the current illuminance analog value according to the average gray value and exposure gain of the image, and adaptively select the filtering mode based on the current illuminance analog value and the ambient illuminance state;

[0036] Step 3, process the image data in the template according to the selected filtering mode to obtain filtered image data;

[0037] Step 4, the internal cache module of the FPGA stores the filtered image data; and

[0038] Step 5, output the filtered image.

[0039] In the foregoing step 1, the internal cache module of the FPGA receives the image output by the CMOS image sensor and performs line buffering processing of the image, including:

[0040] The 12-bit image data output by the CMOS image sensor is denoted as data, the 1-bit row synchronization signal is denoted as hs, the 1-bit frame synchronization signal is denoted as vs, and the 1-bit data valid signal is denoted as de;

[0041] Before the noise reduction processing, the row buffering is performed in the following manner:

[0042] The 15-bit data corresponding to a pixel, i.e., data, hs, vs, and de, are integrated, and the row buffering operation is performed through the shift register inside the FPGA, and the number of buffered rows is the preset filter template size.

[0043] As shown in FIG. 3, an example of a filter template (3 rows) is exemplarily represented, which is a 3x3 filter array. Figure 2

[0044] The three rows of pixels are sequentially denoted as F l , C l , and N l .

[0045] The three columns of pixels are sequentially denoted as F p , C p , and N p .

[0046] The coordinate values of the nine pixels of the filter array are the combinations of the row and column pixel coordinates, and are specifically as follows:

[0047] F l F p ]]> F l C p ]]> F l N p ]]> C l F p ]]> [C l C p ]]> C l N p ]]>

[0010] N l F p ]]> N l C p ]]> N l N p ]]>

[0048] As an optional embodiment, in step 2, the current illumination analog value is estimated according to the average gray value of the image and the exposure gain, and the filter mode is adaptively selected based on the current illumination analog value and the ambient illumination state, including:

[0049] Step 2-1, the average gray values G s (i) of different regions of a frame of image are counted in combination with the vs and hs signals, wherein w represents the weight of the average values of different regions, i represents the bit number of different regions, and k represents the total number of regions; the average gray value G ave of a frame of image is obtained through accumulation and shift operation:

[0050]

[0051] Step 2-2, the current illumination analog value B cu is calculated through the weighted average gray value G ave and the adaptive filter brightness threshold value G adp :

[0052] B cu =G adp *G ave *c​br

[0053] wherein c br is an illuminance threshold coefficient.

[0054] If B cu > B st , it represents that the ambient illuminance is higher, and the first mode of filter processing is performed.

[0055] If B cu ≤ B st , it represents that the ambient illuminance is lower, and the second mode of filter processing is performed.

[0056] wherein as an optional example, the adaptive filter luminance threshold G adp is calculated according to the following formula:

[0057] G adp = t exp · c exp + v gain · c gain

[0058] wherein t exp is a real-time exposure time of the CMOS image sensor, v gain is a real-time analog gain of the CMOS image sensor, c exp is an exposure adjustment coefficient, and c gain is a gain adjustment coefficient.

[0059] In the embodiments of the present application, it should be understood that the real-time exposure time and the real-time gain are controlled by the AEC (Auto Exposure Control) and AGC (Auto Gain Control) modules of the CMOS image sensor.

[0060] In combination Figure 1 , in step 3, the image data in the template is processed according to the selected filter mode to obtain the filtered image data, including:

[0061] The first mode of filter processing:

[0062] (1) Based on the filter template in step 1, the filter array is 3x3:

[0063] Three rows of pixels are represented as F l , C l , and N l , respectively.

[0064] Three columns of pixels are represented as F p , C p , and N p , respectively.

[0065] The coordinate values of the nine pixels of the filter array are the combination of row and column pixel coordinates, which are specifically shown in Figure 2 ;

[0066] (2) The average value P of the pixel values of the filter template is calculated according to the following formula ave , wherein c1-c8 are the weighting coefficients of the pixels around the center pixel:

[0067]

[0068] (3) The median value P of the pixel values of the filter template is calculated by five times of sorting according to the flow shown in Figure 3 ;

[0069] First, the maximum value, the median value and the minimum value of each column of the three columns of data of the filter template are calculated respectively, to obtain a group of maximum values, a group of median values and a group of minimum values;

[0070] Then, the minimum value in the group of maximum values, the median value in the group of median values and the maximum value in the group of minimum values are sorted respectively based on the obtained group of maximum values, the group of median values and the group of minimum values, to obtain a minimum value, a median value and a maximum value in turn;

[0071] Finally, the median value MED of the minimum value, the median value and the maximum value obtained above is calculated, which is taken as the median value P of the filter template cen ;

[0072] (4) Whether the center pixel of the filter template is an edge point is judged by the following process according to the flow shown in Figure 4 ;

[0073] First, the gradients in the horizontal direction, the vertical direction, the 45° direction and the 135° direction with the center pixel C l C p of the filter template as the center are calculated respectively;

[0074] Then, the gradients in the four directions are added to obtain the gradient value G of the center point;

[0075] Finally, the size of G and G st is compared, and G st is a preset edge gradient value:

[0076] If G>G st , P ave is output as the original data D ave for the next step;

[0077] If G≤G st , C l C p is output as the original data D ave for the next step;

[0078] wherein, as an optional way, the gradients G of the four directions are determined in the following way 0° 、G 90° 、G 45° 、G 135° :

[0079] G 0° =|2*C l C p –C l F p –C l N p |

[0080] G 90° =|2*C l C p –F l C p –N l C p |

[0081] G 45° =|2*C l C p –F l N p –N l F p |

[0082] G 135° =|2*C l C p –F l F p –N l N p |

[0083] Then, the gradient value G of the center point is calculated in the following way:

[0084] G 2 =G0 2 +G 90 2 +G 45 2 +G 135 2

[0085] wherein, G 0° , G 90° , G 45° , G 135° respectively represent the gradients of the horizontal, vertical, 45°, 135° four directions;

[0086] (5) Set D val as a preset luminance value tolerance, compare Dave With P ave +D val the size of:

[0087] If D ave >P ave +D val , output P cen as the filtered image data of the center pixel;

[0088] If D ave ≤P ave +D val , output D ave as the filtered image data of the center pixel;

[0089] The filtering process in the second mode:

[0090] (1) According to the processing method of the first mode, the mean value P ave of the pixel values of the filter template and the median value P cen of the pixel values of the filter template are calculated:

[0091] (2) Compare the size of C l C p and P ave , D gal is a preset luminance tolerance:

[0092] If C l C p >P ave +D gal , output P cen as the filtered image data of the center pixel;

[0093] If C l C p ≤P ave +D gal , output C l C p as the filtered image data of the center pixel, i.e. the original image data remains unchanged.

[0094] In step 4, for the filtered image data, the image is cached in the FPGA internal cache module after a preset time delay.

[0095] In step 5, the image in the cache is output using the FPGA internal serial-to-parallel conversion module.

[0096] Next, we will describe the implementation process of CMOS image noise reduction processing based on FPGA luminance adaptive filtering in more detail, taking into account the requirements of real-time performance and hardware performance.

[0097] In CMOS images, 12-bit image data is denoted as data, line synchronization signal is denoted as hs, frame synchronization signal is denoted as vs, and data valid signal is denoted as de.

[0098] Before noise reduction, perform image caching operations according to Method 1.

[0099] When making adaptive judgments, follow method two; when performing noise reduction processing, follow method three.

[0100] Method 1:

[0101] The 15 bits of data corresponding to a certain pixel, namely data (12 bits), hs (1 bit), vs (1 bit), and de (1 bit), are integrated and then line buffered through the shift register inside the FPGA.

[0102] The number of cached lines is the intended template size (taking three lines as an example).

[0103] After caching is completed, filtering and noise reduction processing is performed based on the adaptive judgment result obtained by method two. As described above, the current simulated illuminance value is estimated based on the average gray value and exposure gain of the image, and the filtering mode is adaptively selected based on the current simulated illuminance value and the ambient illuminance state, and the filtering and noise reduction processing is performed according to method three or method four below.

[0104] Method 2:

[0105] 1) Combining the vs and hs signals, calculate the average grayscale value G of different regions in a frame of image. s (i), where w represents the weight of the mean of different regions, i represents the position number of different regions, and k represents the total number of regions; the average grayscale (brightness) value G of a frame of image is obtained through cumulative calculation. ave :

[0106]

[0107] 2) By weighted average gray level G ave and adaptive filtering brightness threshold G adp Calculate the current simulated illuminance value B cu :

[0108] B cu =G adp *G ave *c br

[0109] Where c br This is the illuminance threshold coefficient proposed in this invention.

[0110] If B cu >B st This indicates that the ambient light level is relatively high, so method four is applied.

[0111] If B cu ≤B st , it represents that the ambient illumination is low, and the method three is used for processing.

[0112] It should be understood that in the method two, the noise of the CMOS sensor output image generally shows an upward trend as the illumination decreases. Therefore, in the implementation process of the present application, the weighted average gray value G ave of the whole image of a single frame is calculated, and the calculation result is operated with the real-time exposure time t exp and the real-time analog gain value v gain of the CMOS image sensor, combined with the weighted coefficients c exp and c gain to obtain the current analog ambient illumination B cu (B cu is only used for comparison calculation, and is not the real ambient illumination).

[0113] Method three:

[0114] 1) A 3x3 filter array is established by the three-row data buffer signal established by the method one.

[0115] The three rows of pixels are represented as F l , C l , N l in turn; the three columns of pixels are represented as F p , C p , N p .

[0116] The coordinate values of the nine pixels are the combinations of the row and column pixel coordinates, as shown in the following table. Figure 2

[0117] The center pixel is represented as C l C p , and the first pixel is represented as F l F p .

[0118] The pixel value mean P ave is calculated according to the following formula, wherein c1-c8 are the weighted coefficients of the pixels around the center pixel:

[0119] 2) Combined with Figure 3 , the median value of the pixel value of the filter template is calculated by five times of sorting according to the following flow:

[0120] Firstly, the maximum value, the median value and the minimum value in the three columns of data of the filter template are calculated by sorting A, and the three columns of data are represented as 1, 2, 3, and the maximum value, the median value and the minimum value in the three columns of data are represented as A, B, C respectively.​

[0121] Further, the minimum value (D1), the median value (E2) and the maximum value (F3) in each group of values in the maximum value, the median value and the minimum value of the three columns are calculated respectively by sorting, and three specific values are obtained;

[0122] Further, the median value MED of D1, E2 and F3 is calculated by sorting E, that is, the median value P of the filter template cen ;

[0123] 3) Combined Figure 4 , whether the template center pixel is an edge point is judged by the following flow:

[0124] The gradients in the horizontal, vertical, 45° and 135° directions of the center point C l C p are calculated respectively;

[0125] Further, the gradients in the four directions are added to obtain the gradient value G of the center point;

[0126] The size of G and G st is compared, and G st is the preset value of the edge gradient:

[0127] If G>G st , P ave is output as the original data D ave of the next step;

[0128] If G≤G st , C l C p is output as the original data D ave of the next step;

[0129] 4) D val is the preset luminance value tolerance:

[0130] If D ave >P ave +D val , P cen is output as the center pixel filtered image data;

[0131] On the contrary, if D ave ≤P ave +D val , D ave is output as the center pixel filtered image data.

[0132] Method four:

[0133] 1) The calculation steps in method three are the same, and the mean value and the median value of the pixel values of the filter template are calculated;

[0134] 2) compare C l C p and P ave , D gal is a preset luminance value tolerance:

[0135] If C l C p >P ave +D gal , output P cen as the center pixel filtered image data;

[0136] If C l C p ≤P ave +D gal , output C l C p as the center pixel filtered image data, i.e., the original image data is unchanged.

[0137] In combination with the aforementioned method three and method four, after obtaining the simulated ambient illuminance B cu , the present application processes the salt and pepper noise and Gaussian noise in the image. If the ambient illuminance B cu is less than the illuminance threshold B st , it proves that the noise in the current image is serious, and the mean filter module needs to be turned on to process the Gaussian noise; otherwise, the mean filter module needs to be turned off to reduce power consumption.

[0138] The present application mainly uses mean filtering to process Gaussian white noise. In order to prevent the edge details in the image after mean filtering from being lost seriously, edge judgment is performed before mean filtering, and whether the center pixel of the template is located in the edge of the image is determined by calculating the gradient. The present application formulates a variable gradient threshold G st . If the calculated center pixel gradient G>G st , it indicates that the center pixel is in the edge of the image, and therefore the original data is directly outputted without mean filtering; otherwise, if G>G st , it indicates that the center pixel is not in the edge of the image, and therefore the mean filtering can be performed on it, and then the pixel value P ave after weighted mean filtering is outputted to remove the Gaussian white noise.

[0139] At the same time, after mean filtering, the present application mainly uses median filtering to process the salt and pepper noise. In order to prevent too many details from being lost after median filtering, the present application adopts an adaptive median filtering method, and first judges the salt and pepper noise. The sum of the center pixel value C l C p and the mean P ave of the surrounding eight pixel points is compared.The size of the variable tolerance D gal and D val The setting of the filtering strength (i.e., the mean difference of the center pixel and the surrounding pixels is adjustable) is performed. The variable tolerance D val at high illumination is greater than the variable tolerance D gal at low illumination, which further reduces the smoothing effect of the image by the noise reduction algorithm at high illumination and low noise. When C l C p is greater than P ave and the sum of the tolerance, P cen is enabled instead of C l C p , and vice versa, the center pixel value C l C p is continued to be output.

[0140] So far, by the method of the present application, the CMOS image under different ambient illuminations is processed for noise reduction, which can not only preserve the details of the image at high brightness, but also effectively suppress the Gaussian white noise and salt and pepper noise when the noise increases at low illumination, thereby ensuring the image quality and reducing the power consumption of the hardware.

[0141] In combination with Figure 5 the test comparison chart shown, (a) represents the original image output by the sensor without filtering and noise reduction, (b) represents the image after global filtering using the prior art, and (c) represents the image after filtering processing using the present application. It can be seen that after global filtering processing using the prior art, the details of the CMOS image are severely lost, while after filtering processing using the present application, the original details are still preserved, and the noise at low illumination is effectively controlled without being significantly increased.

[0142] It should be understood that in combination with the above embodiments, the second aspect according to the object of the present application also proposes a CMOS image noise reduction system based on FPGA brightness self-adaptation, comprising:

[0143] one or more processors;

[0144] a memory storing instructions operable to cause the one or more processors to perform operations when executed by the one or more processors, the operations comprising the flow of the CMOS image noise reduction method based on FPGA brightness self-adaptation of any of the preceding embodiments, such as the specific implementation flow shown in Figure 1 .

[0145] In combination with the above embodiments, according to a third aspect of the present application, a computer readable medium storing software is also provided, the software comprising instructions executable by one or more computers, the instructions causing the one or more computers to perform operations by such execution, the operations comprising the flow of the FPGA-based luminance adaptive CMOS image denoising method of any of the preceding embodiments, e.g. Figure 1 the specific implementation flow shown.

[0146] While the present application has been disclosed in connection with the preferred embodiments shown, it should be understood that certain modifications can be made thereto without departing from the spirit and scope of the application as set forth in the appended claims.

Claims

1. A CMOS image denoising method based on FPGA brightness self-adaptation, characterized in that, The method comprises the following steps: Step 1, the FPGA internal cache module receives the image output by the CMOS image sensor and performs row cache processing on the image, wherein the number of cache rows is determined according to a preset filter template size; Step 2, estimate the current illumination analog value according to the average gray value of the image and the exposure gain, and select the filter mode adaptively based on the current illumination analog value and the ambient illumination state; Step 3, process the image data in the template according to the selected filter mode to obtain filtered image data; Step 4, the FPGA internal cache module stores the filtered image data; and Step 5, output the filtered image; In the step of receiving the image output by the CMOS image sensor and performing row cache processing on the image by the FPGA internal cache module, the following steps are included: The 12-bit image data output by the CMOS image sensor is denoted as data, the 1-bit row synchronization signal is denoted as hs, the 1-bit frame synchronization signal is denoted as vs, and the 1-bit data valid signal is denoted as de; Before the noise reduction processing, the row cache is performed in the following manner: The 15-bit data corresponding to a certain pixel, data, hs, vs, and de, are integrated, and the row cache operation is performed through the shift register inside the FPGA, and the number of cache rows is the preset filter template size; The step of estimating the current illumination analog value according to the average gray value of the image and the exposure gain, and selecting the filter mode adaptively based on the current illumination analog value and the ambient illumination state, comprises: Step 2-1, combine vs and hs signal, calculate the average value of gray value of different area G s (i), w represents the weight of the average value of different area, i represents the position number of different area, k represents the total number of area; the average gray value G of a frame image is obtained by accumulation and shift operation ave : Step 2-2, by weighted average gray value G ave and adaptive filter luminance threshold G adp Calculate the current illumination analog value B cu : B cu = G adp * G ave * c br wherein c br is an illuminance threshold coefficient; If B cu > B st , it represents that the ambient illuminance is higher, and the filtering processing of the first mode is performed; If B cu ≤ B st , it represents that the ambient illuminance is low, and the filtering processing of the second mode is performed.

2. The FPGA-based brightness adaptive CMOS image denoising method according to claim 1, characterized in that, The step of processing the image data in the template according to the selected filter mode to obtain filtered image data, comprises: The filter processing of the first mode: (1) The filter array is 3x3 based on the filter template described in step 1: Three rows of pixels are represented in sequence as F l , C l , N l ; The three columns of pixels are denoted F p , C p , N p ; The coordinate values of the nine pixels of the filter array are the combinations of row and column pixel coordinates, which are as follows: (2) The mean value P of the pixel values of the filter template is calculated as follows ave where c1to c8are weighting coefficients for the pixels around the center pixel: (3) The median value of the pixel values of the filter template is calculated through five sorting operations: First, the maximum value, the median value and the minimum value of each column of the three columns of data of the filter template are calculated to obtain a set of maximum values, a set of median values and a set of minimum values; Then, based on the obtained set of maximum values, set of median values and set of minimum values, the minimum value in the set of maximum values, the median value in the set of median values and the maximum value in the set of minimum values are sorted respectively to obtain a minimum value, a median value and a maximum value in turn; Finally, the median MED of the aforementioned minimum, median and maximum values is calculated, i.e. the median P of the filter template cen ; (4) The following process is used to determine whether the center pixel of the filter template is an edge point: First, the gradients of horizontal, vertical, 45°, 135° directions are calculated respectively with the center pixel C of the filter template as the center. l C p point as the center. Then, the gradients in the four directions are added to obtain the gradient value G of the center point; Finally, the size of G and G st is compared, G st is the edge gradient preset value: If G > G st , then output P ave as raw data D ave for the next step. If G < G st , then output C l C p as raw data D ave for next step. (5) Set D val D is a preset luminance value tolerance ave and P ave +D val : If D ave > P ave + D val , then output P cen as the center pixel filtered image data; If D ave ≤ P ave + D val , then output D ave as the center pixel filtered image data; The filter processing of the second mode: (1) According to the processing manner of the first mode, the mean value P of the pixel values of the filter template is calculated ave and the median value P of the pixel values of the filter template cen : (2) Compare C l C p and P ave The size of D gal is a preset luminance value tolerance: If C l C p >P ave +D gal , then output P cen as the center pixel filtered image data; If C l C p ≤ P ave + D gal , then output C l C p The filtered image data as the center pixel, i.e. the original image data is unchanged.

3. The FPGA-based brightness adaptive CMOS image denoising method according to claim 2, characterized in that, In the step 3, the adaptive filter brightness threshold G is determined as follows adp : G adp = t exp • c exp + v gain • c gain where t exp is the real-time exposure time of the CMOS image sensor, v gain is the analog gain of the CMOS image sensor, c exp is the exposure adjustment coefficient, c gain is the gain adjustment coefficient.

4. The FPGA-based brightness adaptive CMOS image denoising method according to claim 3, characterized in that, In said step 3, the gradients G of the four directions are determined in the following way 0° , G 90° , G 45° , G 135° : G 0° = |2*C l C p –C l F p –C l N p | G 90° = |2*C l C p –F l C p –N l C p | G 45° = |2*C l C p –F l N p –N l F p | G 135° = |2*C l C p –F l F p –N l N p | Then, the gradient value G of the center point is calculated in the following manner: G 2 = G0 2 + G 90 2 + G 45 2 + G 135 2 where G 0° , G 90° , G 45° , G 135° respectively represent the gradients of horizontal, vertical, 45°, 135° four directions.

5. The FPGA-based brightness adaptive CMOS image denoising method according to claim 1, characterized in that, In the step 4, for the filtered image data, the image is cached in the FPGA internal cache module after a preset time delay.

6. The FPGA-based brightness adaptive CMOS image denoising method according to claim 1, characterized in that, In the step 5, the image in the cache is output by using the serial-to-parallel conversion module inside the FPGA.

7. A CMOS image denoising system based on FPGA brightness self-adaptation, characterized in that, It comprises: one or more processors; A memory storing instructions operable to be executed, the instructions, when executed by the one or more processors, cause the one or more processors to perform operations comprising the flow of the method for CMOS image noise reduction based on FPGA brightness self-adaption according to any one of claims 1-6.

8. A computer readable medium storing software, characterized in that: The software includes instructions executable by one or more computers, the instructions, through such execution, cause the one or more computers to perform operations comprising the flow of the method for CMOS image noise reduction based on FPGA brightness self-adaption according to any one of claims 1-6.

Images