Adaptive data normalization preprocessing method and system for deep learning image processing
By using an adaptive data normalization preprocessing method, image baselines are identified and calibrated, and outliers are removed. This solves the problem of deep learning models being sensitive to baseline shifts and outliers, and improves the training stability and performance of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-14
AI Technical Summary
Deep learning models are sensitive to baseline shifts and abnormal bright/dark spots in input images, leading to training instability and performance degradation, which existing image normalization preprocessing methods cannot effectively address.
By segmenting the image into blocks and analyzing local statistical characteristics, a dual screening strategy is adopted to identify clean background blocks to determine the lower bound of brightness, remove outliers such as hot pixels, and adaptively determine the dynamic normalization percentage for preprocessing.
It effectively eliminates artifact interference, providing stable and clean input for deep learning models and significantly improving model performance.
Smart Images
Figure CN121861279A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and deep learning technology, and specifically relates to an adaptive data normalization preprocessing method for deep learning image processing. Background Technology
[0002] Deep learning, especially methods based on convolutional neural networks, has achieved remarkable success in low-level visual tasks such as image denoising. Unsupervised or self-supervised denoising methods, because they do not rely on large-scale paired "noisy-clean" training data, have shown great potential in practical applications such as biological microscopy, astronomical observation, and medical image analysis. However, the performance of deep learning models is closely related to the distribution characteristics of their input data. In many scientific and industrial applications, raw image data directly acquired from sensors is often accompanied by two common artifacts that severely interfere with the training process and final results of deep learning models: First, baseline shift. Ideally, background regions in an image containing no signal should have near-zero grayscale values. However, in reality, due to sensor dark current, ambient stray light, or DC bias of electronic devices, the background of an image often appears as a non-zero brightness plane, sometimes even spatially slowly varying. This phenomenon is called baseline shift or background glow. If such images are directly input into a network, the model will be forced to allocate a significant portion of its learning capacity to fitting this low-frequency background pattern, which is irrelevant to the denoising task itself. This encroaches on the effective network capacity used for learning and removing complex noise, leading to a decrease in denoising performance.
[0003] Second, outliers. These mainly manifest as "hot pixels" and "dead pixels." Hot pixels are isolated bright spots with brightness values far exceeding the surrounding real signal, caused by sensor manufacturing defects, cosmic ray impacts, or prolonged exposure; dead pixels are the opposite. These outliers form extreme outliers in the data distribution. During model training, they can generate huge errors in the loss function, leading to gradient explosion or violent oscillations, making it difficult for the optimizer to converge stably. The optimization process may be "hijacked" by these insignificant outliers, deviating from the correct learning of global noise features, severely impairing the model's final performance and generalization ability.
[0004] Existing image normalization preprocessing methods struggle to effectively address these challenges. For instance, the traditional "max-min normalization" method maps the lowest and highest grayscale values of an image to 0 and 1, respectively. This method is extremely sensitive to hot and dead pixels; a single outlier can compress the effective dynamic range of the entire image to an almost indistinguishable degree. One improved method is "fixed percentage normalization," for example, cropping the lowest 10% and the highest 0.1% of pixel values. However, this "one-size-fits-all" fixed ratio cannot adapt to the content of different images. For an image with a broad background and sparse signal, it may fail to effectively remove the baseline; conversely, for an image containing large areas of bright objects, it may incorrectly crop away a large amount of valuable highlight signal.
[0005] Therefore, existing technologies lack a preprocessing method capable of intelligently analyzing image content, adaptively performing baseline calibration, and accurately removing outliers such as hot pixels. Developing such a method to provide downstream deep learning models with a stable and information-pure "ideal" input is crucial for improving model training efficiency and final performance. Summary of the Invention
[0006] This invention provides an adaptive data normalization preprocessing method and system for deep learning image processing. Its purpose is to solve the technical problem that deep learning models (especially unsupervised denoising networks) are sensitive to baseline offset and abnormal bright / dark spots (such as hot pixels) of the input image, which leads to unstable training and performance degradation.
[0007] Firstly, the present invention aims to provide an adaptive data normalization preprocessing method for deep learning image processing, comprising the following steps: S1: Obtain the input image, divide it into multiple image blocks, and calculate the standard deviation of the gray values of the pixels within each image block and the standard deviation of the frequency domain amplitude of the image block. S2: Based on the standard deviation of the grayscale value and the standard deviation of the frequency domain amplitude, the lower bound percentage is adaptively determined; S3: Adaptively determine the upper bound percentage; S4: Based on the lower and upper bound percentages, perform percentage normalization on the input image to generate a preprocessed image for input into the deep learning model.
[0008] Furthermore, a preferred embodiment is provided: in step S1, the calculation steps for the standard deviation of the frequency domain amplitude include: Performing a two-dimensional discrete Fourier transform on the image patch yields a complex spectrum matrix; Calculate the amplitude spectrum of the complex spectrum matrix; Set the center point of the amplitude spectrum to zero, and calculate the standard deviation of all elements in the processed amplitude spectrum. This is the frequency domain amplitude standard deviation.
[0009] Furthermore, a preferred embodiment is provided: S2 includes: S21: Based on the standard deviation of gray values and the standard deviation of frequency domain amplitude of all image blocks, a joint criterion is determined. The joint criterion is used to identify background blocks, and image blocks that meet the conditions of low gray value standard and low frequency domain amplitude standard are identified as the background block set. S22: Collect all pixels in the background block set, calculate their average gray value, and obtain the lower limit value of brightness; S23: Compare the lower limit value of brightness with all pixels of the input image, and calculate the proportion of pixels with a value less than the lower limit value to the total number of pixels, as the lower limit percentage.
[0010] Furthermore, a preferred embodiment is provided: S3 includes: S31: Identify image blocks that do not belong to the background block set in S21 as signal block sets; S32: Calculate the kurtosis of the pixel values of each image block in the signal block set, and filter out the healthy signal block set according to the kurtosis threshold; S33: Collect all pixels in the health signal block set, calculate their preset high percentile value, and obtain the upper limit value of brightness; S34: Compare the upper limit value of brightness with all pixels of the input image, and calculate the proportion of pixels whose value is not greater than the upper limit value to the total number of pixels, as the upper limit percentage.
[0011] Furthermore, a preferred solution is provided: the joint criterion is expressed as: & , in, The threshold is dynamically calculated based on the set of grayscale standard deviations of all blocks. The threshold is dynamically calculated based on the set of standard deviations of the frequency amplitudes of all blocks.
[0012] Furthermore, a preferred solution is provided: In step S32, image blocks smaller than a kurtosis threshold are selected as healthy signal blocks, generating a set of healthy signal blocks. The kurtosis threshold is calculated using robust statistics, and the formula is: , in, It is the interquartile range. Indicates the first Kurtosis value of an image patch The median. is a constant scaling factor.
[0013] Furthermore, a preferred embodiment is provided: S4 includes: S41: In the overall pixel distribution of the original image, determine the actual gray values corresponding to the lower bound percentage and the upper bound percentage; S42: Apply a linear mapping formula to each pixel in the original image to obtain a preprocessed image for input to the deep learning model.
[0014] Secondly, the purpose of this invention is to propose an adaptive data normalization preprocessing system for deep learning image processing, said system being implemented based on an adaptive data normalization preprocessing method for deep learning image processing as described in any one or more of the above-mentioned schemes, said system comprising: Image pre-processing module: used to acquire the input image, divide it into multiple image blocks, and calculate the standard deviation of the gray values of the pixels within each image block and the standard deviation of the frequency domain amplitude of the image block. Lower Bound Percentage Calculation Module: Used to adaptively determine the lower bound percentage based on the standard deviation of the grayscale value and the standard deviation of the frequency domain amplitude; Upper bound percentage calculation module: used to adaptively determine the upper bound percentage; Image normalization module: used to perform percentage normalization processing on the input image based on the lower bound percentage and the upper bound percentage, and generate a preprocessed image for input into the deep learning model.
[0015] Thirdly, the present invention aims to provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and when the processor runs the computer program stored in the memory, the processor executes an adaptive data normalization preprocessing method for deep learning image processing according to any one or more of the above-described schemes.
[0016] Fourthly, the present invention aims to provide a computer-readable storage medium for storing a computer program that executes an adaptive data normalization preprocessing method for deep learning image processing as described in any one or more of the above-described schemes.
[0017] Compared with the prior art, the advantages of the present invention are: The method proposed in this invention divides the image into blocks and analyzes the local statistical characteristics of each block, employing a dual screening strategy: First, "clean background blocks" are identified through low standard deviation screening, and their pixel mean is used to determine a robust lower bound for brightness calibration of the image baseline; then, "healthy signal blocks" are identified through dual screening using high standard deviation and low kurtosis, and their high percentile value is used to determine an upper bound for brightness that excludes hot pixel interference. Finally, a dynamic normalized percentage is calculated based on these two adaptively determined upper and lower bounds. , This invention preprocesses images to effectively eliminate interference artifacts, providing deep learning models with stable and clean input, thereby significantly improving model performance.
[0018] This invention is applicable to image preprocessing scenarios. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating an adaptive data normalization preprocessing method for deep learning image processing, as described in a specific embodiment of the present invention. Detailed Implementation
[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.
[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0023] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.
[0024] Implementation Method 1 This embodiment provides an adaptive data normalization preprocessing method for deep learning image processing. By dividing the image into blocks and analyzing the local statistical characteristics of each block, a dual screening strategy is employed: First, "clean background blocks" are identified through low standard deviation screening, and their pixel mean is used to determine a robust lower bound for brightness calibration of the image baseline. Then, "healthy signal blocks" are identified through dual screening using high standard deviation and low kurtosis, and their high percentile value is used to determine an upper bound for brightness that excludes hot pixel interference. Finally, a dynamic normalization percentage is calculated based on these two adaptively determined upper and lower bounds. , The image is preprocessed. This includes the following steps: Step 1: Obtain the input digital image to be processed , to image Divided into The size is Image blocks and for the set Each image patch in Calculate the statistical characteristics of two different domains, including: Spatial domain characteristics: standard deviation of gray values ( It directly calculates the standard deviation of the gray values of all pixels within a block to measure its flatness in space.
[0025] Frequency domain characteristics: Standard deviation of frequency domain amplitude ( This value measures the dispersion of frequency components within a block, excluding average brightness. A sparsely structured background block, its... It will be very small. The calculation steps are as follows: a. For image patches Performing a two-dimensional discrete Fourier transform (2D-DFT) yields a complex spectrum matrix.
[0026] b. Calculate the amplitude spectrum of the complex spectrum matrix.
[0027] c. Set the center point of the amplitude spectrum (DC component) to zero. This step is to eliminate interference from the DC component (which represents the average brightness of the block and is independent of the structure).
[0028] d. Calculate the standard deviation of all elements in the processed amplitude spectrum to obtain the frequency domain amplitude standard deviation.
[0029] Step 2: Adaptively determine the minimum lower bound percentage ( This step includes: Step 2.1: Based on the set of grayscale standard deviations and frequency domain amplitude standard deviations of all image patches, determine a joint criterion for identifying background patches, and identify image patches that simultaneously satisfy the conditions of low grayscale standard deviation and low frequency domain amplitude standard deviation as the background patch set. ); Furthermore, an image patch The joint criterion for identifying a block as background is: & , in, The threshold is dynamically calculated based on the set of grayscale standard deviations of all image patches. The calculation method is as follows: , is a constant scaling factor.
[0030] The threshold is dynamically calculated based on the set of standard deviations of the frequency amplitudes of all image patches. The calculation method is as follows: , is a constant scaling factor.
[0031] A block is only included in the background block set if it is spatially flat and frequency-sparse enough. .
[0032] Step 2.2: Compile the background block set For all pixels in the dataset, calculate their average grayscale value to obtain the lower limit of brightness. ; Step 2.3: Set the lower limit value of brightness With the input image Compare all pixels, and calculate a value less than The percentage of pixels in a given area relative to the total number of pixels is the lower bound percentage. ; , in, This is an indicator function; its value is 1 if the condition is true, and 0 otherwise. This represents the number of pixels across the image width. This represents the number of pixels at the height of the image.
[0033] The method described in this embodiment can effectively eliminate interference artifacts, providing deep learning models with stable and clean input, thereby significantly improving model performance.
[0034] Step 3: Adaptively determine the upper bound percentage ( This step includes: Step 3.1: Identify image blocks that do not belong to the background block set described in Step 2.1 as signal block sets. Alternatively, using the exact same standard deviation threshold classification method as step 2, all blocks that meet the following conditions are classified as signal blocks, forming a set. : & , Step 3.2: To eliminate abnormal signals that may be caused by hot pixels or other factors in the signal block set, process each pixel block in the signal block set. Calculate the kurtosis of its pixel values Kurtosis measures the sharpness and tail thickness of a data distribution. An image patch containing hot pixels will exhibit an extreme tail, far removed from the main distribution, resulting in an abnormally high kurtosis value. Therefore, a kurtosis threshold is set. A set of health signal blocks is selected based on the kurtosis threshold. ), to remove image patches containing extreme outliers; where It can be calculated using robust statistics: , Where IQR is the interquartile range. Indicates the first Kurtosis value of an image patch The median. The scaling factor is a constant. This will adjust the kurtosis. Less than The signal blocks are filtered out to form a set of "healthy signal blocks". .
[0035] Step 3.3: Collect all pixels in the health signal block set to form a health signal pixel set. Set an extremely high preset percentile for this set. In this embodiment, the preferred value is set to 99.8 or 99.9, where the pixel value serves as a robust upper limit for brightness. , represented as: .
[0036] Since the set of healthy signal pixels has already undergone two rounds of screening and contains virtually no outliers, this step can safely determine the upper limit of brightness within it.
[0037] Step 3.4: Set the upper limit of brightness Compared to the original full-size image Compare all pixels and calculate values less than or equal to The percentage of pixels in the adaptive upper bound is the proportion of the total number of pixels. : .
[0038] Step 4: Use the lower bound percentage determined in Steps 2 and 3 ( ) and upper bound percentage ( ), for the input image Percentage normalization is performed to generate a preprocessed image for input into the deep learning model.
[0039] Specifically, it includes: Perform normalization: to obtain adaptive and Then, first from the original image In the overall pixel distribution, determine the actual grayscale values corresponding to these two percentages. and Then, for the image Each pixel in The following linear mapping formula is applied to obtain the preprocessed pixels used for deep learning. : , The final image has a pixel value smaller than The pixel value is effectively calibrated to 0, and the pixel value is greater than 0. The signal is effectively calibrated to 1, and the dynamic range of the effective signal is fully preserved and mapped to the [0, 1] interval.
[0040] Implementation Method 2 This embodiment provides an adaptive data normalization preprocessing system for deep learning image processing. The system is implemented based on an adaptive data normalization preprocessing method for deep learning image processing as described in Embodiment 1. The system includes: Image pre-processing module: used to acquire the input image, divide it into multiple image blocks, and calculate the standard deviation of the gray values of the pixels within each image block and the standard deviation of the frequency domain amplitude of the image block. Lower Bound Percentage Calculation Module: Used to adaptively determine the lower bound percentage based on the standard deviation of the grayscale value and the standard deviation of the frequency domain amplitude; Upper bound percentage calculation module: used to adaptively determine the upper bound percentage; Image normalization module: used to perform percentage normalization processing on the input image based on the lower bound percentage and the upper bound percentage, and generate a preprocessed image for input into the deep learning model.
[0041] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.
Claims
1. An adaptive data normalization preprocessing method for deep learning image processing, characterized in that, Includes the following steps: S1: Obtain the input image, divide it into multiple image blocks, and calculate the standard deviation of the gray values of the pixels within each image block and the standard deviation of the frequency domain amplitude of the image block. S2: Based on the standard deviation of the grayscale value and the standard deviation of the frequency domain amplitude, the lower bound percentage is adaptively determined; S3: Adaptively determine the upper bound percentage; S4: Based on the lower and upper bound percentages, perform percentage normalization on the input image to generate a preprocessed image for input into the deep learning model.
2. The adaptive data normalization preprocessing method for deep learning image processing according to claim 1, characterized in that, In step S1, the calculation steps for the standard deviation of the frequency domain amplitude include: Performing a two-dimensional discrete Fourier transform on the image patch yields a complex spectrum matrix; Calculate the amplitude spectrum of the complex spectrum matrix; Set the center point of the amplitude spectrum to zero, and calculate the standard deviation of all elements in the processed amplitude spectrum. This is the frequency domain amplitude standard deviation.
3. The adaptive data normalization preprocessing method for deep learning image processing according to claim 1, characterized in that, S2 includes: S21: Based on the standard deviation of gray values and the standard deviation of frequency domain amplitude of all image blocks, a joint criterion is determined. The joint criterion is used to identify background blocks, and image blocks that meet the conditions of low gray value standard and low frequency domain amplitude standard are identified as the background block set. S22: Collect all pixels in the background block set, calculate their average gray value, and obtain the lower limit value of brightness; S23: Compare the lower limit value of brightness with all pixels of the input image, and calculate the proportion of pixels with a value less than the lower limit value to the total number of pixels, as the lower limit percentage.
4. The adaptive data normalization preprocessing method for deep learning image processing according to claim 2, characterized in that, S3 includes: S31: Identify image blocks that do not belong to the background block set in S21 as signal block sets; S32: Calculate the kurtosis of the pixel values of each image block in the signal block set, and filter out the healthy signal block set according to the kurtosis threshold; S33: Collect all pixels in the health signal block set, calculate their preset high percentile value, and obtain the upper limit value of brightness; S34: Compare the upper limit value of brightness with all pixels of the input image, and calculate the proportion of pixels whose value is not greater than the upper limit value to the total number of pixels, as the upper limit percentage.
5. The adaptive data normalization preprocessing method for deep learning image processing according to claim 3, characterized in that, The joint criterion is expressed as follows: & ， in, The threshold is dynamically calculated based on the set of grayscale standard deviations of all blocks. The threshold is dynamically calculated based on the set of standard deviations of the frequency amplitudes of all blocks.
6. The adaptive data normalization preprocessing method for deep learning image processing according to claim 3, characterized in that, In step S32, image blocks with kurtosis less than a kurtosis threshold are selected as healthy signal blocks, and a set of healthy signal blocks is generated. The kurtosis threshold is calculated using robust statistics, and the formula is: ， in, It is the interquartile range. Indicates the first Kurtosis value of an image patch The median. is a constant scaling factor.
7. The adaptive data normalization preprocessing method for deep learning image processing according to claim 1, characterized in that, S4 includes: S41: In the overall pixel distribution of the original image, determine the actual gray values corresponding to the lower bound percentage and the upper bound percentage; S42: Apply the linear mapping formula to each pixel in the original image to obtain a preprocessed image for input to the deep learning model.
8. An adaptive data normalization preprocessing system for deep learning image processing, characterized in that, The system is implemented based on an adaptive data normalization preprocessing method for deep learning image processing as described in any one of claims 1-7, and the system includes: Image pre-processing module: used to acquire the input image, divide it into multiple image blocks, and calculate the standard deviation of the gray values of the pixels within each image block and the standard deviation of the frequency domain amplitude of the image block; Lower Bound Percentage Calculation Module: Used to adaptively determine the lower bound percentage based on the standard deviation of the grayscale value and the standard deviation of the frequency domain amplitude; Upper bound percentage calculation module: used to adaptively determine the upper bound percentage; Image normalization module: used to perform percentage normalization processing on the input image based on the lower bound percentage and the upper bound percentage, and generate a preprocessed image for input into the deep learning model.
9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes an adaptive data normalization preprocessing method for deep learning image processing according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that executes an adaptive data normalization preprocessing method for deep learning image processing according to any one of claims 1-7.