Image correction method, device and electronic equipment

Abstract

The present application provides an image correction method, device and electronic device, which relate to the field of image processing technology. The method can obtain an image frame sequence, perform image stratification on the image frames in the image frame sequence to obtain a basic layer, a background layer, a background detail layer and a target detail layer. Then, a basic non-uniformity correction model is constructed, and the learning rate of the basic non-uniformity correction model is set based on the basic layer, the background layer, the background detail layer and the target detail layer. The basic non-uniformity correction model is trained based on the image frame sequence, and the correction parameters of the basic non-uniformity correction model are optimized based on the learning rate during the training process to obtain a target non-uniformity correction model. The non-uniformity correction is performed on the image frame to be corrected according to the optimized correction parameters through the target non-uniformity correction model. The method can perform stratification processing on the image, and set the learning rate of the non-uniformity correction model according to the stratification result, so as to achieve accurate correction of the image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image correction method, device and electronic equipment. Background Art

[0002] Infrared imaging system is a device or system that receives and processes infrared radiation emitted by an object and converts it into a visible image. Due to the differences in the response characteristics of each pixel in the infrared detector, the captured image is non-uniform, which affects the clarity and accuracy of the image. Therefore, non-uniformity correction can be performed on the image. Non-Uniformity Correction (NUC) is a technology widely used in infrared imaging systems to improve the imaging quality of infrared detectors. The purpose of non-uniformity correction is to eliminate or reduce the non-uniformity of the image and improve the image's spatial resolution, temperature resolution, detection distance and other performance.

[0003] Image non-uniformity correction includes: calibration-based correction method, scene-based adaptive correction method and spatiotemporal adaptive filtering-based correction method. Calibration-based correction method requires external reference source for calibration, the equipment is complex and requires periodic calibration. The scene-based adaptive correction method uses the scene's own information for real-time correction, which can overcome the error caused by response drift, but will cause misleading visual residues in the image. The correction method based on spatiotemporal adaptive filtering cannot adapt to the changes in texture features in different areas of the image, resulting in problems such as blurred edges and uneven background, reducing the effect of non-uniformity correction. Summary of the invention

[0004] The present application provides an image correction method, device and electronic device to solve the problem of poor non-uniformity correction effect.

[0005] In a first aspect, the present application provides an image correction method, comprising:

[0006] Acquire an image frame sequence, wherein the image frame sequence includes a plurality of continuous image frames;

[0007] Performing image layering on the image frame to obtain a basic layer, a background layer, a background detail layer and a target detail layer; the basic layer is used to represent the basic structure information of the image frame, the background layer is used to represent the background information of the image frame, the background detail layer is used to represent the detail information in the background of the image frame, and the target detail layer is used to represent the detail information in the non-background of the image frame;

[0008] Constructing a basic non-uniformity correction model, wherein the basic non-uniformity correction model performs non-uniformity correction based on correction parameters, wherein the correction parameters include a gain parameter and an offset parameter;

[0009] Setting a learning rate of the basic non-uniformity correction model based on the background layer, the background detail layer and the target detail layer, wherein the learning rate is used to control an optimization step size of the correction parameters during training of the basic non-uniformity correction model;

[0010] Training the basic non-uniformity correction model based on the image frame sequence and the base layer, and optimizing correction parameters of the basic non-uniformity correction model based on the learning rate during the training process to obtain a target non-uniformity correction model;

[0011] The target non-uniformity correction model is used to perform non-uniformity correction on the image frame to be corrected according to the optimized correction parameters to obtain a corrected image frame corresponding to the image frame to be corrected.

[0012] In a second aspect, the present application provides an image correction device, comprising:

[0013] A data acquisition module, used for acquiring an image frame sequence, wherein the image frame sequence includes a plurality of continuous image frames;

[0014] The model training module is used to perform image layering on the image frame to obtain a basic layer, a background layer, a background detail layer and a target detail layer; the basic layer is used to represent the basic structure information of the image frame, the background layer is used to represent the background information of the image frame, the background detail layer is used to represent the detail information in the background of the image frame, and the target detail layer is used to represent the detail information in the non-background of the image frame;

[0015] Constructing a basic non-uniformity correction model, wherein the basic non-uniformity correction model performs non-uniformity correction based on correction parameters, wherein the correction parameters include a gain parameter and an offset parameter;

[0016] Setting a learning rate of the basic non-uniformity correction model based on the background layer, the background detail layer and the target detail layer, wherein the learning rate is used to control an optimization step size of the correction parameters during training of the basic non-uniformity correction model;

[0017] Training the basic non-uniformity correction model based on the image frame sequence and the base layer, and optimizing correction parameters of the basic non-uniformity correction model based on the learning rate during the training process to obtain a target non-uniformity correction model;

[0018] The non-uniformity correction module is used to perform non-uniformity correction on the image frame to be corrected according to the optimized correction parameters through the target non-uniformity correction model to obtain a corrected image frame corresponding to the image frame to be corrected.

[0019] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and a bus. The processor and the memory communicate with each other via the bus; the memory stores computer program instructions executable by the processor, and the processor is configured to:

[0020] Acquire an image frame sequence, wherein the image frame sequence includes a plurality of continuous image frames;

[0021] Performing image layering on the image frame to obtain a basic layer, a background layer, a background detail layer and a target detail layer; the basic layer is used to represent the basic structure information of the image frame, the background layer is used to represent the background information of the image frame, the background detail layer is used to represent the detail information in the background of the image frame, and the target detail layer is used to represent the detail information in the non-background of the image frame;

[0022] Constructing a basic non-uniformity correction model, wherein the basic non-uniformity correction model performs non-uniformity correction based on correction parameters, wherein the correction parameters include a gain parameter and an offset parameter;

[0023] Setting a learning rate of the basic non-uniformity correction model based on the background layer, the background detail layer and the target detail layer, wherein the learning rate is used to control an optimization step size of the correction parameters during training of the basic non-uniformity correction model;

[0024] Training the basic non-uniformity correction model based on the image frame sequence and the base layer, and optimizing correction parameters of the basic non-uniformity correction model based on the learning rate during the training process to obtain a target non-uniformity correction model;

[0025] The target non-uniformity correction model is used to perform non-uniformity correction on the image frame to be corrected according to the optimized correction parameters to obtain a corrected image frame corresponding to the image frame to be corrected.

[0026] It can be seen from the above technical solutions that the present application provides an image correction method, device and electronic device, wherein the method can obtain an image frame sequence, perform image stratification on the image frames in the image frame sequence to obtain a basic layer, a background layer, a background detail layer and a target detail layer. Then, a basic non-uniformity correction model is constructed, and a learning rate of the basic non-uniformity correction model is set based on the basic layer, the background layer, the background detail layer and the target detail layer. The basic non-uniformity correction model is trained based on the image frame sequence, and the correction parameters of the basic non-uniformity correction model are optimized based on the learning rate during the training process to obtain a target non-uniformity correction model. Non-uniformity correction is performed on the image frame to be corrected according to the optimized correction parameters through the target non-uniformity correction model. The method can perform stratification processing on the image, and set the learning rate of the non-uniformity correction model according to the stratification result, so as to achieve accurate correction of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A schematic diagram of a flow chart of an image correction method provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of image layering provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of a process for dividing details into layers provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of model training provided in an embodiment of the present application;

[0032] Figure 5 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments will be described in detail below, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following embodiments do not represent all implementation methods consistent with the present application. They are only examples of devices and methods consistent with some aspects of the present application.

[0034] Infrared imaging systems have been evolving along with the development of infrared detectors. In the first generation of infrared imaging systems, linear detectors were used to achieve imaging through one-dimensional optical scanning. With the development of charge-coupled device (CCD) related technologies, infrared focal plane array (IRFPA) detectors were used in the second generation of staring infrared imaging systems. Compared with linear detectors, focal plane detectors have the advantages of high spatial resolution, strong detection capability, and high frame rate during imaging.

[0035] However, the non-uniformity problem of infrared focal plane array will affect the imaging quality of the infrared imaging system, reduce the system's spatial resolution, temperature resolution, detection distance and radiation measurement accuracy, etc., directly restricting the final performance of the system.

[0036] In some embodiments, to address the non-uniformity problem of the infrared focal plane array, the possible response differences between different pixels of the image sensor can be corrected based on non-uniformity correction (NUC), thereby improving the uniformity and accuracy of imaging. Through non-uniformity correction, the image sensor can output stable and reliable image data under different lighting or radiation conditions, thereby improving the performance of the entire imaging system.

[0037] In some embodiments, a calibration-based correction method can be used. This type of method is simple in principle, the hardware is easy to implement and integrate, the correction accuracy is high, and it can be used to measure scene temperature without restrictions on the target. However, this type of method is limited by the correction error caused by the response drift of the infrared focal plane array. During actual correction, a reference source is required for calibration, making the equipment relatively complex. At the same time, periodic calibration is required in practical applications, and the calibration frequency depends on the stability of the system. For application scenarios that require rapid response, such as airborne and missile-borne, it is not easy to achieve rapid response, and the flexibility and real-time performance are poor.

[0038] In some embodiments, scene-based adaptive correction methods can be used, including temporal high pass filtering, constant statistics, neural networks, Kalman filtering, image registration, etc. This type of method can overcome the correction error caused by the response drift of the infrared focal plane array to a certain extent. However, when processing still images after fast-moving targets, this type of method may produce misleading visual residues (ghosting problems), affect visual effects, and lead to problems such as wrong target judgment.

[0039] In some embodiments, a spatiotemporal adaptive filtering non-uniformity correction algorithm can be used. By improving the neural network algorithm and using the guided filtered image as the desired template, the edges of the image are prevented from being smoothed by the filter. When the scene moves, the non-uniformity correction parameters are continuously updated through the strategy of temporal iteration. At the same time, in order to suppress the ghosting phenomenon in the algorithm, an adaptive learning rate based on the combination of the spatial local variance and the temporal scene change rate is set, and the threshold is adaptively adjusted using the previous and next correction parameters.

[0040] However, this type of method has certain limitations. The guided filtering method used in this type of method uses a fixed regularization parameter and may not be able to adapt to the changes in texture features in different areas of the image, resulting in blurred edges, uneven backgrounds, and other problems. In addition, after the guided filtering, the calculation of the adaptive learning rate using the local variance will waste computing resources and slow down the convergence speed, affecting real-time performance. If a lower learning rate is used for the edge area of ​​the image, the neural network learning update of the moving target in the edge area is ignored. In practical applications, the moving target may be located exactly at the edge of the image, and a too low learning rate will reduce the correction effect of these areas. If the learning rate is updated by comparing the scene information of the entire image, when a large number of targets suddenly rush into the scene, if a larger learning rate is still used when the difference between the previous and next information is large, it may lead to over-correction or instability when the actual scene has not changed.

[0041] Therefore, some embodiments of the present application provide an image correction method, which can perform layered processing on the image through weighted guided image filtering (WGIF) and Gaussian filtering, and construct a non-uniformity correction model based on the neural network method, so as to achieve accurate correction of the image. It should be noted that the image correction method provided in the embodiments of the present application is not limited to application in infrared imaging systems, but can also be applied to other fields that need to compensate for non-uniformity phenomena, so as to eliminate the influence of non-uniformity and improve the clarity and accuracy of the image.

[0042] like Figure 1 As shown, Figure 1 A schematic diagram of a flow chart of an image correction method provided in an embodiment of the present application, wherein the image correction method comprises the following steps:

[0043] S101: Acquire an image frame sequence.

[0044] The image frame sequence includes a plurality of continuous image frames. For example, continuous video frame images collected by an infrared focal plane array can be obtained as the image frame sequence.

[0045] S102: Perform image layering on the image frame to obtain a basic layer, a background layer, a background detail layer and an object detail layer.

[0046] Among them, the basic layer is used to represent the basic structure information of the image frame, the background layer is used to represent the background information of the image frame, the background detail layer is used to represent the detail information in the background of the image frame, and the target detail layer is used to represent the detail information in the non-background of the image frame.

[0047] In some embodiments, Figure 2As shown, it is a schematic diagram of image layering provided by an embodiment of the present application. When performing image layering on an image frame, a weighted guided filtering can be performed on the image frame to obtain a basic layer. Gaussian filtering is performed on the image frame to obtain a background layer. The pixel point difference between the basic layer and the background layer is then calculated to obtain a detail layer, and the detail layer is used to characterize the detail information in the image frame. Finally, the detail layer is divided into a background detail layer and a target detail layer. The target detail layer is an area in the detail layer where the pixel point difference value of adjacent image frames is greater than or equal to the difference threshold, and the background detail layer is an area in the detail layer where the pixel point difference value of adjacent image frames is less than the difference threshold.

[0048] In some embodiments, weighted guided filtering is performed on the image frame, that is, a weighted guided filtering method is used to generate a basic layer of detail enhancement and background smoothing, and the detail information in the image is retained by using weighted guided filtering to facilitate subsequent layering operations. For example, weighted guided filtering can be performed on the image frame according to the following formula:

[0049] ;

[0050] ;

[0051] ;

[0052] in, is the output image, which is obtained by linear transformation of the filtered input image p, that is, It is the basic layer obtained by weighted guided filtering of image frames in the image frame sequence. To guide the image, the filtered input image p can be used as the guide image, that is, is an image frame in the image frame sequence. is a window, and the window size can be 5×5. To guide the image In the window The mean value in To guide the image In the window The variance in . is the gain coefficient of the image linear transformation, is the bias coefficient of the linear transformation of the image. For the input image p in the window The mean value in . Representative Window The number of pixels in is 25. is the regularization coefficient, which is used to control the filtering effect of the guided filter.

[0053] When ε is small, the filter output is similar to the original image. When ε is large, in the low variance area, it is equivalent to weighted mean filtering. In the high variance area, the filtering effect on the image becomes weaker, which helps to maintain the edge. Therefore, the control parameter can be introduced , used to control the size of ε, the formula is as follows:

[0054] ;

[0055] ;

[0056] in, is the control parameter. is the pixel i in the image frame in the window The variance within . Pixels in the image frame In the window The variance within Take all the pixels in the image frame. N is the total number of pixels in the image frame. is the regularization factor, which can be , L is the dynamic range of the image frame, which represents the number of pixels of the image. For example, the dynamic range of an image frame with a pixel level of 8 is 256.

[0057] From the above formula, we can see that The size of is positively correlated with the local variance of the image. The larger the local variance, The larger the value is, the more edge smoothing background can be selectively retained in pixels with large local variance. Selecting the filtered input image p as the guide image can obtain the final weighted guided filtering model:

[0058] ;

[0059] ;

[0060] By performing weighted guided filtering on the image frame through the weighted guided filtering model, a basic layer including edge enhancement can be obtained. By introducing a variable regularization parameter, the texture characteristics of different regions of the image can be adapted.

[0061] In some embodiments, a Gaussian filter is performed on the image frame based on a Gaussian filter window of a preset size to obtain a background layer. For example, a 5×5 Gaussian filter window is used to filter the image frame to blur details and smooth the background, thereby obtaining a background layer of the image frame. The discrete form of the Gaussian function is:

[0062] ;

[0063] in, is the standard deviation of the Gaussian function, which can be taken as 1, x and y are the pixel positions of the pixel points, and r is the window radius, which can be taken as 2. The above formula is used to construct a Gaussian filter to smooth the image frame, remove the noise and obtain the background layer of the image frame.

[0064] After obtaining the base layer and the background layer, the detail layer of the image frame can be obtained by subtracting the background layer from the base layer. That is, the detail layer of the image frame can be calculated according to the following formula:

[0065] ;

[0066] in, The pixel coordinates in the detail layer are The pixel value of The pixel coordinates in the base layer are The pixel value of The pixel coordinates in the background layer are The pixel value of .

[0067] In some embodiments, after the detail layer is acquired, the detail layer can be layered based on the frame difference method, and the detail layer can be divided into a background detail layer and a target detail layer. Figure 3 As shown, it is a schematic diagram of the process of dividing the detail layer provided by the embodiment of the present application. A first detail layer and a second detail layer can be obtained, and the first detail layer is the detail layer of the previous frame adjacent to the second detail layer. The pixel point difference value between the first detail layer and the second detail layer is calculated, and the pixel point difference value is the absolute value of the pixel point difference value between the first detail layer and the second detail layer. Obtain a differential threshold, traverse the pixel point differential value, and compare the pixel point differential value with the differential threshold, mark the area where the pixel point differential value is greater than or equal to the differential threshold as the target detail layer, and mark the area where the pixel point differential value is less than the differential threshold as the background detail layer.

[0068] For example, the detail layers corresponding to two consecutive image frames in the image frame sequence are selected, which are the second detail layer at time t and the first detail layer at time t-1. The difference between the two detail layers is calculated and the absolute value is obtained according to the following formula to obtain the difference image O(x, y) of the two frames:

[0069] ;

[0070] in, The pixel coordinates in the second detail layer are The pixel value of . The pixel coordinates in the first detail layer are The pixel value of . The pixel coordinates in the difference image are The pixel value of .

[0071] Then, according to the following formula, the area in the difference image O(x, y) where the pixel value is greater than or equal to the difference threshold is marked as the target detail layer, and the background detail layer is obtained by subtracting the target detail layer from the detail layer:

[0072] ;

[0073] (x,y);

[0074] in, The pixel coordinates in the target detail layer are The pixel value of (x, y) is the pixel coordinate in the background detail layer. The pixel value of The pixel coordinates in the detail layer are The pixel value of The pixel coordinates in the difference image are The pixel value of is the difference threshold, which can be 30.

[0075] It is understandable that the background detail layer is the main cause of ghosting, and the learning rate of background details needs to be reduced to reduce the generation of ghosting. The target detail layer does not produce ghosting due to its fast movement, so the learning rate of target details can be increased to speed up the convergence of the model.

[0076] S103: Constructing a basic non-uniformity correction model.

[0077] Among them, the basic non-uniformity correction model is a neural network model for performing non-uniformity correction. The basic non-uniformity correction model performs non-uniformity correction based on correction parameters, and the correction parameters include gain parameters and bias parameters. In the subsequent model training process, the basic non-uniformity correction model can take the image frame sequence as input, and learn and adjust the gain parameter and bias parameter of each pixel based on the difference between the input image and the expected image, so as to minimize the error between the output image and the expected image, thereby achieving the purpose of correction.

[0078] In some embodiments, a pixel response expression of a basic non-uniformity correction model may be constructed based on a two-point model, and the pixel response expression is:

[0079] ;

[0080] in, The pixel coordinates in the image frame of the nth frame in the image frame sequence input to the basic non-uniformity correction model are The pixel value of . The pixel coordinates of the nth frame in the image frame sequence output by the basic non-uniformity correction model are The pixel value of . The pixel coordinates in the image frame of the nth frame in the image frame sequence are The pixel gain parameter is The pixel coordinates in the image frame of the nth frame in the image frame sequence are The purpose of the neural network model is to update the coefficients of the pixel response expression according to the expected function so as to minimize the error between the output image result and the expected image and achieve the purpose of correction.

[0081] S104: Setting a learning rate of a basic non-uniformity correction model based on the background layer, the background detail layer, and the target detail layer.

[0082] The learning rate is used to control the optimization step size of the correction parameters during the training of the basic non-uniformity correction model. In order to avoid the generation of ghost images during the correction process and speed up the convergence of the model, different learning rates can be set for different layers of the image frame.

[0083] S105: training a basic non-uniformity correction model based on a basic layer of the image frame sequence, and optimizing correction parameters of the basic non-uniformity correction model based on a learning rate during the training process to obtain a target non-uniformity correction model.

[0084] In some embodiments, Figure 4 As shown, it is a schematic diagram of the model training provided by the embodiment of the present application. The pixel response expression can be inversely transformed to obtain the coefficient to be optimized. The pixel response expression after inverse transformation is:

[0085] ;

[0086] ;

[0087] ;

[0088] in, and is the coefficient to be optimized.

[0089] During the training process, the neural network model will look for a set of optimal gain parameters and bias parameters that can minimize the error between the corrected output image and the expected image. To this end, an error function can be set to find the gain parameters and bias parameters that minimize the error function. For example, the mean square error function can be used as the error function, that is, the error function is:

[0090] ;

[0091] ;

[0092] in, is the error function, is the loss function, The pixel coordinates in the image frame of the nth frame in the image frame sequence are The output of the pixel point, The pixel coordinates in the image frame of the nth frame of the expected function of the neural network model are The output of the pixel points, in this embodiment, the basic layer after weighted guided filtering is selected as the expected function, that is, The pixel coordinates in the base layer are The pixel value of .

[0093] In the process of training the model, the gain parameters and bias parameters can be iteratively updated through the optimization algorithm to obtain the optimal gain parameters and bias parameters. For example, the update expression of the coefficient to be optimized can be obtained through the gradient descent method of the neural network:

[0094] ;

[0095] in, The pixel coordinates in the image frame of the nth frame are The optimized coefficients corresponding to the pixel points are The pixel coordinates in the image frame of the n+1th frame are The optimized coefficients corresponding to the pixel points are is the learning rate. The coefficients to be optimized are iteratively optimized according to the update expression until the model converges to obtain the correction parameters when the error function is minimized, thereby obtaining the target non-uniformity correction model.

[0096] In some embodiments, the learning rate of the update expression can be set for the background detail layer, the background layer, and the target detail layer according to the following formula:

[0097] ;

[0098] in, is the learning rate of the background detail layer, is the learning rate of the background layer, is the learning rate of the target detail layer.

[0099] The learning rate of the background detail layer is calculated as follows:

[0100] ;

[0101] in, The pixel coordinates in the background detail layer are The pixel value of is the preset threshold corresponding to the background detail layer, which can be 20. When the pixel value in the background detail layer is less than or equal to the preset threshold, the learning rate of the background detail layer is set to 1. When the pixel value in the background detail layer is greater than the preset threshold, it is considered to be a prominent detail of the background, and the learning rate of the background detail layer is set to 0. The learning rate is not updated to reduce the appearance of background ghosts. In other words, when the background details are prominent, the learning update of the background details is stopped to prevent the generation of ghosts.

[0102] The learning rate of the background layer is calculated as follows:

[0103] ;

[0104] ;

[0105] in, The pixel coordinates in the first background layer are The pixel value of The pixel coordinates in the second background layer are The pixel value of the first background layer is the background layer of the previous frame adjacent to the second background layer. In this embodiment, the information difference of the background layer is determined based on the frame difference method. When the information change of two adjacent image frames is small, it means that the background is almost unchanged. Continuous learning of the background will introduce ghosting, so the learning rate of the background layer is determined according to the information difference of the background layer. M represents the sum of the information differences of the pixels in the background layer. M is positively correlated with the change of background information. The smaller M is, the less the background information is considered to change, so that the learning rate of the background can be reduced to avoid the generation of ghosting. If M is larger, the greater the background information change is considered to change, the learning rate of the background is increased, and the convergence speed of the model is increased.

[0106] The learning rate of the target detail layer is calculated as follows:

[0107] ;

[0108] in, The pixel coordinates in the target detail layer are The pixel value of is the maximum pixel value in the target detail layer. This formula normalizes the learning rate of the target detail layer. It can be understood that the faster the target position changes, the faster the target details change. Ghosting is caused by continuous learning of information about the target object that remains stationary. Therefore, learning the details of the target that is constantly moving will not introduce ghosting, and will speed up the update of the learning rate and accelerate the convergence of the model.

[0109] In this embodiment, a weighted guided filter is used as the desired image of the neural network to better protect the edge and have more useful information. A layered strategy is used to separate the background layer, background detail layer and target detail layer of the image. Setting different learning rates of the neural network for different layers can converge faster and suppress the occurrence of ghosting. The guided filter local variance is used in conjunction to calculate the adaptive learning rate, which improves the image filtering quality while keeping resource consumption unchanged. A targeted learning rate is used for moving targets to ensure their convergence. The information difference is judged based on the background layer to avoid the problem of a large learning rate caused by a large number of new targets entering the scene.

[0110] S106: performing non-uniformity correction on the image frame to be corrected according to the optimized correction parameters by using the target non-uniformity correction model to obtain a corrected image frame corresponding to the image frame to be corrected.

[0111] After the target non-uniformity correction model is trained, the image frame to be corrected can be input into the target non-uniformity correction model, and the non-uniformity correction is performed on the image frame to be corrected through the target non-uniformity correction model. During the correction process, the target non-uniformity correction model will adjust the output signal of each pixel in the image frame to be corrected according to the optimized gain parameters and bias parameters to eliminate the deviation caused by non-uniformity and improve the imaging quality.

[0112] In this embodiment, the image is layered based on the method of weighted guided filtering and Gaussian filtering, and the learning rate of the neural network model is updated according to the results of the layering. Compared with the processing based on the entire image, layering the image can more accurately separate the requirements of different features for learning rates. Based on the frame difference method, the detail layer is divided into a target detail layer and a background detail layer, so that the judgment of the details is more accurate, and the learning rate is updated in a targeted manner to avoid ghosting while reducing the convergence time. Compared with using the variance method and other methods to judge the details, the accuracy is improved, and compared with the method of uniformly setting the learning rates of background details and target details to 0, the convergence time of the model is reduced. At the same time, a learning rate formula suitable for layer processing is set to meet the learning rate of the model for different layers, and the learning rate is set in a targeted manner according to the characteristics of the layer to meet the learning needs of the model for different features.

[0113] Based on the above-mentioned image correction method, some embodiments of the present application further provide an image correction device, which includes a data acquisition module, a model training module and a non-uniformity correction module.

[0114] The data acquisition module is used to acquire an image frame sequence, where the image frame sequence includes a plurality of continuous image frames.

[0115] The model training module is used to perform image layering on the image frame to obtain a basic layer, a background layer, a background detail layer and a target detail layer. The basic layer is used to represent the basic structural information of the image frame, the background layer is used to represent the background information of the image frame, the background detail layer is used to represent the detail information in the background of the image frame, and the target detail layer is used to represent the detail information in the non-background of the image frame.

[0116] A basic non-uniformity correction model is constructed, wherein the basic non-uniformity correction model performs non-uniformity correction based on correction parameters, and the correction parameters include gain parameters and offset parameters.

[0117] The learning rate of the basic non-uniformity correction model is set based on the basic layer, the background layer, the background detail layer and the target detail layer. The learning rate is used to control the optimization step size of the correction parameters during the training process of the basic non-uniformity correction model.

[0118] A basic non-uniformity correction model is trained based on an image frame sequence, and correction parameters of the basic non-uniformity correction model are optimized based on a learning rate during the training process to obtain a target non-uniformity correction model.

[0119] The non-uniformity correction module is used to perform non-uniformity correction on the image frame to be corrected according to the optimized correction parameters through the target non-uniformity correction model to obtain a corrected image frame corresponding to the image frame to be corrected.

[0120] Based on the above image correction method, some embodiments of the present application also provide an electronic device, such as Figure 5 As shown, Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present application, and the electronic device 500 includes: at least one processor 501, at least one communication interface 502, at least one memory 503 and at least one bus 504. Among them, the bus 504 is used to realize the direct connection and communication of these components, the communication interface 502 is used to communicate signaling or data with other node devices, and the memory 503 stores computer program instructions that can be executed by the processor 501. When the electronic device 500 is running, the processor 501 communicates with the memory 503 through the bus 504, and the processor 501 calls the computer program stored in the memory 503 and executes the computer program to realize the image correction method provided in the embodiment of the present application.

[0121] Based on the above-mentioned image correction method, some embodiments of the present application also provide a computer program product, including a computer program stored on a computer-readable storage medium, the computer program including computer program instructions, and when the computer program instructions are executed by a computer, the computer can execute the image correction method in the above-mentioned embodiment.

[0122] Based on the above-mentioned image correction method, some embodiments of the present application also provide a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the image correction method described in the above-mentioned method embodiment.

[0123] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. An image correction method, characterized in that: include: Acquire an image frame sequence, wherein the image frame sequence includes a plurality of continuous image frames; Performing image layering on an image frame to obtain a basic layer, a background layer, a background detail layer and a target detail layer, wherein the basic layer is used to characterize the basic structural information of the image frame, the background layer is used to characterize the background information of the image frame, the background detail layer is used to characterize the detail information in the background of the image frame, and the target detail layer is used to characterize the detail information in the non-background of the image frame; the basic layer is obtained by performing weighted guided filtering on the image frame, the background layer is obtained by performing Gaussian filtering on the image frame based on a Gaussian filtering window of a preset size, the background detail layer and the target detail layer are obtained by dividing the detail layer based on a frame difference method, and the detail layer is obtained based on the pixel point difference value between the basic layer and the background layer; Constructing a basic non-uniformity correction model, wherein the basic non-uniformity correction model performs non-uniformity correction based on correction parameters, wherein the correction parameters include a gain parameter and an offset parameter; Setting the learning rate of the basic non-uniformity correction model based on the product of the learning rate of the background layer, the learning rate of the background detail layer and the learning rate of the target detail layer, wherein the learning rate is used to control the optimization step size of the correction parameters during the training process of the basic non-uniformity correction model; Training the basic non-uniformity correction model based on the image frame sequence and the base layer, and optimizing correction parameters of the basic non-uniformity correction model based on the learning rate during the training process to obtain a target non-uniformity correction model; The target non-uniformity correction model is used to perform non-uniformity correction on the image frame to be corrected according to the optimized correction parameters to obtain a corrected image frame corresponding to the image frame to be corrected.

2. The image correction method according to claim 1, characterized in that: A weighted guided filtering is performed on the image frame according to the following formula: ； ； ； ； in, For the basic layer, is the guide image, the guide image is the image frame, For the window, To guide the image In the window The mean value in To guide the image In the window The variance in is the gain coefficient of the image linear transformation, is the bias coefficient of the linear transformation of the image, is the regularization coefficient, To control the parameters, is the pixel i in the image frame in the window The variance within Pixels in the image frame In the window The variance within is, N is the total number of pixels in the image frame, is the regularization factor.

3. The image correction method according to claim 1, characterized in that: The step of dividing the detail layer into a background detail layer and a target detail layer comprises: Acquire a first detail layer and a second detail layer, where the first detail layer is a detail layer of a previous frame adjacent to the second detail layer; Calculating a pixel difference value between the first detail layer and the second detail layer, where the pixel difference value is an absolute value of a pixel difference value between the first detail layer and the second detail layer; Get the difference threshold; Traversing the pixel point difference values; Marking the area where the pixel point difference value is greater than or equal to the difference threshold as the target detail layer; The area where the pixel point difference value is less than the difference threshold is marked as a background detail layer.

4. The image correction method according to claim 3, characterized in that: The method further comprises: The pixel response expression of the basic non-uniformity correction model is constructed, and the pixel response expression is: ； in, The pixel coordinates in the image frame of the nth frame in the image frame sequence input to the basic non-uniformity correction model are The pixel value of The pixel coordinates in the image frame of the nth frame in the image frame sequence output by the basic non-uniformity correction model are The pixel value of The pixel coordinates in the image frame of the nth frame in the image frame sequence are The pixel gain parameter is The pixel coordinates in the image frame of the nth frame in the image frame sequence are The bias parameters of the pixel points; An inverse transformation is performed on the pixel response expression, and the pixel response expression after the inverse transformation is: ； ； ； Set the error function, which is: ； ； in, is the error function, is the loss function, The pixel coordinates in the base layer are The pixel value of The correction parameters of the basic non-uniformity correction model are iteratively optimized according to the following formula to obtain the correction parameters when the error function is minimized: ； in, The pixel coordinates in the image frame of the nth frame are The optimized coefficients corresponding to the pixel points are The pixel coordinates in the image frame of the n+1th frame are The optimized coefficients corresponding to the pixel points are is the learning rate; The pixel coordinates in the image frame of the nth frame are , The pixel coordinates in the image frame of the n+1th frame are Another optimized coefficient corresponding to the pixel point.

5. The image correction method according to claim 4, characterized in that: The learning rate of the basic non-uniformity correction model is set as follows: ； in, is the learning rate of the background detail layer, is the learning rate of the background layer, is the learning rate of the target detail layer.

6. The image correction method according to claim 4, characterized in that: The learning rate of the background detail layer is calculated as follows: ； in, The pixel coordinates in the background detail layer are The pixel value of is the preset threshold.

7. The image correction method according to claim 4, characterized in that: The learning rate of the background layer is calculated as follows: ； ； in, The pixel coordinates in the first background layer are The pixel value of The pixel coordinates in the second background layer are The first background layer is a background layer of a previous frame adjacent to the second background layer.

8. The image correction method according to claim 4, characterized in that: The learning rate of the target detail layer is calculated as follows: ； in, The pixel coordinates in the target detail layer are The pixel value of is the maximum pixel value in the target detail layer.

9. An image correction device, characterized in that: include: A data acquisition module, used for acquiring an image frame sequence, wherein the image frame sequence includes a plurality of continuous image frames; A model training module, for performing image layering on the image frame to obtain a base layer, a background layer, a background detail layer, and an object detail layer; The basic layer is used to characterize the basic structure information of the image frame, the background layer is used to characterize the background information of the image frame, the background detail layer is used to characterize the detail information in the background of the image frame, and the target detail layer is used to characterize the detail information in the non-background of the image frame; the basic layer is obtained by performing weighted guided filtering on the image frame, the background layer is obtained by performing Gaussian filtering on the image frame based on a Gaussian filtering window of a preset size, the background detail layer and the target detail layer are obtained by dividing the detail layer based on a frame difference method, and the detail layer is obtained based on the pixel difference value between the basic layer and the background layer; Constructing a basic non-uniformity correction model, wherein the basic non-uniformity correction model performs non-uniformity correction based on correction parameters, wherein the correction parameters include a gain parameter and an offset parameter; Setting the learning rate of the basic non-uniformity correction model based on the product of the learning rate of the background layer, the learning rate of the background detail layer and the learning rate of the target detail layer, wherein the learning rate is used to control the optimization step size of the correction parameters during the training process of the basic non-uniformity correction model; Training the basic non-uniformity correction model based on the image frame sequence and the base layer, and optimizing correction parameters of the basic non-uniformity correction model based on the learning rate during the training process to obtain a target non-uniformity correction model; The non-uniformity correction module is used to perform non-uniformity correction on the image frame to be corrected according to the optimized correction parameters through the target non-uniformity correction model to obtain a corrected image frame corresponding to the image frame to be corrected.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores computer program instructions executable by the processor, the processor being configured to: Acquire an image frame sequence, wherein the image frame sequence includes a plurality of continuous image frames; Perform image layering on the image frame to obtain a basic layer, a background layer, a background detail layer and a target detail layer; the basic layer is used to characterize the basic structural information of the image frame, the background layer is used to characterize the background information of the image frame, the background detail layer is used to characterize the detail information in the background of the image frame, and the target detail layer is used to characterize the detail information in the non-background of the image frame; the basic layer is obtained by performing weighted guided filtering on the image frame, the background layer is obtained by performing Gaussian filtering on the image frame based on a Gaussian filtering window of a preset size, the background detail layer and the target detail layer are obtained by dividing the detail layer based on a frame difference method, and the detail layer is obtained based on the pixel point difference value between the basic layer and the background layer; Constructing a basic non-uniformity correction model, wherein the basic non-uniformity correction model performs non-uniformity correction based on correction parameters, wherein the correction parameters include a gain parameter and an offset parameter; Setting the learning rate of the basic non-uniformity correction model based on the product of the learning rate of the background layer, the learning rate of the background detail layer and the learning rate of the target detail layer, wherein the learning rate is used to control the optimization step size of the correction parameters during the training process of the basic non-uniformity correction model; Training the basic non-uniformity correction model based on the image frame sequence and the base layer, and optimizing correction parameters of the basic non-uniformity correction model based on the learning rate during the training process to obtain a target non-uniformity correction model; The target non-uniformity correction model is used to perform non-uniformity correction on the image frame to be corrected according to the optimized correction parameters to obtain a corrected image frame corresponding to the image frame to be corrected.

Images