A sequence image distortion and severe thermal radiation effect fusion correction method

Abstract

The application discloses a kind of sequence image distortion and severe thermal radiation effect fusion correction method, this method includes: obtaining the image sequence containing different degree atmospheric turbulence distortion and severe thermal radiation degradation, and preliminary thermal radiation processing is carried out to each image in image sequence, obtain preprocessed image sequence;Distortion correction is carried out to preprocessed image sequence, and distortion correction image sequence is obtained;Distortion correction image sequence is normalized, and weight image sequence is generated by calculating local contrast feature and brightness feature, and using from front to back strategy, distortion correction image sequence and weight image sequence are fused into a correction image.The method of the application is simple and efficient, can carry out image correction to with atmospheric turbulence distortion and severe thermal radiation superposition effect, by executing the method, the distortion and thermal radiation degradation effect of image can be effectively removed, image quality and signal-to-noise ratio are improved, which is helpful for subsequent implementation target identification and detection etc. Operation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image distortion and aero-optical thermal radiation effect correction, and particularly relates to a sequence image distortion and severe thermal radiation effect fusion correction method. BACKGROUND

[0002] When an aircraft flies at a high speed in the atmosphere, a boundary layer, a wake and other flow phenomena are formed on the surface of the aircraft, and the air temperature, density and pressure in these flow regions are different from those of the surrounding atmosphere, thereby causing a change in refractive index. When light passes through these non-uniform atmospheric regions, the propagation path of the light is disturbed, thereby causing aberration or light wave distortion.

[0003] Meanwhile, the optical head cover of the detector is affected by airflow during high-speed movement, and the kinetic energy of the airflow in the boundary layer near the surface of the head cover is dissipated and converted into heat energy, causing the temperature of the optical window to rapidly rise. This convection heat exchange between the high-speed airflow and the surface of the head cover is called aerodynamic heating.

[0004] The above two degradation factors have an impact on the optical system or sensor on the aircraft, causing distortion and thermal radiation degradation of the detected image, and further causing failure of subsequent target detection and identification.

[0005] In order to eliminate the influence of distortion and thermal radiation effect on imaging of optical equipment, there are currently some effective methods, such as replacing infrared window materials with high transmission and low emissivity, variable integration time infrared imaging technology, adaptive optical technology, and methods based on deep learning. Although the above technologies can effectively prevent the saturation of optical equipment, the method is relatively complex to operate and has a high cost.

[0006] In addition, most of the existing correction methods for image distortion and severe thermal radiation degradation effect are for correcting single degradation effect of a single image, and there are few methods for correcting multiple degradation effect sequence images. SUMMARY

[0007] In view of the defects in the prior art, the application provides a sequence image distortion and severe thermal radiation effect fusion correction method, which can simply and efficiently remove image distortion and severe thermal radiation effect, improve image signal-to-noise ratio, and help subsequent implementation of target identification and detection operations.

[0008] The technical solution adopted by the application to solve its technical problems is as follows:

[0009] The first aspect of the application provides a sequence image distortion and severe thermal radiation effect fusion correction method, which comprises the following steps:

[0010] S1, obtain an image sequence containing different degrees of atmospheric turbulence distortion and severe thermal radiation degradation, and perform preliminary thermal radiation processing on each image in the image sequence to obtain a preprocessed image sequence;

[0011] S2, perform distortion correction on the preprocessed image sequence to obtain a distortion-corrected image sequence;

[0012] S3, perform normalization processing on the distortion-corrected image sequence, generate a weight image sequence by calculating local contrast features and brightness features, and use a front-to-back strategy to fuse the distortion-corrected image sequence and the weight image sequence into a corrected image, specifically including:

[0013] S301, perform normalization processing on the distortion-corrected image sequence, and calculate the local contrast features and brightness features of each image, calculate the contrast weight term of each image based on the local contrast features of all images, take the brightness features of each image as the brightness weight term, and generate a weight image sequence according to the contrast weight term and the brightness weight term of each image;

[0014] S302, perform Gaussian filtering and down-sampling on the first and second images of the distortion-corrected image sequence to obtain two Gaussian pyramids of the images, perform Gaussian filtering and down-sampling on the weight images corresponding to the two images to obtain two Gaussian pyramids of the weight images, multiply the two Gaussian pyramids of the images and the corresponding Gaussian pyramids of the weight images pixel by pixel to obtain two Gaussian pyramids containing weight information, add the two Gaussian pyramids containing weight information in the same scale to form a new Gaussian pyramid, restore the new Gaussian pyramid to the original image to obtain a first fused image, and at the same time, add the two Gaussian pyramids of the weight images in the same scale to obtain a Gaussian pyramid of the weight image corresponding to the fused image;

[0015] S303, take the current fused image and the next image of the distortion-corrected image sequence as the two images to be fused, fuse them according to the same fusion logic to obtain the next fused image and the Gaussian pyramid of the weight image corresponding to the next fused image, and continue to use the front-to-back strategy until all images of the distortion-corrected image sequence are fused into a corrected image.

[0016] In the above scheme, step S1 specifically includes:

[0017] S101, perform weighted least squares filtering on each image in the image sequence;

[0018] S102, input the filtered image into a cubic spline surface equation to fit a thermal radiation effect bias field for each image;

[0019] S103, subtract the thermal radiation effect bias field of each image from the image to obtain a pre-processed image sequence.

[0020] In the above scheme, step S2 specifically comprises:

[0021] S201, extract the heavy thermal radiation saturation region and the non-saturation effective image information region block of each image in the pre-processed image sequence, calculate the median pixel value of each pixel position in the non-saturation effective information region block of the pre-processed image sequence, and obtain a median pixel value prior image;

[0022] S202, determine the overlapping region of the non-saturation effective image information region block of the first image to the last image in the pre-processed image sequence, obtain the overlapping non-saturation effective image information region block of the first image to the last image in the pre-processed image sequence, and extract the feature point pair of the median pixel value prior image and the overlapping non-saturation effective image information region block of the first image to the last image in the pre-processed image sequence.

[0023] S203, according to the feature point pair of the median pixel value prior image and the overlapping non-saturation effective image information region block of the first image to the last image in the pre-processed image sequence, determine the inverse transformation matrix of the first image to the last image in the pre-processed image sequence to the median pixel value prior image, and calculate the average inverse transformation matrix, and perform distortion correction on the last image in the pre-processed image sequence based on the average inverse transformation matrix.

[0024] S204, determine the overlapping region of the non-saturation effective image information region block of the first image to the second last image in the pre-processed image sequence, and perform distortion correction on the second last image according to the same distortion correction logic; continue to use this strategy from back to front, and sequentially complete the distortion correction of the third last image to the first image, and finally obtain a distortion corrected image sequence.

[0025] In the above scheme, in step S201, the heavy thermal radiation saturation region and the non-saturation effective image information region block of each image in the pre-processed image sequence are extracted, comprising:

[0026] The pre-processed image sequence is normalized to obtain a normalized image sequence.

[0027] The region of each image in the normalized image sequence with a pixel gray value greater than a preset gray threshold is regarded as a heavy thermal radiation saturation region, and other regions are regarded as non-saturation effective image information region blocks, and the non-saturation effective image information region block of each image is extracted from the pre-processed image sequence.

[0028] In the above scheme, step S2 specifically further comprises:

[0029] S205, updating the median pixel value prior image according to the distortion correction image sequence;

[0030] S206, performing distortion correction on the distortion preliminary correction image sequence again based on the updated median pixel value prior image, to obtain a final distortion correction image sequence.

[0031] In the above scheme, step S301 specifically comprises:

[0032] performing normalization processing on each image in the distortion correction image sequence Q, to obtain a normalized distortion correction image sequence R;

[0033] calculating the local contrast feature of the i-th image in the normalized distortion correction image sequence R, and the formula is as follows: R i

[0034]

[0035] In the formula, x and y are pixel coordinates; is the local contrast feature; SIFT is a SIFT operator; is an L1 norm;

[0036] calculating the contrast weight term of the i-th image based on the local contrast features of all images:

[0037]

[0038] In the formula, n is the total number of images in the image sequence;

[0039] calculating the brightness feature of the i-th image as the brightness weight term:

[0040]

[0041] In the formula, is a brightness parameter;

[0042] calculating the weight image of the i-th image according to the contrast weight term and the brightness weight term of the i-th image:

[0043]

[0044] performing normalization on the weight image:

[0045]

[0046] In the formula, ​​​​​​​​​​is a very small positive number to prevent the denominator from being zero; is the normalized weight image, thus obtaining a weight image sequence E.

[0047] In the above scheme, step S301 further specifically includes:

[0048] The weight image sequence is guided filter optimized to obtain an optimized weight image sequence W.

[0049] In the above scheme, step S302 specifically includes:

[0050] The first image Q 1 and the second image Q 2 of the distortion correction image sequence Q are Gaussian filtered and down-sampled to construct a Gaussian pyramid of the two images and :

[0051] ;

[0052] In the formula, m , n is the relative coordinate in the kernel; k is the Gaussian kernel radius; is the Gaussian kernel weight, wherein is the standard deviation of the Gaussian distribution; x and y is the pixel coordinate, is the down-sampling factor; l =0,1,… L indicates the number of pyramid layers, L is the maximum number of layers, indicates the l layer image, the 0th layer image is the original image Q;

[0053] Each layer image of the Gaussian pyramid of the two images is adaptively edge-aware guided filtered to obtain filtered pyramid images and :

[0054] ;

[0055] In the formula, is the filtered pyramid image; r represents the local window radius, is the regularization parameter, is the input image; indicates guided filtering;

[0056] Similarly, the weight images corresponding to the two images W 1 andW 2. Perform Gaussian filtering and down-sampling to get the Gaussian pyramid of two weight images and ;

[0057] Multiply the filtered Gaussian pyramid of two images and the corresponding Gaussian pyramid of weight images pixel by pixel to get two Gaussian pyramids containing weight information and Add the two Gaussian pyramids containing weight information in the same scale to form a new Gaussian pyramid :

[0058] ;

[0059] Up-sample and accumulate the new Gaussian pyramid from the coarsest scale l = L to the finest scale l =0 in turn to get a new image and normalize the image pixel value to [0, 255] to get the first fused image.

[0060] At the same time, add the Gaussian pyramids of two weight images in the same scale to get the Gaussian pyramid of the weight image corresponding to the fused image.

[0061] According to the second aspect of the present application, a computer device is provided, comprising a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to realize the steps of the sequence image distortion and severe thermal radiation effect fusion correction method according to any one of the first aspect.

[0062] According to the third aspect of the present application, a computer readable storage medium is provided, which stores programs or instructions, and the programs or instructions are executed by the processor to realize the steps of the sequence image distortion and severe thermal radiation effect fusion correction method according to any one of the first aspect.

[0063] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0064] The application provides a sequence image distortion and severe thermal radiation effect fusion correction method, which realizes the collaborative correction of atmospheric turbulence distortion and severe thermal radiation effect in a sequence image. Firstly, a weighted least square filter and a cubic spline surface fitting are used to remove a thermal radiation bias field, so as to reserve effective information for subsequent processing. Secondly, a median pixel value prior image and an inverse sequence distortion correction strategy are combined, feature point matching and non-reflective similarity transformation are used to realize distortion correction. Finally, an adaptive weight image sequence is generated based on local contrast and brightness features, and after guided filtering optimization, a Gaussian pyramid multi-scale fusion is used to fuse the distortion correction image sequence and the weight image sequence, so as to obtain the final corrected image. The method is simple and efficient, can correct images with atmospheric turbulence distortion and severe thermal radiation superposition effect, and can effectively remove the distortion and thermal radiation degradation effect of the image, improve the image quality and signal-to-noise ratio, and help subsequent target recognition and detection operations. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 A flowchart of a sequence image distortion and severe thermal radiation effect fusion correction method provided by the embodiment of the application is shown in the figure.

[0066] Figure 2 A schematic diagram of a raw degraded image sequence Z provided by the embodiment of the application is shown in the figure.

[0067] Figure 3 A schematic diagram of a preprocessed image sequence S obtained after thermal radiation effect initial correction provided by the embodiment of the application is shown in the figure.

[0068] Figure 4 A schematic diagram of a distortion initial correction result sequence Y provided by the embodiment of the application is shown in the figure.

[0069] Figure 5 A schematic diagram of a distortion final correction result sequence Q provided by the embodiment of the application is shown in the figure.

[0070] Figure 6 A clear image provided by the embodiment of the application is shown in the figure.

[0071] Figure 7 A final distortion and thermal radiation saturation region correction image provided by the embodiment of the application is shown in the figure.

[0072] Figure 8 An image fusion schematic diagram provided by the embodiment of the application is shown in the figure.

[0073] Figure 9 A hardware structure schematic diagram of a computer device provided by the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0074] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0075] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application, and for those of ordinary skill in the art, the present application can also be applied to other similar scenarios without creative efforts based on the accompanying drawings. In addition, it can be understood that although the efforts made in the development process can be complex and lengthy, for those of ordinary skill in the art related to the disclosure of the present application, some designs, manufacturing or production changes based on the technical content disclosed in the present application are only routine technical means and should not be understood as insufficient disclosure of the present application.

[0076] In the present application, "embodiments" means that the specific features, structures or properties described in combination with the embodiments can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily refer to the same embodiments, nor is it necessarily mutually exclusive or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0077] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terms "a", "an", "one", "this", and the like, as used in the present application, do not denote number restriction, but can denote singular or plural. The terms "include", "comprise", "have", and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device including a series of steps or modules (units) is not limited to the listed steps or units, but can further include steps or units not listed, or can further include other steps or units inherent to the process, method, product, or device. The terms "connect", "connected", "couple", and the like, are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Multiple" refers to two or more. "And / or" describes the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. The terms "first", "second", "third", and the like, are merely to distinguish similar objects, and do not represent a specific order for the objects.

[0078] The present application provides a sequence image distortion and severe thermal radiation effect fusion correction method, comprising the following steps: preliminary correction of the thermal radiation effect of the degraded image sequence; extracting an effective information region block to obtain a prior image; according to the feature points matched by feature extraction, using a back-to-front strategy, constructing a distortion correction image sequence; calculating the local contrast and brightness features of the distortion correction image sequence to generate a weight image sequence; using a guided filter to optimize the weight image sequence, and performing weighted fusion on the distortion correction image sequence to obtain the final corrected image. The method of the present application is simple and efficient, and can correct images with atmospheric turbulence distortion and severe thermal radiation superposition effect. By executing the method of the present application, the distortion and thermal radiation degradation effect of the above images can be effectively removed, and the image quality and signal-to-noise ratio can be improved.

[0079] As shown in Figure 1 The sequence image distortion and severe thermal radiation effect fusion correction method of the present application comprises the following steps:

[0080] S1, obtaining an image sequence containing different degrees of atmospheric turbulence distortion and severe thermal radiation degradation, and preliminarily processing each image in the image sequence for thermal radiation, the specific process comprising:

[0081] S101, performing weighted least squares filtering on the degraded image sequence;

[0082] S102, the filtered image is brought into a cubic spline surface equation, and a thermal radiation effect bias field is fitted out;

[0083] S103, the thermal radiation effect is preliminarily removed, and a preprocessed image sequence is obtained.

[0084] S2, distortion correction is performed on the preprocessed image sequence based on a non-saturated effective information area block, and the specific process includes:

[0085] S201, a heavy thermal radiation saturated area and an effective image information area block of each image in the image preprocessing sequence are extracted, a median pixel value of each pixel position in the effective information area block is calculated, and a median pixel value prior image is obtained;

[0086] S202, the median pixel value prior image and the effective image information area block of each frame of image in the preprocessed image sequence are matched by using a SURF algorithm for local irregular area feature extraction and matching, and a matched feature point pair is saved;

[0087] S203, according to the matched feature point pair in step S202, in the non-saturated effective image information area block overlapped by the first image to the last image in the preprocessed image sequence, the median pixel value prior image is referred to, a geometric transformation and an average inverse transformation matrix are calculated by a non-reflection similarity transformation, and a distortion preliminary correction result of the last degenerated image of the preprocessed image sequence is constructed;

[0088] S204, a strategy from back to front is adopted, step S203 is repeated, a distortion correction result of the second last degenerated image is constructed based on the non-saturated effective image information area block overlapped by the first image to the second last image in the preprocessed image sequence, and the distortion preliminary correction result of all degenerated images in the sequence is obtained by circulation in turn;

[0089] S205, the median pixel value prior image is updated according to the distortion preliminary correction result, the updated median pixel value prior image is referred to, the image sequence after the distortion preliminary correction is matched again, a geometric transformation and an average inverse transformation matrix are calculated by a non-reflection similarity transformation, and a distortion final correction result of the last degenerated image of the sequence degenerated image is constructed;

[0090] S206, a strategy from back to front is adopted, step S205 is repeated, distortion correction is performed based on the non-saturated effective image information area block overlapped by the first image to the second last image in the image sequence after the distortion preliminary correction, a distortion final correction result of the second last degenerated image is constructed, and the distortion final correction result of all degenerated images in the sequence is obtained by circulation in turn.

[0091] S3, adopt the strategy from front to back, fuse the optimized weight image and the distortion final correction image, thereby correcting the severe thermal radiation effect saturation area, the specific process comprises:

[0092] S301, normalize the distortion final correction image sequence, calculate the local contrast and brightness features of each image, calculate the weight image sequence according to the local contrast and brightness features, use the guided filter to optimize, and obtain the optimized weight image sequence.

[0093] S302, the first and second images of the distortion final correction image sequence are set as the first and second images to be fused, the Gaussian filter and the down-sampling processing are used on the first and second images to be fused, the Gaussian pyramid of the image to be fused is obtained, then the adaptive edge perception guided filtering is carried out on the Gaussian pyramid of the image to be fused, the guided filtered image Gaussian pyramid is obtained, the Gaussian filter is used on the optimized weight image and the down-sampling processing is carried out, the Gaussian pyramid of the weight image is obtained, the corresponding scale images of the Gaussian pyramid of the image to be fused and the Gaussian pyramid of the weight image are multiplied pixel by pixel, the two Gaussian pyramids containing weight information are obtained, then the same scale images of the two Gaussian pyramids are added to form a new Gaussian pyramid, the new Gaussian pyramid is up-sampled and accumulated in turn from the coarsest scale to the finest scale, a new image is obtained, and the pixel value of the fused image is normalized to [0, 255], the first fused image is obtained, and meanwhile, the Gaussian pyramids of the two weight images are added in the same scale to obtain the Gaussian pyramid of the weight image corresponding to the fused image.

[0094] S303, according to the strategy from front to back, the next image of the distortion final correction image sequence is set as a new second image to be fused, step S302 is repeated until all images of the distortion final correction image sequence are fused into a correction image.

[0095] In some embodiments, the method in step S1 is specifically:

[0096] An image sequence containing different degrees of atmospheric turbulence distortion and severe thermal radiation degradation is obtained, each image in the image sequence is preliminarily processed for thermal radiation: each image in the degraded image sequence is processed by weighted least squares filtering; the filtered image is brought into a cubic spline surface equation, and a thermal radiation effect bias field is fitted; the original image is subtracted from the thermal radiation bias field to remove the thermal radiation effect, the thermal radiation effect is preliminarily removed, and a preprocessed image sequence is obtained.

[0097] In some embodiments, the method of step S201 is specifically:

[0098] The image sequence S is normalized, and the part with pixel value greater than 0.9 is regarded as a severe thermal radiation saturation region, and other regions are regarded as effective image information region blocks, corresponding to the division of the severe thermal radiation saturation region and the effective image information region block in the non-normalized image sequence S. All effective image information region blocks in the image pre-processing sequence S are extracted, the median pixel value of each pixel position in the effective information region block is calculated, and a median pixel value prior image is obtained I 0 (here, the median pixel value prior image refers to obtaining the median pixel of the same position pixel of a group of images, obtaining the median pixel of all positions, and forming a new image).

[0099] In some embodiments, the method of step S202 is specifically:

[0100] The SURF algorithm is used for feature point detection, in order to match the feature points between two image blocks, the Euclidean distance is used for matching, and when the Euclidean distance of two descriptors is less than a threshold value 0.7, the feature points of the two images are regarded as a group of feature point pairs.

[0101] Specifically, the effective image information region blocks are processed, the overlapping regions are extracted, and the median pixel value prior image I 0 and the effective image information region blocks extracted from each frame image in the pre-processing image sequence S are detected by using the SURF algorithm, the detection principle is to calculate the Hessian matrix of the pixel points on different Gaussian scales, and the local maximum point of the determinant is selected as the feature point. After the feature points are calculated, the SURF descriptor d needs to be calculated, d is a 64-dimensional vector formed by splicing the response values of 16 sub-regions. In order to match the feature points between two image blocks, the Euclidean distance is used for matching, and when the Euclidean distance of two descriptors is less than a threshold value 0.7, the feature points of the two images are regarded as a group of feature point pairs.

[0102] In some embodiments, the method of step S203 is specifically:

[0103] The parameterized transformation matrix is constructed based on the non-reflection similarity transformation model, the parameters are solved based on the feature point pairs, the transformation matrix is obtained, and the average inverse transformation matrix is calculated based on the transformation matrix obtained from each image, so as to realize the preliminary correction of the image distortion.

[0104] Specifically, according to the matched feature point pairs in step S202, in the non-saturated effective image information region blocks overlapped by the first image to the last image in the pre-processing image sequence S, the median pixel value prior image I0, constructing a parameterized transformation matrix based on the non-reflective similarity transformation model; solving the parameters based on the feature point pairs to obtain the transformation matrix; calculating the average inverse transformation matrix based on the transformation matrix obtained from each image; and obtaining the last image in the sequence S S n applying the average inverse transformation matrix to obtain the initial distortion correction result I n .

[0105] In some embodiments, the method of step S204 is specifically:

[0106] An image sequence is obtained by adopting a back-to-front strategy until all images in the image sequence are completed initial correction.

[0107] Specifically, the step S203 is repeated based on the first to the second-to-last image in the preprocessed image sequence S, i.e., the first to the 99th image, to perform distortion correction on the overlapped non-saturated effective image information area block, to construct the distortion correction result of the second-to-last image in the initial distortion correction sequence Y, i.e., the 99th image, and to sequentially cycle down until the initial distortion correction result of all the degraded images in the sequence is obtained, to obtain the complete initial distortion correction image sequence Y.

[0108] In some embodiments, the method of step S205 and step S206 is specifically:

[0109] According to the initial distortion correction result, the median pixel value prior image and the feature point pairs are updated, the final distortion correction of the image is realized, and the complete final distortion correction image sequence is composed.

[0110] Specifically, step S205 updates the median pixel value prior image according to the initial distortion correction result as I 0', and I 0' pairs the feature points again for the initial distortion correction image sequence Y, calculates the geometric transformation of the overlapped non-saturated area from the first image to the last image in the initial distortion correction image sequence Y through the non-reflective similarity transformation, applies the average inverse transformation matrix to the reference non-saturated area, and constructs the last image in the sequence Q that is the final distortion correction image I n ’.

[0111] Step S206 repeats step S205 based on the overlapped non-saturated effective image information area block of the first to the second-to-last image in the initial distortion correction image sequence to perform distortion correction, constructs the second-to-last final distortion correction image in the sequence Q, and sequentially cycles down until the final distortion correction result of all the degraded images in the sequence is obtained, to obtain the complete final distortion correction image sequence Q.

[0112] In some embodiments, the method of step S301 is specifically as follows:

[0113] The local contrast and brightness features of the images are calculated to generate a sequence of weight images.

[0114] Specifically, the sequence of distortion-corrected images Q is normalized, and the pixel values in all images are normalized from [0, 255] to [0, 1] to obtain a sequence of normalized images R; then the local contrast and brightness features of the images are calculated to generate a sequence of weight images E; and the generated weight images are guided filtered to obtain a sequence of optimized weight images W.

[0115] In some embodiments, the method of step S302 is specifically as follows:

[0116] The two images to be fused are subjected to Gaussian filtering and down-sampling to obtain Gaussian pyramids of the two images 、 Then, the images of different scales in the two Gaussian pyramids are subjected to adaptive edge-aware guided filtering to obtain two filtered Gaussian pyramids 、 The weight images corresponding to the images to be fused are subjected to Gaussian filtering and down-sampling to obtain a Gaussian pyramid of the weight images 、 The images of corresponding scales in the Gaussian pyramids of the images to be fused and the Gaussian pyramid of the weight images are multiplied pixel by pixel to obtain two Gaussian pyramids containing weight information 、 The images of the same scale in the two Gaussian pyramids are added to form a new Gaussian pyramid The new Gaussian pyramid is up-sampled and accumulated from the coarsest scale to the finest scale to obtain a new image F 1and the pixel values of the image are normalized to [0, 255] to obtain a first fused image I 1. At the same time, the Gaussian pyramids of the two weight images are added to obtain a Gaussian pyramid of the weight images corresponding to the fused image.

[0117] In some embodiments, the method of step S303 is specifically as follows:

[0118] According to the strategy from front to back, the obtained fused image I 1is set as the first image to be fused in a new round of fusion, and the third image of the sequence of distortion-corrected images Q is set as the second image to be fused in the new round of fusion, and fusion is performed to obtain a new fused image. This process is repeated until all images of the sequence Q are fused into a corrected image I f .

[0119] By implementing the method, image distortion and severe thermal radiation effects can be simply and efficiently removed, image signal-to-noise ratio is improved, and subsequent target recognition and detection operations are facilitated.

[0120] As shown in Figure 1 The sequence image distortion and severe thermal radiation effect fusion correction method of the embodiment of the application comprises the following steps:

[0121] S1, obtaining an image sequence containing different degrees of atmospheric turbulence distortion and severe thermal radiation degradation, and preliminarily performing thermal radiation processing on each image in the image sequence, and the specific process comprises:

[0122] S101, performing weighted least squares filtering on the degraded image sequence;

[0123] S102, inputting the filtered image into a cubic spline surface equation to fit a thermal radiation effect bias field;

[0124] S103, preliminarily removing the thermal radiation effect to obtain a preprocessed image sequence.

[0125] S2, performing distortion correction on the preprocessed image sequence based on a non-saturated effective information area block, and the specific process comprises:

[0126] S201, extracting a severe thermal radiation saturated area and an effective image information area block of each image in the image preprocessing sequence, calculating the median pixel value of each pixel position in the effective information area block, and obtaining a median pixel value prior image;

[0127] S202, using the SURF algorithm to perform local irregular area feature extraction and matching on the median pixel value prior image and the effective image information area block of each frame of image in the preprocessed image sequence, and saving the matched feature point pairs;

[0128] S203, according to the matched feature point pairs in step S202, in the non-saturated effective image information area block overlapping the first image to the last image in the preprocessed image sequence, referring to the median pixel value prior image, calculating the geometric transformation and average inverse transformation matrix through non-reflection similarity transformation, and constructing a distortion preliminary correction result of the last degraded image of the sequence degraded image sequence;

[0129] S204, adopting a back-to-front strategy, repeating step S203, performing distortion correction based on the non-saturated effective image information area block overlapping the first image to the second last image in the preprocessed image sequence, constructing a distortion correction result of the second last degraded image, and sequentially circulating until the distortion preliminary correction results of all the degraded images of the sequence are obtained;

[0130] S205, updating the median pixel value prior image according to the distortion preliminary correction result, referring to the updated median pixel value prior image, performing feature point pairing on the image sequence after the distortion preliminary correction again, calculating the geometric transformation and the average inverse transformation matrix through the non-reflective similarity transformation, and constructing the distortion final correction result of the last degraded image of the sequence degraded image;

[0131] S206, adopting a strategy from back to front, repeating step S205, performing distortion correction on the penultimate degraded image based on the non-saturated effective image information area block of the first to the penultimate image overlap in the image sequence after the distortion preliminary correction, constructing the distortion final correction result of the penultimate degraded image, and sequentially circulating until the distortion final correction result of all the degraded images of the sequence is obtained.

[0132] S3, adopting a strategy from front to back, fusing the optimized weight image and the distortion final correction image, so as to correct the severe thermal radiation effect saturated area, and the specific process comprises:

[0133] S301, normalizing the distortion final correction image sequence, calculating the local contrast and brightness features of each image, calculating the weight image sequence according to the local contrast and brightness features, and using a guided filter to optimize, so as to obtain the optimized weight image sequence;

[0134] S302, taking the first two images of the distortion final correction image sequence as the first and second to-be-fused images, using Gaussian filtering and down-sampling processing on the first and second to-be-fused images to obtain the Gaussian pyramid of the to-be-fused image, then performing adaptive edge-aware guided filtering on the Gaussian pyramid of the to-be-fused image at each scale to obtain the guided filtered image Gaussian pyramid, using Gaussian filtering and down-sampling processing on the optimized weight image to obtain the Gaussian pyramid of the weight image, multiplying the Gaussian pyramids of the to-be-fused image and the weight image at corresponding scales pixel by pixel to obtain two Gaussian pyramids containing weight information, and then adding the Gaussian pyramids at the same scale to form a new Gaussian pyramid; up-sampling and accumulating the new Gaussian pyramid from the coarsest scale to the finest scale in turn to obtain a new image, and normalizing the pixel value of the fused image to [0, 255] to obtain the first fused image; meanwhile, adding the Gaussian pyramids of the two weight images at the same scale to obtain the Gaussian pyramid of the weight image corresponding to the fused image;

[0135] S303, according to the strategy from front to back, setting the next image of the distortion final correction image sequence as a new second to-be-fused image, repeating step S302 until all the images of the distortion final correction image sequence are fused into a corrected image.

[0136] Further, the method in step S1 of the embodiment of the present application is specifically:

[0137] The image sequence with different degrees of atmospheric turbulence distortion and severe thermal radiation degradation is acquired, and each image in the image sequence is preliminarily processed for thermal radiation: the degraded image sequence is subjected to weighted least square filtering; the filtered image is brought into a cubic spline surface equation to fit a thermal radiation effect bias field; the thermal radiation effect is preliminarily removed to obtain a preprocessed image sequence.

[0138] The specific process is as follows:

[0139] For each degraded image in the degraded image sequence as shown in Figure 2 Z i , 1 < i < n, in the example, n = 100, weighted least square filtering is performed to obtain a filtered image G i , and the formula is as follows:

[0140] ;

[0141] , wherein, w ( x , y ) is a weight function, and the value of the weight function at the position ( x , y ) determines the importance of the position in the optimization process; is a regularization parameter, which controls the trade-off between the smoothing term and the data fidelity term; is a smoothing term, wherein G represents the gradient of the image G .

[0142] The filtered image is brought into a cubic spline surface equation to fit a thermal radiation effect bias field B i , and the formula is as follows:

[0143] ;

[0144] , wherein, is a regularization parameter, and S3 is a cubic spline function space, which ensures that the surface is continuous and smooth at the boundary.

[0145] The thermal radiation effect is removed to obtain a corrected image S i , and the formula is as follows:

[0146] .

[0147] Finally, a preprocessed image sequence S is obtained, as shown in Figure 3 .

[0148] ​Further, in step S2 of the embodiment of the present application, the method of step S201 is specifically as follows:

[0149] The image sequence S is normalized, and the part with pixel value greater than 0.9 is regarded as a severe thermal radiation saturation region, and other regions are regarded as effective image information region blocks. The severe thermal radiation saturation region and the effective image information region blocks in the normalized image sequence S are divided. All the effective image information region blocks in the image pre-processing sequence S are extracted, the median pixel value of each pixel position in the effective information region block is calculated, and a median pixel value prior image is obtained I 0 (here, the median pixel value prior image refers to a new image formed by calculating the median pixel value of the same position pixel of a group of images, and calculating the median pixel value of all positions), and the formula is as follows:

[0150] ;

[0151] In the formula, represents the median pixel value of the position (x, y). x , y Thus, the median pixel value prior image is obtained.

[0152] Further, in step S2 of the embodiment of the present application, the method of step S202 is specifically as follows:

[0153] The effective image information region blocks are processed, the overlapping regions are extracted, and the median pixel value prior image I 0 is matched with the overlapping effective image information region blocks extracted from each frame of image in the pre-processing image sequence S by using the SURF algorithm for feature point detection. The detection principle is that the Hessian matrix of the pixel points is calculated on different Gaussian scales, and the local maximum points of the determinant are selected as the feature points. After the feature points are calculated, the SURF descriptor d needs to be further calculated, and d is a 64-dimensional vector formed by splicing the response values of 16 sub-regions. In order to match the feature points between two image blocks, the Euclidean distance is used for matching, and when the Euclidean distance of two descriptors is less than a threshold value 0.7, the feature points of the two images are regarded as a group of feature point pairs.

[0154] The specific process is as follows:

[0155] Feature point detection is performed:

[0156] ;

[0157] In the formula, represents the SURF feature point detection, represents the feature points of the median pixel value prior image I 0, represents the feature points of the pre-processing image sequence S, and i represents the m-th image in the pre-processing image sequence SS i characteristic points.

[0158] generate descriptors:

[0159] ;

[0160] wherein, SURF descriptor generation, k and l is the characteristic point number.

[0161] characteristic point matching:

[0162] ;

[0163] wherein, Euclidean distance, and M is the characteristic point number pair set.

[0164] output the characteristic point pair:

[0165] ;

[0166] wherein, is the characteristic point pair set.

[0167] thus obtaining the median pixel value prior image I 0 and the characteristic point pairs extracted from the overlapped effective image information region block of each frame image in the preprocessed image sequence S.

[0168] Further, in step S2 of the embodiment of the present application, the method of step S203 is specifically:

[0169] According to the matched characteristic point pairs in step S202, in the overlapped non-saturated effective image information region block of the first image to the last image in the preprocessed image sequence S, the reference median pixel value prior image I 0 is used to construct a parameterized transformation matrix based on the non-reflective similarity transformation model; the parameters are solved based on the characteristic point pairs to obtain the transformation matrix; the average inverse transformation matrix is calculated based on the transformation matrix obtained for each image; and the last image S n apply the average inverse transformation matrix to obtain the initial distortion correction result I n .

[0170] The specific process is as follows:

[0171] The non-reflective similarity transformation is a special affine transformation, which can be represented as a combination of rotation, translation and uniform scaling for any point (x, y) in the effective image information region block, and its formula is:

[0172] ;

[0173] in,( () are the transformed coordinates. It is the rotation angle, and s is the scaling factor. ) is a translation vector. The transformation matrix is ​​represented as:

[0174] ;

[0175] in , This ensures that the scaling factor is non-negative.

[0176] The transformation matrix is ​​obtained by solving for the parameters based on feature point pairs. Assume that from... Si and I Matching N pairs of feature points in 0 The goal is to find the optimal one. This makes the transformed point as close as possible to the target point:

[0177] .

[0178] For each pair of feature points, write two equations:

[0179] ;

[0180] Stack the equations for all point pairs into a matrix form:

[0181] ;

[0182] Solving parameters using the least squares method :

[0183] .

[0184] From the optimized parameter matrix Solve the transformation matrix :

[0185] ;

[0186] in , Encoding rotation and scaling, , .

[0187] Based on the transformation matrix obtained for each image, calculate the average inverse transformation matrix. :

[0188] ;

[0189] wherein is an inverse matrix of S i to I 0. The average operation can suppress single image matching errors and improve robustness.

[0190] The last image of the image sequence S is S n The average inverse transformation matrix is applied to obtain the initial distortion correction result of the last image I n :

[0191] ;

[0192] Thus, the initial distortion correction image of the last image is obtained.

[0193] Further, in step S2 of the embodiment of the present application, the method of step S204 is specifically as follows:

[0194] The first to the second last image, i.e., the first to the 99th image, of the preprocessed image sequence S is repeatedly subjected to the distortion correction in step S203, and the distortion correction result of the second last image, i.e., the 99th image, of the initial distortion correction sequence Y is constructed, and the process is sequentially repeated until the distortion correction results of all the images of the sequence are obtained and are superimposed to form the complete initial distortion correction image sequence Y, as shown in Figure 4 .

[0195] Further, in step S2 of the embodiment of the present application, the method of step S205 is specifically as follows:

[0196] The median pixel value prior image is updated according to the initial distortion correction result to be I 0', and the image sequence Y after the initial distortion correction is subjected to feature point pairing again with reference to I 0', the geometric transformation of the non-saturated region of the first image to the last image of the initial distortion correction image sequence Y is calculated through the non-reflection similarity transformation, the average inverse transformation matrix is applied to the last image, a new empty sequence Q is created for storing the final distortion correction image, and the last image I n ' of the sequence Q after the final distortion correction is constructed.

[0197] Further, in step S2 of the embodiment of the present application, the method of step S206 is specifically as follows:

[0198] With the strategy from back to front, repeat step S205, based on the first to the second last image in the sequence of distortion initial correction of the image overlap non-saturated effective image information area block, the second last image distortion correction, the construction sequence Q in the second last image distortion final correction after the image, in turn, down until the distortion final correction result of all degradation images in the sequence is obtained, the complete distortion final correction image sequence Q, as shown in Figure 5 .

[0199] Further, in step S3 of the embodiment of the application, the method of step S301 is specifically:

[0200] The distortion final correction image sequence Q is normalized, and the pixel values in all images are normalized from [0, 255] to [0, 1], and the normalized image sequence R is obtained after processing; then the local contrast and brightness features of the image are calculated, thereby generating a weight map sequence E; the generated weight map is guided filtering optimized to obtain an optimized weight image sequence W.

[0201] As shown in Figure 8 , the specific process is as follows:

[0202] The images in the image sequence Q are normalized Q i to obtain the images in the normalized image sequence R R i :

[0203] .

[0204] The local contrast and brightness features are calculated to obtain a weight map. In this embodiment, the SIFT operator is used to calculate the local contrast of the image , and the formula is as follows:

[0205] .

[0206] The contrast weight term is calculated :

[0207] .

[0208] The brightness feature weight term is calculated , and here, the pixels with pixel value range in [0.1, 0.9] in the normalized image are considered as well-exposed pixels, and the brightness feature weight term is set to 1, and the formula is as follows:

[0209] .

[0210] When taking pictures with a traditional camera, some areas appear dark in the image, i.e. these areas are underexposed, and some areas appear bright; i.e. overexposed. The basic idea of the brightness feature is to avoid these areas and only take the areas from the well-exposed source image. The well-exposed range in this method is 0.1 to 0.9. Any pixel value in the range of 0.1 to 0.9 is considered as a well-exposed pixel, and for these pixels, the brightness feature is set to 1.

[0211] According to the calculated contrast weight term and the brightness feature weight term , the weight image is calculated To make the sum of the weight map at each pixel equal to 1, the weight map needs to be normalized to , which is calculated as follows:

[0212] ;

[0213] ;

[0214] wherein is a very small positive number to prevent the denominator from being zero, for example 10 -25 .

[0215] The generated weight map is guided filtered to obtain the optimized weight image W i :

[0216] ;

[0217] wherein is the size of the local window Ω, i.e. the total number of pixels in the window; is the variance of within the local window, which measures the degree of variation of pixel values within the window; is a small regularization constant to avoid the denominator being zero and to control the degree of smoothing; is the mean value of within the local window; is the mean value of the local window, which is the same as here, and the optimized weight image sequence W can be calculated according to the above.

[0218] Further, in step S3 of the embodiment of the present application, the method of step S302 is specifically:

[0219] The two images to be fused are Gaussian filtered and down-sampled to obtain the Gaussian pyramids of the two images , Then, adaptive edge-aware guided filtering is performed on the images of different scales of the two Gaussian pyramids to obtain two filtered Gaussian pyramids 、 The weight image corresponding to the image to be fused is subjected to Gaussian filtering and down-sampling to obtain a Gaussian pyramid of the weight image 、 The corresponding scale images of the Gaussian pyramid of the image to be fused and the Gaussian pyramid of the weight image are multiplied pixel by pixel to obtain two Gaussian pyramids containing weight information 、 The same scale images of the two Gaussian pyramids are added to form a new Gaussian pyramid The new Gaussian pyramid is successively up-sampled and accumulated to fuse a new image from the coarsest scale to the finest scale F 1and the pixel value of the image is normalized to [0, 255] to obtain a first fused image I 1.

[0220] The specific process is as follows:

[0221] The two images to be fused are subjected to Q 1、 Q 2 Gaussian filtering and down-sampling to construct a Gaussian pyramid 、 , l =0,1,…, L denote the number of pyramid layers (n l =0 is the finest scale, l = L is the coarsest scale):

[0222] ;

[0223] wherein, m 、 n are relative coordinates in the kernel; k is a Gaussian kernel radius; is a Gaussian kernel weight, calculated by ; is a Gaussian distribution standard deviation, controlling the smoothing strength;2 x 、2 y controls the down-sampling step (the size of each layer is halved).

[0224] Subsequently, adaptive edge-aware guided filtering is performed on each layer image of the two Gaussian pyramids to obtain filtered pyramid images 、 :

[0225] ;

[0226] wherein,r Represents the local window radius. For regularization parameters, The input filtered image, Indicates guided filtering .

[0227] The corresponding weight images of the images to be fused W 1 、W 2. Perform Gaussian filtering and downsampling to obtain the Gaussian pyramid of the weighted image. , :

[0228] ;

[0229] in, m , n Relative coordinates within the kernel; k The radius of the Gaussian kernel; The Gaussian kernel weights are determined by... calculate; The standard deviation of the Gaussian distribution is used to control the smoothing intensity; 2 x 2 y Control the downsampling step (halve the size of each layer).

[0230] The filtered pyramid image , Gaussian pyramids of weighted images , By multiplying the corresponding scale images pixel by pixel, two Gaussian pyramids containing weight information are obtained. , Then, the two Gaussian pyramid images of the same scale are added together to form a new Gaussian pyramid. :

[0231] .

[0232] The new Gauss Pyramid From the coarsest scale l =L to the finest scale l =0, sequentially upsample and accumulate The images are merged to obtain a new image. F 1. First, perform initialization (coarsest scale):

[0233] .

[0234] Then, upsampling and accumulation are performed layer by layer:

[0235] ;

[0236] Wherein, Up() is an up-sampling operation (such as bilinear interpolation), which expands the image size by 2 times.

[0237] The final fused image is obtained as follows:

[0238] .

[0239] The pixel value of the fused image F 1 is normalized to [0, 255] to obtain I 1, which is the first fused image.

[0240] Further, in step S3 of the embodiment of the present application, the method of step S303 is specifically as follows:

[0241] According to the front-to-back strategy (fusion from front to back, which follows the original order of image acquisition, making the fusion process more in line with the logic of data generation), the obtained fused image I 1 is set as the first image to be fused in a new round of fusion, and the third image of the sequence Q of the final corrected images is set as the second image to be fused in the new round of fusion, and fusion is performed to obtain a new fused image. Repeat this process until all images of the sequence Q are fused into one corrected image I f As shown in Figure 7 , the iterative fusion process is as follows:

[0242] ;

[0243] Wherein, I is the fused image, and Fuse represents the image fusion process, I f is the final corrected image.

[0244] Figure 6 A clear image provided by the embodiment of the present application can be seen, and the information in the corrected image obtained by the method of the present application can be clearly seen. Figure 6

[0245] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0246] The sequence image distortion and severe thermal radiation effect fusion correction method of the embodiment of the present application described in combination Figure 1 with the accompanying drawings can be realized by a computer device. Figure 9 The hardware structure schematic diagram of the computer device of the embodiment of the present application is shown in Figure 9 ​As shown, the device can include a processor 301 and a memory 302 having stored computer program instructions.

[0247] In particular, the processor 301 can include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0248] The memory 302 can include a mass storage for data or instructions. By way of example, and without limitation, the memory 302 can include a Hard Disk Drive (HDD), a floppy disk drive, a Solid State Drive (SSD), a flash drive, a Compact Disc Read Only Memory (CD-ROM), a Digital Versatile Disk (DVD), a Blu-Ray, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 302 can be removable and / or non-removable (or fixed) as appropriate. The memory 302 can be internal or external as appropriate. In certain embodiments, the memory 302 is a Non-Volatile Memory. In certain embodiments, the memory 302 includes a Read-Only Memory (ROM) and a Random-Access Memory (RAM). The ROM can be a Mask-Programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended Data Output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0249] The memory 302 can be used to store or buffer various data files required for processing and / or communication, and possible computer program instructions executed by the processor 301.

[0250] The processor 301 reads and executes the computer program instructions stored in the memory 302 to implement any one of the sequence image distortion and severe thermal radiation effect fusion correction methods in the above embodiments.

[0251] In some embodiments, the computer device can further include a communication interface 303 and a bus 300. As shown, the processor 301, the memory 302, and the communication interface 303 are connected through the bus 300 and complete communication with each other. Figure 9

[0252] The communication interface 303 is used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application. The communication interface 303 can also realize data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations, etc.

[0253] ​Bus 300 includes hardware, software, or both, to couple components of the computer device to each other and to couple components to other components in the environment. Bus 300 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, a local bus, etc. By way of example and not limitation, bus 300 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or another suitable bus or combination of two or more of these. Where appropriate, bus 300 can include one or more buses. Although the present embodiments describe and show a particular bus, the present embodiments contemplate any suitable bus or interconnect.

[0254] The computer device can execute the sequence image distortion and severe thermal radiation effect fusion correction method in the embodiments of the present application, thereby realizing the combination of Figure 1 The sequence image distortion and severe thermal radiation effect fusion correction method is described.

[0255] In addition, in combination with the sequence image distortion and severe thermal radiation effect fusion correction method in the above embodiments, the embodiments of the present application can provide a computer readable storage medium for implementation. The computer readable storage medium stores computer program instructions; the computer program instructions are executed by a processor to implement any one of the sequence image distortion and severe thermal radiation effect fusion correction methods in the above embodiments.

[0256] It should be noted that the technical features of the above embodiments can be combined in any manner. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present disclosure. In addition, according to the needs of implementation, each step / component described in the present application can be split into more steps / components, or two or more steps / components or parts of the operation of the steps / components can be combined into a new step / component, to achieve the purpose of the present application.

[0257] Those skilled in the art will readily understand that the above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application patent should be subject to the appended claims.

Claims

1. A method for fusing and correcting sequence image distortion and severe thermal radiation effects, characterized in that, The method comprises: S1, obtaining an image sequence containing different degrees of atmospheric turbulence distortion and severe thermal radiation degradation, and performing preliminary thermal radiation processing on each image in the image sequence to obtain a pre-processed image sequence; S2, performing distortion correction on the pre-processed image sequence to obtain a distortion-corrected image sequence, specifically comprising: S201, extracting a non-saturated effective image information region block of a severe thermal radiation saturated region of each image in the pre-processed image sequence, calculating the median pixel value of each pixel position in the non-saturated effective information region block of the pre-processed image sequence, and obtaining a median pixel value prior image; S202, determining an overlapping region of the non-saturated effective image information region block of the first image to the last image in the pre-processed image sequence, obtaining an overlapping non-saturated effective image information region block of the first image to the last image in the pre-processed image sequence, and extracting a feature point pair of the median pixel value prior image and the overlapping non-saturated effective image information region block of the first image to the last image in the pre-processed image sequence; S203, determining an inverse transformation matrix of the first image to the last image in the pre-processed image sequence to the median pixel value prior image according to the feature point pair of the median pixel value prior image and the overlapping non-saturated effective image information region block of the first image to the last image in the pre-processed image sequence, and calculating an average inverse transformation matrix, and performing distortion correction on the last image in the pre-processed image sequence based on the average inverse transformation matrix; S204, determining an overlapping region of the non-saturated effective image information region block of the first image to the second last image in the pre-processed image sequence, and performing distortion correction on the second last image according to the same distortion correction logic; continue to use this strategy from back to front, and sequentially complete the distortion correction of the third last image to the first image, and finally obtain the distortion-corrected image sequence; S3, performing normalization processing on the distortion-corrected image sequence, generating a weight image sequence by calculating local contrast features and brightness features, and adopting a strategy from front to back to fuse the distortion-corrected image sequence and the weight image sequence into a corrected image, specifically comprising: S301, performing normalization processing on the distortion-corrected image sequence, and calculating local contrast features and brightness features of each image, calculating a contrast weight term of each image based on the local contrast features of all images, taking the brightness features of each image as a brightness weight term, and generating a weight image sequence according to the contrast weight term and the brightness weight term of each image; S302, Gaussian filtering and down-sampling the first image and the second image of the distortion corrected image sequence to obtain the Gaussian pyramids of the two images, Gaussian filtering and down-sampling the weight images corresponding to the two images to obtain the Gaussian pyramids of the two weight images; pixel-by-pixel multiplication of the Gaussian pyramids of the two images and the Gaussian pyramids of the corresponding weight images to obtain two Gaussian pyramids containing weight information, same-scale addition of the two Gaussian pyramids containing weight information to form a new Gaussian pyramid, and restoration of the new Gaussian pyramid to the original image to obtain the first fused image; meanwhile, same-scale addition of the Gaussian pyramids of the two weight images to obtain the Gaussian pyramid of the weight image corresponding to the fused image; S303, taking the current fused image and the next image of the distortion corrected image sequence as the two images to be fused, and performing fusion according to the same fusion logic to obtain the next fused image and the Gaussian pyramid of the weight image corresponding thereto; continue to use this strategy from the front to the back until all the images of the distortion corrected image sequence are fused into one corrected image.

2. The method according to claim 1, wherein the method further comprises: Step S1 specifically comprises: S101, performing weighted least square filtering on each image in the image sequence; S102, fitting a thermal radiation effect bias field of each image by bringing the filtered image into a cubic spline surface equation; S103, subtracting the thermal radiation effect bias field of each image from the image to obtain a pre-processed image sequence.

3. The method according to claim 1, wherein the method further comprises: determining a distortion of each of the sequence images; and correcting the distortion of each of the sequence images. In step S201, a heavy thermal radiation saturated area and a non-saturated effective image information area block of each image in the pre-processed image sequence are extracted, comprising: normalizing the pre-processed image sequence to obtain a normalized image sequence; regarding an area with a pixel gray value greater than a preset gray threshold in each image in the normalized image sequence as a heavy thermal radiation saturated area, regarding other areas as non-saturated effective image information area blocks, and extracting the non-saturated effective image information area blocks of each image from the pre-processed image sequence.

4. The method according to claim 3, wherein the method further comprises: Step S2 specifically further comprises: S205, updating the median pixel value prior image according to the distortion corrected image sequence; S206, performing distortion correction again on the distortion corrected image sequence based on the updated median pixel value prior image to obtain a final distortion corrected image sequence.

5. The method according to claim 1, wherein Step S301 specifically comprises: performing normalization processing on each image in the distortion corrected image sequence Q to obtain a normalized distortion corrected image sequence R; Compute the i-th image of the sequence of normalized distortion corrected images R R i local contrast feature of the i-th image of the sequence of normalized distortion corrected images R, as follows: ； wherein x and y are pixel coordinates; is a local contrast feature; SIFT is a SIFT operator; is an L1 norm; calculating a contrast weight term for the i-th image based on local contrast features of all images : ； wherein n is the total number of images in the image sequence; calculating a luminance feature of the i-th image as a luminance weight term : ； In the formula, Y is a luminance parameter; calculating a weight image of the i-th image according to the contrast weight term and the brightness weight term of the i-th image : ； normalizing the weight image is performed: ； wherein is a very small positive number to prevent the denominator from being zero; is the normalized weight image, thus obtaining the sequence of weight images E.

6. The method according to claim 5, wherein the method further comprises: Step S301 specifically further comprises: performing guided filtering optimization on the weight image sequence to obtain an optimized weight image sequence.

7. The method according to claim 1, wherein the method further comprises: determining a distortion of each of the sequence images; and correcting the distortion of each of the sequence images. Step S302 specifically comprises: correcting a first image of a sequence of distorted images Q Q 1 and a second image Q 2 are Gaussian filtered and downsampled to build a Gaussian pyramid of the two images and : ； wherein m , n is the relative coordinate in the kernel; k is the Gaussian kernel radius; is the Gaussian kernel weight, wherein is the standard deviation of the Gaussian distribution; x and y is the pixel coordinate, is the down-sampling factor; l =0,1,… L denotes the number of pyramid layers, L is the maximum number of layers, denotes the l layer image, the 0th layer image is the original image Q; Each image of the Gaussian pyramid of the two images is adaptively edge-aware guided filtered to obtain a filtered pyramid image and : ； wherein is the filtered pyramid image; r represents a local window radius, is a regularization parameter, is the input image; denotes a guided filtering; Similarly, the weight images corresponding to the two images W 1 and W 2 are Gaussian filtered and down-sampled to obtain a Gaussian pyramid of two weight images and ; The two filtered Gaussian pyramids of images and the corresponding Gaussian pyramids of weight images are multiplied pixel by pixel to obtain two Gaussian pyramids containing weight information and The two Gaussian pyramids containing weight information are added in the same scale to form a new Gaussian pyramid : ； A new Gaussian pyramid is created From the coarsest scale l = L to the finest scale l =0 is up-sampled and accumulated in turn, a new image is obtained and the image pixel value is normalized to [0, 255], and the first fusion image is obtained; Meanwhile, same-scale addition of the Gaussian pyramids of the two weight images to obtain the Gaussian pyramid of the weight image corresponding to the fused image.

8. A computer device, comprising: comprising: a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the sequence image distortion and heavy thermal radiation effect fusion correction method in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, A computer program product, comprising a computer readable storage medium having stored thereon computer readable program code, the computer readable program code comprising program code implementing the steps of the sequence image distortion and severe thermal radiation effect fusion correction method according to any one of claims 1 to 7 when executed by a processor. A computer program product, comprising a computer readable storage medium having stored thereon computer readable program code, the computer readable program code comprising program code implementing the steps of the sequence image distortion and severe thermal radiation effect fusion correction method according to any one of claims 1 to 7 when executed by a processor.

Images