Local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation

Abstract

The present invention provides a local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation, comprising the following steps: Step 1: Inputting the original infrared image into a multi-order infrared image degradation model to obtain a low-resolution degraded image; Step 2: Inputting the low-resolution degraded image into the generator of a super-resolution reconstruction network to generate a super-resolution reconstructed image; Step 3: After feature fusion of the super-resolution reconstructed image and the original infrared image, the image is input into the local discriminator of the super-resolution reconstruction network for processing and obtaining a local discriminant loss; Step 4: Taking a weighted sum of the local discriminant loss and a preset model loss to obtain an image reconstruction training loss; When the image reconstruction training loss is greater than the minimum loss value, the method returns to Step 2; Otherwise, the trained super-resolution reconstruction network is output. The present invention can more meticulously capture the deviation of local details, supervise the generator to better optimize local details, and improve the accuracy of texture detail restoration in complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation. Background Art

[0002] In the context of today's technological development, infrared images have a wide range of applications in military, security, medical, and industrial fields. However, due to hardware limitations and cost considerations, the resolution of infrared cameras is often relatively low. This leads to the loss and blurring of detailed information in infrared images, limiting their accuracy and reliability in tasks such as target recognition, target tracking, and anomaly detection. To address this problem, super-resolution reconstruction technology has become a popular research direction. However, existing super-resolution reconstruction methods based on deep learning achieve super-resolution reconstruction by constructing a deep neural network, allowing the model to learn the end-to-end mapping relationship from low-resolution infrared images to high-resolution infrared images, and automatically extracting image features. However, there are still some problems: super-resolution reconstruction algorithms may introduce blurring or artifact effects, causing the image to lose its realism. This is because the algorithm cannot accurately restore missing detailed information or over-smoothes the image. Therefore, the present invention proposes a local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation. Summary of the Invention

[0003] The present invention provides a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation to solve the above problems.

[0004] The present invention provides a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation, comprising:

[0005] Step 1: Input the original infrared image into the multi-order infrared image degradation model to obtain a low-resolution degraded image;

[0006] Step 2: Input the low-resolution degraded image into the generator of the super-resolution reconstruction network to generate a super-resolution reconstructed image;

[0007] Step 3: After feature fusion of the super-resolution reconstructed image and the original infrared image, the image is input into the local discriminator of the super-resolution reconstruction network for processing, and the local discriminant loss is obtained;

[0008] Step 4: Perform weighted summation of the local discriminant loss and the preset model loss to obtain the image reconstruction training loss of the super-resolution reconstructed image;

[0009] When the image reconstruction training loss is greater than the minimum loss value, return to step 2;

[0010] Otherwise, the training is terminated and the trained super-resolution reconstruction network is output.

[0011] Preferably, in a local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation, step 1 includes:

[0012] When the original infrared image is input into the multi-order infrared image degradation model, the multi-order infrared image degradation model randomly arranges multiple image degradation strategies based on the image degradation strategy database to generate a multi-order infrared image degradation process;

[0013] Based on the multi-stage infrared image degradation process, the original infrared image is processed to obtain a low-resolution degraded image;

[0014] Wherein, the multi-order infrared image degradation model is a third-order degradation model.

[0015] Preferably, in a local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation, step 2 includes:

[0016] After the low-resolution infrared image is input after the first convolution processing, it is processed based on N RRDB modules to obtain the first output image;

[0017] After performing a second convolution process on the first output image, upsampling is performed to obtain an upsampled image;

[0018] Based on the residual dense network, the image features extracted by each network layer of the upsampled image and the low-resolution infrared image are obtained respectively, and the upsampled image features and the original image features of the low-resolution infrared image are obtained;

[0019] The upsampled image features and the original image features are convolutionally fused to obtain a super-resolution reconstructed image.

[0020] Preferably, in a local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation, step 3 includes:

[0021] Perform non-overlapping sliding on the super-resolution reconstructed image based on a preset rectangular window, calculate the local entropy corresponding to the preset window, and determine whether the local entropy is greater than a local entropy threshold;

[0022] If yes, adaptively adjust the rectangular window based on the error between the local entropy and the local entropy threshold, and compare and calculate the pixel data in the adjusted rectangular window with the pixel data corresponding to the original infrared image to obtain a local residual map;

[0023] Otherwise, the pixel data within the preset rectangular window is obtained and compared with the pixel data corresponding to the original infrared image to obtain a local residual map;

[0024] Based on the local residual map, calculating the local variance of each pixel point in the adjusted rectangular window or the preset rectangular frame, and when the local variance is greater than the preset variance, determining that the current pixel point is an artifact pixel point;

[0025] Otherwise, the current pixel is determined to be a true shadow pixel;

[0026] Based on the local variance corresponding to all artifact pixels in the super-resolution reconstructed image, the local discriminant loss is calculated.

[0027] Preferably, in a local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation, adaptively adjusting the rectangular window based on the error value between the local entropy and the local entropy threshold includes:

[0028] Based on big data technology, the window information entropy corresponding to the infrared images of rectangular windows of different sizes is collected;

[0029] Grouping the window information entropy based on the window size to obtain rectangular window data groups of multiple sizes, determining the window information entropy range corresponding to each rectangular window data group, and determining whether there is an overlapping range between the window information entropy ranges corresponding to rectangular windows of different sizes;

[0030] If it does not exist, the window information entropy range corresponding to the current rectangular windows of different sizes is used as the final window information entropy range;

[0031] If so, obtaining rectangular window data groups corresponding to the overlapping rectangular windows as the first target data group and the second target data group, respectively, clustering the data in the first target data group and the second target data group to obtain the first cluster center and the second cluster center;

[0032] Based on the first cluster center and the second cluster center, determine the target window information entropy value corresponding to the distance center point, and judge whether the target window information entropy value is within the overlapping range. If so, based on the target window information entropy value, respectively correct the upper limit value of the window information entropy range corresponding to the first target data group and the lower limit value of the window information entropy range corresponding to the second target data group;

[0033] Otherwise, when the target window information entropy value is less than or equal to the lower limit value of the overlapping range, the lower limit value of the window information entropy range corresponding to the second target data group is modified based on the lower limit value of the overlapping range;

[0034] When the target window information entropy value is greater than or equal to the upper limit value of the overlapping range, the lower limit value of the window information entropy range corresponding to the first target data group is modified based on the upper limit value of the overlapping range;

[0035] Based on the correction results, the final window information entropy range corresponding to rectangular windows of different sizes is determined;

[0036] According to the final window information entropy range corresponding to rectangular windows of different sizes, the window information entropy difference interval between rectangular windows of different sizes is determined;

[0037] The error value between the local entropy and the local entropy threshold is matched with the window information entropy difference interval between multiple sized rectangular windows and the preset rectangular window to determine the size corresponding to the optimal rectangular window, and the window size of the preset rectangular window is adaptively adjusted according to the size corresponding to the optimal rectangular window.

[0038] Preferably, in a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation, the local discrimination loss is calculated based on the local variance corresponding to all artifact pixels in the super-resolution reconstructed image, including:

[0039] The standard local discrimination loss corresponding to the super-resolution reconstructed image is calculated based on the preset local discrimination loss model, and the artifact pixels in the super-resolution reconstructed image are marked. Based on the marking results, the distribution characteristics of the artifact pixels are extracted;

[0040] Based on the distribution characteristics, each image element in the super-resolution reconstructed image is evaluated, and according to the evaluation result, the image distortion rate of the super-resolution reconstructed image is determined;

[0041] evaluating a global reconstruction error of the super-resolution reconstructed image based on the image distortion rate; obtaining a global reconstruction error index based on an error ratio between the reconstruction error and the preset error when the reconstruction error is greater than a preset error; and correcting a standard local discrimination error based on the global reconstruction error index to obtain an output local discrimination error;

[0042] Otherwise, the standard local discriminant error is used as the output local discriminant error.

[0043] Preferably, in a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation, each image element in the super-resolution reconstructed image is evaluated based on distribution characteristics, including:

[0044] Perform edge detection on the super-resolution reconstructed image, and based on the edge detection results, segment the image information on the super-resolution reconstructed image into multiple image element information;

[0045] Based on the distribution characteristics of artifact pixels and the image element information segmentation results, the correlation relationship between the image representation information of each artifact image on the valid image is determined;

[0046] Based on the association relationship, determining the number of artifact pixels and position distribution sub-features on each image element;

[0047] Obtaining the total number of pixels corresponding to each image element having an artifact pixel, and obtaining an artifact index of the corresponding image element based on the total number of pixels and the number of artifact pixels of the corresponding image element;

[0048] When the artifact index is greater than or equal to a preset threshold, determining that the corresponding image element is distorted;

[0049] Otherwise, based on the position distribution sub-features corresponding to the artifact pixels on the corresponding image element, determine whether there is an artifact pixel concentration area on the corresponding image element. If so, obtain the element distortion ratio based on the ratio of the number of artifact pixels in the artifact pixel concentration area to the total number of pixels corresponding to the corresponding image element.

[0050] Determining a maximum artifact pixel concentration area based on the element distortion ratio, and determining that the corresponding image element is distorted when the element distortion ratio corresponding to the maximum artifact pixel concentration area is greater than or equal to a threshold distortion ratio;

[0051] Otherwise, it is determined that the corresponding image element is not distorted.

[0052] Preferably, in a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation, determining the image distortion rate of the super-resolution reconstructed image according to the evaluation result includes:

[0053] Traversing the image elements on the super-resolution reconstructed image to determine the total number of image elements and the number of distorted image elements corresponding to the super-resolution reconstructed image;

[0054] Based on the total number of image elements corresponding to the super-resolution reconstructed image and the number of distorted image elements, a distortion element ratio of the super-resolution reconstructed image is obtained, and the distortion element ratio is used as the image distortion rate of the super-resolution reconstructed image.

[0055] Preferably, the local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation further includes:

[0056] The user inputs the low-resolution infrared image to be processed into the trained super-resolution reconstruction network for processing to obtain a super-resolution reconstructed infrared image;

[0057] The super-resolution reconstructed infrared image is sent to a corresponding user terminal for display based on a preset sending address.

[0058] Preferably, in a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation, the method further includes:

[0059] Based on big data, a large number of high-resolution infrared images are obtained as original infrared images to establish a training database, and the output super-resolution reconstructed images corresponding to the trained super-resolution reconstruction network are collected to update the training database;

[0060] When the data update amount of the training database reaches a minimum update threshold, a plurality of original infrared images are randomly sampled based on the latest training database, and steps 1-4 are executed.

[0061] Other features and advantages of the present invention will be described in the following description, and in part will be apparent from the description or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in this application document.

[0062] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0064] Figure 1 This is a flow chart of a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation according to the present invention;

[0065] Figure 2 This is a flowchart of step 1 of a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation according to the present invention;

[0066] Figure 3 This is a flowchart of step 2 of a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation of the present invention. DETAILED DESCRIPTION

[0067] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0068] Example 1:

[0069] The present invention provides a local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation, such as Figure 1 As shown, including:

[0070] Step 1: Input the original infrared image into the multi-order infrared image degradation model to obtain a low-resolution degraded image;

[0071] Step 2: Input the low-resolution degraded image into the generator of the super-resolution reconstruction network to generate a super-resolution reconstructed image;

[0072] Step 3: After feature fusion of the super-resolution reconstructed image and the original infrared image, the image is input into the local discriminator of the super-resolution reconstruction network for processing, and the local discriminant loss is obtained;

[0073] Step 4: Perform weighted summation of the local discriminant loss and the preset model loss to obtain the image reconstruction training loss of the super-resolution reconstructed image;

[0074] When the image reconstruction training loss is greater than the minimum loss value, return to step 2;

[0075] Otherwise, the training is terminated and the trained super-resolution reconstruction network is output.

[0076] In this embodiment, the preset model loss indicates pixel-by-pixel reconstruction loss, perceptual loss, and adversarial loss, which are obtained by multiplying and summing their corresponding balance parameters. The corresponding balance parameters of the three are usually set to 0.01, 1, and 0.005, respectively.

[0077] The beneficial effects of the above technical solution: the present invention generates a low-resolution degraded image through a multi-order infrared image degradation model, which can highly simulate various degradation conditions that infrared images may encounter during actual acquisition and transmission, such as noise interference, downsampling blur, etc. The degradation simulation makes the subsequent training and reconstruction of the super-resolution reconstruction network more targeted. Compared with directly processing the original clear image, the reconstructed super-resolution image can better meet the demand for image quality improvement in actual application scenarios. The low-resolution degraded image is then input into the generator of the super-resolution reconstruction network to generate a super-resolution reconstructed image. The super-resolution reconstructed image and the original infrared image are feature fused and input into the local discriminator of the super-resolution reconstruction network for processing to obtain the local discrimination loss of the super-resolution reconstructed image, and the local discrimination loss is analyzed. The weighted summation of the loss and the preset model loss is performed to obtain the image reconstruction training loss of the super-resolution reconstructed image, and then the image reconstruction loss is used to judge whether the super-resolution reconstruction network is trained. It can combine the enhanced details in the reconstructed image with the inherent feature information of the original image, so that the subsequent discriminator can more accurately judge the local quality of the reconstructed image, and use the local discriminator to process the fused features, focusing on evaluating the authenticity and accuracy of the local area of ​​the reconstructed image, and can capture the deviation of local details more carefully. The addition of local discrepancy loss enriches the training loss content of the overall model, which can better supervise the generator to better optimize local details, improve the accuracy of texture detail recovery in complex scenes, improve the overall visual quality of the reconstructed image, and obtain a super-resolution image that is closest to the actual display content of the infrared image.

[0078] Example 2:

[0079] On the basis of Example 1, step 1, as Figure 2 As shown, including:

[0080] Step 101: while inputting the original infrared image into the multi-order infrared image degradation model, the multi-order infrared image degradation model randomly arranges multiple image degradation strategies based on the image degradation strategy database to generate a multi-order infrared image degradation process;

[0081] Step 102: Based on the multi-stage infrared image degradation process, the original infrared image is processed to obtain a low-resolution degraded image;

[0082] Wherein, the multi-order infrared image degradation model is a third-order degradation model.

[0083] In this embodiment, the image degradation strategy database includes but is not limited to noise interference (for example, Gaussian noise, salt and pepper noise, Poisson noise, etc.), blur degradation (for example, motion blur, defocus blur, Gaussian blur, etc.), geometric distortion (for example, rotation / scaling distortion, perspective distortion, barrel / pincushion distortion, etc.), color distortion (for example, white balance error, color saturation abnormality, channel imbalance, etc.), compression artifacts (for example, block effect, ringing effect, etc.), and low-light degradation (for example, dark corners, low signal-to-noise ratio, etc.).

[0084] The beneficial effects of the above technical solution are as follows: the present invention generates low-resolution degraded images through a multi-order infrared image degradation model, which can highly simulate various degradation conditions that infrared images may encounter during actual acquisition and transmission, such as noise interference, downsampling blur, etc. The degradation simulation makes the subsequent training and reconstruction of the super-resolution reconstruction network more targeted. Compared with directly processing the original clear image, the reconstructed super-resolution image can better meet the demand for image quality improvement in actual application scenarios. At the same time, it also overcomes the limitation of the amount of training data of the super-resolution reconstruction network, and improves the foundation for large-scale data training of the super-resolution reconstruction network.

[0085] Example 3:

[0086] On the basis of Example 1, step 2, as Figure 3 As shown, including:

[0087] Step 201: After a low-resolution infrared image is subjected to a first convolution process and input, it is processed based on N RRDB modules to obtain a first output image;

[0088] Step 202: performing a second convolution process on the first output image and then upsampling it to obtain an upsampled image;

[0089] Step 203: acquiring image features extracted by each network layer of the upsampled image and the low-resolution infrared image based on the residual dense network, thereby obtaining features of the upsampled image and original image features of the low-resolution infrared image;

[0090] Step 204: Perform convolution fusion on the upsampled image features and the original image features to obtain a super-resolution reconstructed image.

[0091] In this embodiment, the output of each layer is connected to the input of the subsequent layer through dense connection within N RRDB modules.

[0092] In this embodiment, the dense block (i.e., RRDB module) is composed of a convolutional layer and a LeakyRelu activation function. The convolutional layer is mainly used to extract feature maps, while the activation function is mainly used to increase the nonlinearity of the network. Thanks to the RRDB structure, the generator can capture more complex features and enhance the ability to restore image details.

[0093] In this embodiment, the convolution kernels of the first convolution process and the second convolution process may be the same or different, and can be customized according to the actual needs of the user;

[0094] The first convolution process is a single convolution, and the second convolution process is multiple continuous convolutions. The convolution kernels of the multiple continuous convolutions may be the same or different.

[0095] The beneficial effects of the above technical solution are as follows: the present invention performs a first convolution processing on a low-resolution infrared image and then processes it based on N RRDB modules to obtain a first output image; the first output image is subjected to a second convolution processing and then up-sampled to obtain an up-sampled image; based on a residual dense network, the image features extracted from each network layer of the up-sampled image and the low-resolution infrared image are respectively obtained to obtain the up-sampled image features and the original image features of the low-resolution infrared image; the up-sampled image features and the original image features are convolutionally fused to obtain a super-resolution reconstructed image, thereby realizing autonomous reconstruction of the low-resolution infrared image.

[0096] Example 4:

[0097] On the basis of Example 1, step 3 includes:

[0098] Perform non-overlapping sliding on the super-resolution reconstructed image based on a preset rectangular window, calculate the local entropy corresponding to the preset window, and determine whether the local entropy is greater than a local entropy threshold;

[0099] If yes, adaptively adjust the rectangular window based on the error between the local entropy and the local entropy threshold, and compare and calculate the pixel data in the adjusted rectangular window with the pixel data corresponding to the original infrared image to obtain a local residual map;

[0100] Otherwise, the pixel data within the preset rectangular window is obtained and compared with the pixel data corresponding to the original infrared image to obtain a local residual map;

[0101] Based on the local residual map, calculating the local variance of each pixel point in the adjusted rectangular window or the preset rectangular frame, and when the local variance is greater than the preset variance, determining that the current pixel point is an artifact pixel point;

[0102] Otherwise, the current pixel is determined to be a true shadow pixel;

[0103] Based on the local variance corresponding to all artifact pixels in the super-resolution reconstructed image, the local discrimination loss is calculated;

[0104] The step of adaptively adjusting the rectangular window based on an error between the local entropy and a local entropy threshold comprises:

[0105] Based on big data technology, the window information entropy corresponding to the infrared images of rectangular windows of different sizes is collected;

[0106] Grouping the window information entropy based on the window size to obtain rectangular window data groups of multiple sizes, determining the window information entropy range corresponding to each rectangular window data group, and determining whether there is an overlapping range between the window information entropy ranges corresponding to rectangular windows of different sizes;

[0107] If it does not exist, the window information entropy range corresponding to the current rectangular windows of different sizes is used as the final window information entropy range;

[0108] If so, obtaining rectangular window data groups corresponding to the overlapping rectangular windows as the first target data group and the second target data group, respectively, clustering the data in the first target data group and the second target data group to obtain the first cluster center and the second cluster center;

[0109] Based on the first cluster center and the second cluster center, determine the target window information entropy value corresponding to the distance center point, and judge whether the target window information entropy value is within the overlapping range. If so, based on the target window information entropy value, respectively correct the upper limit value of the window information entropy range corresponding to the first target data group and the lower limit value of the window information entropy range corresponding to the second target data group;

[0110] Otherwise, when the target window information entropy value is less than or equal to the lower limit value of the overlapping range, the lower limit value of the window information entropy range corresponding to the second target data group is modified based on the lower limit value of the overlapping range;

[0111] When the target window information entropy value is greater than or equal to the upper limit value of the overlapping range, the lower limit value of the window information entropy range corresponding to the first target data group is modified based on the upper limit value of the overlapping range;

[0112] Based on the correction results, the final window information entropy range corresponding to rectangular windows of different sizes is determined;

[0113] According to the final window information entropy range corresponding to rectangular windows of different sizes, the window information entropy difference interval between rectangular windows of different sizes is determined;

[0114] The error value between the local entropy and the local entropy threshold is matched with the window information entropy difference interval between multiple sized rectangular windows and the preset rectangular window to determine the size corresponding to the optimal rectangular window, and the window size of the preset rectangular window is adaptively adjusted according to the size corresponding to the optimal rectangular window.

[0115] The beneficial effects of the above technical solution are as follows: based on a preset rectangular window, non-overlapping sliding is performed on the super-resolution reconstructed image, and the local entropy corresponding to the preset window is calculated to determine whether the local entropy is greater than the local entropy threshold; if so, the rectangular window is adaptively adjusted based on the error value between the local entropy and the local entropy threshold, while ensuring the accurate capture of the local features of the image, ensuring that the regional information within each rectangular frame meets a certain standard, effectively improving the accuracy of subsequent local variance calculation, and then obtaining the pixel point data within the rectangular window and the pixel point data corresponding to the original infrared image for comparison and calculation to obtain a local residual map; based on the local residual map, the local variance of each pixel point in the adjusted rectangular window or the preset rectangular frame is calculated, and when the local variance is greater than the preset variance, the current pixel point is determined to be an artifact pixel point; otherwise, the current pixel point is determined to be a true shadow pixel point, thereby realizing the identification of artifact pixels in the local area, and based on the local variance corresponding to all artifact pixels in the super-resolution reconstructed image, the local discrimination loss is calculated, which provides a basis for further training of the super-resolution reconstruction network.

[0116] The present invention is based on

[0117] Example 5:

[0118] On the basis of Example 4, the local discriminant loss is calculated based on the local variance corresponding to all artifact pixels in the super-resolution reconstructed image, including:

[0119] The standard local discrimination loss corresponding to the super-resolution reconstructed image is calculated based on the preset local discrimination loss model, and the artifact pixels in the super-resolution reconstructed image are marked. Based on the marking results, the distribution characteristics of the artifact pixels are extracted;

[0120] Based on the distribution characteristics, each image element in the super-resolution reconstructed image is evaluated, and according to the evaluation result, the image distortion rate of the super-resolution reconstructed image is determined;

[0121] evaluating a global reconstruction error of the super-resolution reconstructed image based on the image distortion rate; obtaining a global reconstruction error index based on an error ratio between the reconstruction error and the preset error when the reconstruction error is greater than a preset error; and correcting a standard local discrimination error based on the global reconstruction error index to obtain an output local discrimination error;

[0122] Otherwise, the standard local discriminant error is used as the output local discriminant error.

[0123] Among them, each image element in the super-resolution reconstructed image is evaluated, including:

[0124] Perform edge detection on the super-resolution reconstructed image, and based on the edge detection results, segment the image information on the super-resolution reconstructed image into multiple image element information;

[0125] Based on the distribution characteristics of artifact pixels and the image element information segmentation results, the correlation relationship of the image representation information of each artifact pixel on the valid image is determined;

[0126] Based on the association relationship, determining the number of artifact pixels and position distribution sub-features on each image element;

[0127] Obtaining the total number of pixels corresponding to each image element having an artifact pixel, and obtaining an artifact index of the corresponding image element based on the total number of pixels and the number of artifact pixels of the corresponding image element;

[0128] When the artifact index is greater than or equal to a preset threshold, determining that the corresponding image element is distorted;

[0129] Otherwise, based on the position distribution sub-features corresponding to the artifact pixels on the corresponding image element, determine whether there is an artifact pixel concentration area on the corresponding image element. If so, obtain the element distortion ratio based on the ratio of the number of artifact pixels in the artifact pixel concentration area to the total number of pixels corresponding to the corresponding image element.

[0130] Determining a maximum artifact pixel concentration area based on the element distortion ratio, and determining that the corresponding image element is distorted when the element distortion ratio corresponding to the maximum artifact pixel concentration area is greater than or equal to a threshold distortion ratio;

[0131] Otherwise, it is determined that the corresponding image element is not distorted;

[0132] According to the evaluation result, the image distortion rate of the super-resolution reconstructed image is determined, including:

[0133] Traversing the image elements on the super-resolution reconstructed image to determine the total number of image elements and the number of distorted image elements corresponding to the super-resolution reconstructed image;

[0134] Based on the total number of image elements corresponding to the super-resolution reconstructed image and the number of distorted image elements, a distortion element ratio of the super-resolution reconstructed image is obtained, and the distortion element ratio is used as the image distortion rate of the super-resolution reconstructed image.

[0135] The beneficial effects of the above technical solution: the present invention can accurately locate and analyze the areas and patterns where artifacts appear in the image by marking the artifact pixels in the super-resolution reconstructed image and extracting their distribution characteristics, which helps to gain a deep understanding of the distribution law of artifacts in the image and provide detailed and targeted information for subsequent evaluation. Afterwards, each image element is evaluated based on the distribution characteristics of the artifact pixels, and then the image distortion rate is determined, which can comprehensively and objectively reflect the degree of difference between the super-resolution reconstructed image and the original image. The evaluation method from the micro-pixel level to the macro-image level can more accurately measure the quality loss of the image, and then the image distortion rate is used to evaluate the reconstruction error of the super-resolution reconstructed image, establish a connection between image quality and reconstruction error, and intuitively understand the degree of difference between the reconstruction process due to artifacts, etc. The degree of deviation from the ideal reconstruction result caused by factors provides a quantitative basis for judging the effectiveness of the reconstruction algorithm. When the reconstruction error is greater than the preset error, the standard local judgment loss is corrected in combination with the error rate between the reconstruction error and the preset error. This adaptive adjustment method can optimize the loss function of the model in a targeted manner according to the error of the actual reconstruction result, so that the model pays more attention to the areas with artifacts and large reconstruction errors in the subsequent training process, thereby reducing artifacts in a targeted manner and improving the quality of the reconstructed image. Moreover, by continuously adjusting the local judgment loss, the loss adjustment in the entire training process is achieved, so that the model can gradually learn how to avoid artifacts and improve the accuracy of the reconstructed image, which helps the model to continuously optimize its own parameters during the training process, thereby improving the overall performance of the super-resolution reconstruction algorithm.

[0136] Example 6:

[0137] Based on Example 1, a local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation further includes:

[0138] The user inputs the low-resolution infrared image to be processed into the trained super-resolution reconstruction network for processing to obtain a super-resolution reconstructed infrared image;

[0139] The super-resolution reconstructed infrared image is sent to a corresponding user terminal for display based on a preset sending address.

[0140] The beneficial effects of the above technical solution are as follows: the user of the present invention inputs the low-resolution infrared image to be processed into a trained super-resolution reconstruction network for processing to obtain a super-resolution reconstructed infrared image; and based on a preset sending address, the super-resolution reconstructed infrared image is sent to the corresponding user terminal for display, thereby realizing the automatic reconstruction of the low-resolution infrared image and the intelligent data transmission of the super-resolution reconstructed image.

[0141] Example 7:

[0142] Based on Example 1, a local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation further includes:

[0143] Based on big data, a large number of high-resolution infrared images are obtained as original infrared images to establish a training database, and the output super-resolution reconstructed images corresponding to the trained super-resolution reconstruction network are collected to update the training database;

[0144] When the data update amount of the training database reaches the minimum update threshold, a plurality of original infrared images are randomly selected based on the latest training database, and steps 1-4 are executed.

[0145] The beneficial effects of the above technical solution are as follows: the present invention obtains a large number of high-resolution infrared images based on big data as original infrared images to establish a training database, which provides a data basis for model training. At the same time, the output super-resolution reconstructed images corresponding to the trained super-resolution reconstruction network are collected to update the training database. When the data update amount of the training database reaches the minimum update threshold, based on the latest training database, multiple original infrared images are randomly extracted to re-train the super-resolution reconstruction network, thereby realizing autonomous learning and regular updating of the super-resolution reconstruction network, which is beneficial to improving the accuracy of image reconstruction.

[0146] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation, characterized in that: include: Step 1: Input the original infrared image into the multi-order infrared image degradation model to obtain a low-resolution degraded image; Step 2: Input the low-resolution degraded image into the generator of the super-resolution reconstruction network to generate a super-resolution reconstructed image; Step 3: After feature fusion of the super-resolution reconstruction and the original infrared image, the image is input into the local discriminator of the super-resolution reconstruction network for processing, and the local discriminant loss is obtained; Step 4: Perform weighted summation of the local discriminant loss and the preset model loss to obtain the image reconstruction training loss of the super-resolution reconstructed image; When the image reconstruction training loss is greater than the minimum loss value, return to step 2; Otherwise, the training is terminated and the trained super-resolution reconstruction network is output; Wherein, step 2 includes: After the low-resolution infrared image is subjected to the first convolution processing and input, it is processed based on N RRDB modules to obtain a first output image; After performing a second convolution process on the first output image, upsampling is performed to obtain an upsampled image; Based on the residual dense network, the image features extracted by each network layer of the upsampled image and the low-resolution infrared image are obtained respectively, and the upsampled image features and the original image features of the low-resolution infrared image are obtained; Perform convolution fusion on the upsampled image features and the original image features to obtain a super-resolution reconstructed image; Step 3 includes: Perform non-overlapping sliding on the super-resolution reconstructed image based on a preset rectangular window, calculate the local entropy corresponding to the preset window, and determine whether the local entropy is greater than a local entropy threshold; If yes, adaptively adjust the rectangular window based on the error between the local entropy and the local entropy threshold, and compare and calculate the pixel data in the adjusted rectangular window with the pixel data corresponding to the original infrared image to obtain a local residual map; Otherwise, the pixel data within the preset rectangular window is obtained and compared with the pixel data corresponding to the original infrared image to obtain a local residual map; Based on the local residual map, calculating the local variance of each pixel point in the adjusted rectangular window or the preset rectangular frame, and when the local variance is greater than the preset variance, determining that the current pixel point is an artifact pixel point; Otherwise, the current pixel is determined to be a true shadow pixel; Based on the local variance corresponding to all artifact pixels in the super-resolution reconstructed image, the local discriminant loss is calculated.

2. The local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation according to claim 1, wherein step 1 comprises: When the original infrared image is input into the multi-order infrared image degradation model, the multi-order infrared image degradation model randomly arranges multiple image degradation strategies based on the image degradation strategy database to generate a multi-order infrared image degradation process; Based on the multi-stage infrared image degradation process, the original infrared image is processed to obtain a low-resolution degraded image; Wherein, the multi-order infrared image degradation model is a third-order degradation model.

3. The local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation according to claim 1, characterized in that: Adaptively adjusting the rectangular window based on an error value between the local entropy and a local entropy threshold includes: Based on big data technology, the window information entropy corresponding to the infrared images of rectangular windows of different sizes is collected; Grouping the window information entropy based on the window size to obtain rectangular window data groups of multiple sizes, determining the window information entropy range corresponding to each rectangular window data group, and determining whether there is an overlapping range between the window information entropy ranges corresponding to rectangular windows of different sizes; If it does not exist, the window information entropy range corresponding to the current rectangular windows of different sizes is used as the final window information entropy range; If so, obtaining rectangular window data groups corresponding to the overlapping rectangular windows as the first target data group and the second target data group, respectively, clustering the data in the first target data group and the second target data group to obtain the first cluster center and the second cluster center; Based on the first cluster center and the second cluster center, determine the target window information entropy value corresponding to the distance center point, and judge whether the target window information entropy value is within the overlapping range. If so, based on the target window information entropy value, respectively correct the upper limit value of the window information entropy range corresponding to the first target data group and the lower limit value of the window information entropy range corresponding to the second target data group; Otherwise, when the target window information entropy value is less than or equal to the lower limit value of the overlapping range, the lower limit value of the window information entropy range corresponding to the second target data group is modified based on the lower limit value of the overlapping range; When the target window information entropy value is greater than or equal to the upper limit value of the overlapping range, the lower limit value of the window information entropy range corresponding to the first target data group is modified based on the upper limit value of the overlapping range; Based on the correction results, the final window information entropy range corresponding to rectangular windows of different sizes is determined; According to the final window information entropy range corresponding to rectangular windows of different sizes, the window information entropy difference interval between rectangular windows of different sizes is determined; The error value between the local entropy and the local entropy threshold is matched with the window information entropy difference interval between multiple sized rectangular windows and the preset rectangular window to determine the size corresponding to the optimal rectangular window, and the window size of the preset rectangular window is adaptively adjusted according to the size corresponding to the optimal rectangular window.

4. The local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation according to claim 3, wherein the local discriminant loss is calculated based on the local variance corresponding to all artifact pixels in the super-resolution reconstructed image, comprising: The standard local discrimination loss corresponding to the super-resolution reconstructed image is calculated based on the preset local discrimination loss model, and the artifact pixels in the super-resolution reconstructed image are marked. Based on the marking results, the distribution characteristics of the artifact pixels are extracted; Based on the distribution characteristics, each image element in the super-resolution reconstructed image is evaluated, and according to the evaluation result, the image distortion rate of the super-resolution reconstructed image is determined; evaluating a global reconstruction error of the super-resolution reconstructed image based on the image distortion rate, obtaining a global reconstruction error index based on an error ratio between the global reconstruction error and the preset error when the global reconstruction error is greater than a preset error, and correcting a standard local discrimination error based on the global reconstruction error index to obtain an output local discrimination error; Otherwise, the standard local discriminant error is used as the output local discriminant error.

5. The local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation according to claim 4, wherein each image element in the super-resolution reconstructed image is evaluated based on distribution characteristics, comprising: Perform edge detection on the super-resolution reconstructed image, and based on the edge detection results, segment the image information on the super-resolution reconstructed image into multiple image element information; Based on the distribution characteristics of artifact pixels and the image element information segmentation results, the correlation relationship between the image representation information of each artifact image on the valid image is determined; Based on the association relationship, determining the number of artifact pixels and position distribution sub-features on each image element; Obtaining the total number of pixels corresponding to each image element having an artifact pixel, and obtaining an artifact index of the corresponding image element based on the total number of pixels and the number of artifact pixels of the corresponding image element; When the artifact index is greater than or equal to a preset threshold, determining that the corresponding image element is distorted; Otherwise, based on the position distribution sub-features corresponding to the artifact pixels on the corresponding image element, determine whether there is an artifact pixel concentration area on the corresponding image element. If so, obtain the element distortion ratio based on the ratio of the number of artifact pixels in the artifact pixel concentration area to the total number of pixels corresponding to the corresponding image element. Based on the element distortion ratio, determining the area where the maximum artifact pixel point is concentrated, when the element distortion ratio corresponding to the area where the maximum artifact pixel point is concentrated is greater than or equal to the threshold distortion ratio, determining that the corresponding image element is distorted; Otherwise, it is determined that the corresponding image element is not distorted.

6. The local discriminant super-resolution reconstruction method based on multi-order infrared image degradation simulation according to claim 5, wherein the image distortion rate of the super-resolution reconstructed image is determined based on the evaluation result, comprising: Traversing the image elements on the super-resolution reconstructed image to determine the total number of image elements and the number of distorted image elements corresponding to the super-resolution reconstructed image; Based on the total number of image elements corresponding to the super-resolution reconstructed image and the number of distorted image elements, a distortion element ratio of the super-resolution reconstructed image is obtained, and the distortion element ratio is used as the image distortion rate of the super-resolution reconstructed image.

7. The local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation according to claim 1, characterized in that: Also includes: The user inputs the low-resolution infrared image to be processed into the trained super-resolution reconstruction network for processing to obtain a super-resolution reconstructed infrared image; The super-resolution reconstructed infrared image is sent to a corresponding user terminal for display based on a preset sending address.

8. The local discrimination super-resolution reconstruction method based on multi-order infrared image degradation simulation according to claim 7, characterized in that: Also includes: Based on big data, a large number of high-resolution infrared images are obtained as original infrared images to establish a training database, and the output super-resolution reconstructed images corresponding to the trained super-resolution reconstruction network are collected to update the training database; When the data update amount of the training database reaches a minimum update threshold, a plurality of original infrared images are randomly sampled based on the latest training database, and steps 1-4 are executed.

Images