Multi-feature fusion infrared ship image restoration method, device, equipment and medium

Abstract

The present application discloses a method, device, equipment and medium for repairing infrared ship images with multi-feature fusion, the method comprising obtaining first image data and real image data, performing manual feature extraction and deep feature extraction on the first image data to obtain manual feature data and deep feature data; fusing the manual feature data with the deep feature data to obtain fusion features; inputting the fusion features into a convolutional neural network model to repair the first image data to obtain second image data; obtaining a reconstruction loss function based on the second image data and the real image data, optimizing the convolutional neural network model based on the reconstruction loss function to obtain an infrared ship image repair model; and repairing the infrared ship image based on the infrared ship image repair model. The present application has strong repair robustness in complex scenes and extreme conditions, and can stably restore infrared image details and improve the overall repair quality.

Description

Technical Field

[0001] The present application relates to the field of image restoration, and in particular to a method, device, equipment and medium for restoring infrared ship images with multi-feature fusion. Background Art

[0002] Infrared imaging technology has attracted widespread attention and application due to its outstanding performance in low light and complex meteorological conditions. Compared with visible light imaging, infrared imaging can capture the thermal radiation characteristics of objects under a variety of adverse conditions, and has stronger anti-interference ability and environmental adaptability. In marine applications, infrared imaging can ensure stable observation of ships in complex conditions such as night and haze, and improve the generalization and robustness of the model. However, infrared images often face problems such as damage, noise interference, and low resolution in practical applications. These challenges lead to a decline in image quality and affect the accuracy of subsequent target recognition or detection. Therefore, it is particularly critical to effectively repair infrared images.

[0003] In the related technology, manual features are used for image restoration. Manual features use classic image processing operators to extract geometric and texture detail information, providing rich and direct visual features. They have unique advantages, especially in capturing complex details. However, the disadvantages of image restoration using manual features are low accuracy and slow speed.

[0004] In recent years, infrared ship classification methods based on deep learning have received widespread attention from scholars. Researchers have used deep convolutional neural network models to achieve infrared ship image feature extraction with higher accuracy and faster speed than traditional manual feature classification models. In addition, deep convolutional neural network models can be trained and tested end-to-end, which greatly simplifies the design process and reduces the burden of manual feature extraction.

[0005] However, sole reliance on depth features may lead to defects such as missing information, large data requirements, and poor interpretability, resulting in the inability of existing infrared image restoration methods to stably restore infrared image details in complex scenes and extreme conditions, resulting in poor restoration quality. Summary of the invention

[0006] The present application aims to propose a multi-feature fusion infrared ship image restoration method, device, equipment and medium, which has strong restoration robustness in complex scenes and extreme conditions, can stably restore infrared image details and improve the overall restoration quality.

[0007] In a first aspect, an embodiment of the present application provides a multi-feature fusion infrared ship image restoration method, comprising the following steps:

[0008] Acquire first image data and real image data, wherein the first image data is a damaged infrared ship image, and the real image data is a real image corresponding to the infrared ship image;

[0009] Performing manual feature extraction on the first image data to obtain manual feature data;

[0010] Performing depth feature extraction on the first image data to obtain depth feature data;

[0011] Fusing the manual feature data with the deep feature data to obtain fused features;

[0012] Inputting the fused features into a convolutional neural network model to repair the first image data to obtain second image data;

[0013] Obtaining a reconstruction loss function according to the second image data and the real image data, wherein the reconstruction loss function is used to represent the difference between the restored image and the real image;

[0014] Optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model;

[0015] The infrared ship image is repaired according to the infrared ship image repair model.

[0016] According to some embodiments of the present application, the manual feature data includes a manual feature vector, and the deep feature data includes a deep feature vector;

[0017] The step of optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model includes:

[0018] Perform collaborative learning calculation based on the manual feature vector and the deep feature vector to obtain an information matching loss function;

[0019] Obtaining a total loss function according to the reconstruction loss function and the information matching loss function;

[0020] The convolutional neural network model is optimized according to the total loss function to obtain an infrared ship image restoration model.

[0021] According to some embodiments of the present application, the manual feature data includes a manual feature vector and a manual loss function, and the depth feature data includes a depth feature vector and a depth loss function;

[0022] The step of optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model includes:

[0023] Perform contribution allocation calculation according to the manual feature vector and the deep feature vector to obtain a manual feature weight value and a deep feature weight value;

[0024] Obtaining a feature weighted loss function according to the manual feature weight value, the deep feature weight value, the manual loss function and the deep loss function;

[0025] Obtaining a total loss function according to the reconstruction loss function and the feature weighted loss function;

[0026] The convolutional neural network model is optimized according to the total loss function to obtain an infrared ship image restoration model.

[0027] According to some embodiments of the present application, the manual feature data includes a manual feature vector and a manual loss function, and the deep feature data includes a deep feature vector and a deep loss function;

[0028] The step of optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model includes:

[0029] Perform collaborative learning calculation based on the manual feature vector and the deep feature vector to obtain an information matching loss function;

[0030] Perform contribution allocation calculation according to the manual feature vector and the deep feature vector to obtain a manual feature weight value and a deep feature weight value;

[0031] Obtaining a feature weighted loss function according to the manual feature weight value, the deep feature weight value, the manual loss function and the deep loss function;

[0032] Obtaining a total loss function according to the reconstruction loss function, the information matching loss function and the feature weighted loss function;

[0033] The convolutional neural network model is optimized according to the total loss function to obtain an infrared ship image restoration model.

[0034] According to some embodiments of the present application, the step of manually extracting features from the first image data to obtain manual feature data includes:

[0035] Select at least two manual feature extraction operators;

[0036] Manual feature extraction is performed on the first image data using at least two manual feature extraction operators to obtain at least two types of manual feature data.

[0037] According to some embodiments of the present application, the manual feature data includes a manual feature map, and the deep feature data includes at least two deep feature maps corresponding one-to-one to the manual feature extraction operators;

[0038] The fusing the manual feature data with the deep feature data to obtain a fused feature includes:

[0039] At least two of the manual feature maps are fused with the deep feature map to obtain a fused feature.

[0040] According to some embodiments of the present application, inputting the fusion features into a convolutional neural network model to repair the first image data to obtain the second image data includes:

[0041] The fused features are processed through the spatial Dropout layer and attention layer of the convolutional neural network model to generate fused feature representation;

[0042] The fused feature representation is sent to the decoder of the convolutional neural network model to repair the first image data to obtain the second image data.

[0043] In a second aspect, the embodiment of the present application provides a multi-feature fusion infrared ship image restoration device, comprising:

[0044] A data acquisition module, used to acquire first image data and real image data, wherein the first image data is a damaged infrared ship image, and the real image data is a real image corresponding to the infrared ship image;

[0045] A manual feature extraction module, used for performing manual feature extraction on the first image data to obtain manual feature data;

[0046] A depth feature extraction module, used to extract depth features from the first image data to obtain depth feature data;

[0047] A feature fusion module, used to fuse the manual feature data with the deep feature data to obtain a fused feature;

[0048] A first restoration module, used for inputting the fusion features into a convolutional neural network model to restore the first image data to obtain second image data;

[0049] A reconstruction loss module, used to obtain a reconstruction loss function according to the second image data and the real image data, wherein the reconstruction loss function is used to represent the difference between the restored image and the real image;

[0050] A model optimization module, used for optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model;

[0051] The second restoration module is used to restore the infrared ship image according to the infrared ship image restoration model.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0053] When the processor executes the computer program instructions, the infrared ship image restoration method of multi-feature fusion as described in the first aspect is implemented.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the infrared ship image restoration method with multi-feature fusion as described in the first aspect is implemented.

[0055] The multi-feature fusion infrared ship image restoration method, device, equipment and medium of the embodiment of the present application have at least the following beneficial effects:

[0056] In the implementation mode of the present application, firstly, the first image data and the real image data are obtained, and then the first image data is subjected to manual feature extraction to obtain manual feature data; the first image data is subjected to deep feature extraction to obtain deep feature data; then the manual feature data is fused with the deep feature data to obtain fused features; the fused features are input into the convolutional neural network model to repair the first image data to obtain the second image data; according to the second image data and the real image data, a reconstruction loss function is obtained, and the reconstruction loss function is used to represent the difference between the repaired image and the real image; according to the reconstruction loss function, the convolutional neural network model is optimized to obtain an infrared ship image repair model; finally, the infrared ship image is repaired according to the infrared ship image repair model. The present application integrates manual features with deep features and constructs a reconstruction loss function so that the manual features and deep features can work together. The present application has strong repair robustness in complex scenes and extreme conditions, and can stably restore infrared image details and improve the overall repair quality.

[0057] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present application is further described below with reference to the accompanying drawings and embodiments, wherein:

[0059] Figure 1A schematic flow chart of an embodiment of the multi-feature fusion infrared ship image restoration method provided in the present application;

[0060] Figure 2 A schematic diagram of the structure of the infrared ship image restoration device with multi-feature fusion provided in this application;

[0061] Figure 3 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0062] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0063] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0064] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment and medium for repairing infrared ship images by fusion of multiple features. The following first introduces the repairing of infrared ship images by fusion of multiple features provided by the embodiments of the present application.

[0065] Figure 1 The flowchart of the multi-feature fusion infrared ship image restoration method provided in the embodiment of the present application is shown. The method is applied to electronic equipment, a multi-feature fusion infrared ship image restoration method, comprising the following steps:

[0066] S101, obtaining first image data and real image data, where the first image data is a damaged infrared ship image, and the real image data is a real image corresponding to the infrared ship image;

[0067] S102, performing manual feature extraction on the first image data to obtain manual feature data;

[0068] S103, extracting depth features from the first image data to obtain depth feature data;

[0069] S104, fusing the manual feature data with the deep feature data to obtain fused features;

[0070] S105, inputting the fusion feature into the convolutional neural network model to repair the first image data to obtain second image data;

[0071] S106, obtaining a reconstruction loss function according to the second image data and the real image data, where the reconstruction loss function is used to represent the difference between the restored image and the real image;

[0072] S107, optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model;

[0073] S108. Repair the infrared ship image according to the infrared ship image repair model.

[0074] In this embodiment, firstly, the first image data and the real image data are obtained, and then the first image data is subjected to manual feature extraction to obtain manual feature data; the first image data is subjected to deep feature extraction to obtain deep feature data; then the manual feature data is fused with the deep feature data to obtain fused features; the fused features are input into the convolutional neural network model to repair the first image data to obtain the second image data; according to the second image data and the real image data, a reconstruction loss function is obtained, and the reconstruction loss function is used to represent the difference between the repaired image and the real image; according to the reconstruction loss function, the convolutional neural network model is optimized to obtain an infrared ship image repair model; finally, the infrared ship image is repaired according to the infrared ship image repair model. This application integrates manual features with deep features and constructs a reconstruction loss function so that manual features and deep features can work together. This application has strong repair robustness in complex scenes and extreme conditions, and can stably restore infrared image details and improve the overall repair quality.

[0075] In the above step S101, the damaged infrared ship image refers to a given damaged infrared image to be repaired, and the real image is a real image corresponding to the infrared ship image. The first image data and the real image data are obtained by obtaining the preprocessed image data or the original image. When obtaining the original image, the damaged infrared ship image and the real image need to be preprocessed and converted into grayscale images as image data.

[0076] In the above step S102, manual feature extraction is performed on the first image data to obtain manual feature data. Manual feature extraction refers to extracting features from a damaged infrared image by a feature extraction operator, and manual feature data refers to data related to manual features, such as manual feature vectors, manual feature mappings, category prediction probabilities of manual features, manual loss functions, etc.

[0077] Among them, the manual feature vector and the manual feature map are both the outputs of the feature extractor, the category prediction probability of the manual feature is the output of the manual feature auxiliary classifier, and the manual loss function is used to train the manual feature extraction branch.

[0078] It should be noted that the feature extraction operator may be a FAST operator, a SIFT operator, a DoG operator or other feature extraction operators, or only one feature extraction operator may be used to perform manual feature extraction on the first image data, or two or more feature extraction operators may be used to perform manual feature extraction on the first image data. After the manual feature extraction, the extracted manual features are dimensionally unified and saved.

[0079] It should be noted that after the manual features of the first image data are extracted and saved, the manual feature data is used as the input of the manual feature encoder, and the manual feature vector, manual feature map and category prediction probability of the manual feature are obtained through the manual feature auxiliary classifier, and the manual loss function is calculated.

[0080] Specifically, if the number of manual features used is n, any number of manual feature data corresponding to the image data is used as the input of the manual feature encoder, and the manual feature auxiliary classifier is used to obtain n manual feature vectors. , manual feature mapping And the category prediction probability of manual features , the class prediction probability of the manual feature refers to the class probability output of the manual feature auxiliary classifier, and the manual loss function is calculated , the specific calculation formula of the manual loss function is as follows:

[0081] ;

[0082] in, represents the manual feature extraction branch, Represents the total number of categories, Representative samples in The true value label of the category, Represents the manual feature-assisted classifier for the sample belonging to the category The predicted probability of .

[0083] In the above step S103, deep feature extraction is performed on the first image data to obtain deep feature data, which means extracting deep features through a deep convolutional neural network model. The deep feature data refers to data related to deep features, such as deep feature vectors, deep feature maps, category prediction probabilities of deep features, deep loss functions, etc.

[0084] Among them, the deep feature vector and deep feature map are both the outputs of the feature extractor, the category prediction probability of the deep feature is the output of the deep feature auxiliary classifier, and the deep loss function is used to train the deep feature extraction branch.

[0085] Specifically, the processed image data is input into the deep feature encoder, and the deep feature vector is obtained through the deep feature auxiliary classifier. , deep feature map and the class prediction probability of deep features , the category prediction of deep features refers to the class probability output of the deep feature auxiliary classifier, and the deep loss function is calculated , the specific calculation formula of the depth loss function is as follows:

[0086] ;

[0087] in, represents the deep feature extraction branch, Represents the total number of categories, Representative samples in The true value label of the category, Represents the deep feature auxiliary classifier for the sample belonging to the category The predicted probability of .

[0088] In the above step S104, the manual feature data and the deep feature data are fused to obtain fused features; this means that the manual features are fused with the deep features to obtain fused features, and the fused features can have the advantages of both manual features and deep features.

[0089] Specifically, manual features use classic image processing operators to extract geometric and texture detail information, providing rich and direct visual features, especially with unique advantages in capturing complex details. Deep features extract high-level semantic information through convolutional neural networks. After fusing the two, manual features and deep features can complement each other and enhance the robust representation capability of infrared images.

[0090] In the above step S105, the fused features are input into the convolutional neural network model to repair the first image data to obtain the second image data, which means that the fused features are input into the convolutional neural network model to generate a fused feature representation, and the fused features are sent to the decoder of the convolutional neural network model. The image details are restored by stepwise upsampling to obtain the second image data. The second image data refers to the image data after preliminary repair.

[0091] Specifically, in the embodiment of the present application, a convolutional neural network model with a U-Net structure is used for image restoration, that is, the fused features are sent to the U-Net decoder for image restoration, wherein the encoder part of the U-Net structure integrates a multi-feature fusion module and a semantic guidance module to ensure efficient extraction of multi-level features and enhance the semantic consistency of image restoration.

[0092] In the above step S106, the reconstruction loss function is obtained according to the second image data and the real image data, which means that the difference between the image after the preliminary restoration and the real image is measured by the reconstruction loss function.

[0093] Specifically, the L1 norm is used in the embodiment of the present application to calculate the reconstruction loss function , reconstruction loss function The specific calculation formula is as follows:

[0094] ;

[0095] Where N is the total number of pixels in the image, represents the value of the restored image at the i-th pixel position, Represents the value of the true image at the i-th pixel position.

[0096] In the above step S107, the convolutional neural network model is optimized according to the reconstruction loss function to obtain the infrared ship image restoration model, which means that the reconstruction loss function is optimized according to the reconstruction loss function. As the total loss function To optimize the convolutional neural network model, that is The infrared ship image restoration model refers to a convolutional neural network model that has been iteratively trained until the model converges.

[0097] Specifically, after obtaining the total loss function, in this embodiment, the gradient of the total loss function to the model parameters is calculated by the back propagation algorithm, and the model parameters of the convolutional neural network model are updated by the optimizer to minimize the total loss function. Iterative training is performed until the model converges to ensure that the optimized infrared ship image restoration model can achieve high-quality restoration in various damaged infrared image scenes. It should be understood that in addition to the back propagation algorithm, other algorithms can also be used to calculate the loss gradient.

[0098] The above step S108, repairing the infrared ship image according to the infrared ship image repair model, refers to repairing the damaged infrared ship image according to the optimized iterative convolutional neural network model, eliminating the damage and noise interference of the infrared ship image, improving the resolution and quality of the image, and improving the accuracy of subsequent tasks (such as target recognition and detection).

[0099] In some embodiments, the manual feature data includes a manual feature vector, and the deep feature data includes a deep feature vector;

[0100] The convolutional neural network model is optimized according to the reconstruction loss function to obtain the infrared ship image restoration model, which may include:

[0101] Perform collaborative learning calculation based on manual feature vectors and deep feature vectors to obtain the information matching loss function;

[0102] According to the reconstruction loss function and the information matching loss function, the total loss function is obtained;

[0103] The convolutional neural network model is optimized according to the total loss function to obtain the infrared ship image restoration model.

[0104] In this implementation, firstly, collaborative learning calculation is performed based on manual feature vectors and deep feature vectors to obtain the information matching loss function; then, the total loss function is obtained based on the reconstruction loss function and the information matching loss function; finally, the convolutional neural network model is optimized based on the total loss function to obtain the infrared ship image restoration model, which can further improve the image restoration quality.

[0105] The above-mentioned collaborative learning calculation based on the manual feature vector and the deep feature vector to obtain the information matching loss function refers to the knowledge collaborative learning calculation of the deep feature and the manual feature. In this embodiment, the information matching loss between each manual feature and the deep feature is calculated by KL divergence to encourage collaborative learning between different features. The information matching loss function The specific calculation formula is as follows:

[0106] ;

[0107] in, is a manual feature vector, is the deep feature vector, μ is the information loss of the knowledge matching between them, and μ=1 is set in this embodiment to ensure fairness. N represents the total number of data samples, and H represents the number of manual feature extraction branches. represent Manual feature extraction branch.

[0108] It should be understood that, in addition to the KL divergence algorithm, the Hellinger distance, Jensen-Shannon divergence or other related algorithms may also be used to calculate the information matching loss between each manual feature and the deep feature.

[0109] The above-mentioned total loss function is obtained based on the reconstruction loss function and the information matching loss function, which means that the reconstruction loss function And information matching loss function Add together to get the total loss function , the formula is as follows:

[0110] ;

[0111] The above optimization of the convolutional neural network model according to the total loss function to obtain the infrared ship image restoration model refers to the reconstruction loss function And information matching loss function The total loss function obtained by adding The convolutional neural network model is optimized to obtain the infrared ship image restoration model.

[0112] It should be noted that since both manual features and deep features correspond to the same image, there should be a certain similarity. Therefore, calculating the information matching loss can shorten the distance between manual features and deep features in the feature space to improve the similarity between features. Therefore, the information matching loss function obtained by knowledge collaborative learning can improve the effect of feature fusion. Therefore, the information matching loss function is added to the total loss function in order to integrate manual features in the restoration process and supplement the information of deep features, thereby further improving the restoration quality.

[0113] In some embodiments, the manual feature data includes a manual feature vector and a manual loss function, and the deep feature data includes a deep feature vector and a deep loss function;

[0114] The convolutional neural network model is optimized according to the reconstruction loss function to obtain the infrared ship image restoration model, which may include:

[0115] Contribution allocation is calculated based on the manual feature vector and the deep feature vector to obtain the manual feature weight value and the deep feature weight value;

[0116] Obtain a feature weighted loss function according to the manual feature weight value, the deep feature weight value, the manual loss function and the deep loss function;

[0117] According to the reconstruction loss function and the feature weighted loss function, the total loss function is obtained;

[0118] The convolutional neural network model is optimized according to the total loss function to obtain the infrared ship image restoration model.

[0119] In this implementation, firstly, the contribution distribution calculation is performed based on the manual feature vector and the deep feature vector to obtain the manual feature weight value and the deep feature weight value, then the feature weighted loss function is obtained based on the manual feature weight value, the deep feature weight value, the manual loss function and the deep loss function, then the total loss function is obtained based on the reconstruction loss function and the feature weighted loss function, and finally the convolutional neural network model is optimized based on the total loss function to obtain the infrared ship image restoration model, which can further improve the image restoration quality.

[0120] The above contribution distribution calculation based on the manual feature vector and the deep feature vector to obtain the manual feature weight value and the deep feature weight value refers to calculating the contribution distribution of the deep feature and the manual feature. When there are multiple manual features, it refers to calculating the contribution distribution of the deep feature and multiple manual features, as follows:

[0121] First concatenate the depth feature vector and handcrafted feature vectors Get feature map , the importance difference between deep features and manual features is weighted through the channel attention mechanism. By weighting the importance of channels, the model pays more attention to channels that contribute to the task and suppresses unimportant channels. The equation of the channel attention mechanism is as follows:

[0122]

[0123] Specifically, we first perform global average pooling (AvgPool) on the input feature map to extract the global information of each channel and obtain a channel-level feature vector. Then, we perform a nonlinear transformation on the feature vector through a multi-layer perceptron (MLP) to capture the dependencies and interaction information between channels. Next, we use the sigmoid function to map the output to the range [0, 1] to generate the attention weights for each channel. Finally, these weights are used to adjust the contribution of each channel in the input feature map, highlight important channels, and suppress irrelevant or redundant information, thereby improving the adaptability and robustness of the model to the target task, and further improving the quality of restoration.

[0124] Generating weights for deep features and the weights of the manual features ,use Represents the total number of categories, and the formula is as follows:

[0125] , ;

[0126] The above method obtains the feature weighted loss function based on the manual feature weight value, the deep feature weight value, the manual loss function and the deep loss function, which means that the feature weighted loss function is obtained based on the deep feature weight value. and manual feature weights , and the depth loss function And the manual loss function , to form a feature-weighted loss, thereby balancing the contribution of different features to model classification.

[0127] The above-mentioned total loss function is obtained based on the reconstruction loss function and the feature weighted loss function, which means that the reconstruction loss function Add the feature weighted loss function to get the total loss function , the formula is as follows:

[0128] ;

[0129] The above optimization of the convolutional neural network model according to the total loss function to obtain the infrared ship image restoration model refers to the reconstruction loss function The total loss function obtained by adding the feature weighted loss function The convolutional neural network model is optimized to obtain the infrared ship image restoration model.

[0130] In some embodiments, the manual feature data includes a manual feature vector and a manual loss function, and the deep feature data includes a deep feature vector and a deep loss function;

[0131] The convolutional neural network model is optimized according to the reconstruction loss function to obtain the infrared ship image restoration model, including:

[0132] Perform collaborative learning calculation based on manual feature vectors and deep feature vectors to obtain the information matching loss function;

[0133] Contribution allocation is calculated based on the manual feature vector and the deep feature vector to obtain the manual feature weight value and the deep feature weight value;

[0134] Obtain a feature weighted loss function according to the manual feature weight value, the deep feature weight value, the manual loss function and the deep loss function;

[0135] According to the reconstruction loss function, the information matching loss function and the feature weighted loss function, the total loss function is obtained;

[0136] The convolutional neural network model is optimized according to the total loss function to obtain the infrared ship image restoration model.

[0137] In this implementation, firstly, collaborative learning calculation is performed based on manual feature vectors and deep feature vectors to obtain information matching loss function; then contribution allocation calculation is performed based on manual feature vectors and deep feature vectors to obtain manual feature weight value and deep feature weight value; then feature weighted loss function is obtained based on manual feature weight value, deep feature weight value, manual loss function and deep loss function; then total loss function is obtained based on reconstruction loss function, information matching loss function and feature weighted loss function; finally, convolutional neural network model is optimized based on total loss function to obtain infrared ship image restoration model. The image restoration quality can be improved to the maximum extent.

[0138] The calculation process of the above information matching loss function and feature weighted loss function is the same as before and will not be described in detail here.

[0139] According to the reconstruction loss function, information matching loss function and feature weighted loss function, the total loss function is obtained. The specific formula is as follows:

[0140] ;

[0141] This embodiment can enhance the effectiveness and complementarity of feature expression by introducing matching loss function and feature weighted loss function at the same time. Through the collaborative loss mechanism, the two feature branches can be interactively optimized to strengthen information flow and feature sharing. In addition, combined with the attention mechanism, the weights are adaptively adjusted according to the importance of the features, and the feature contributions are reasonably distributed. Therefore, the image restoration quality can be maximized.

[0142] In some implementations, performing manual feature extraction on the first image data to obtain manual feature data may include:

[0143] Select at least two manual feature extraction operators;

[0144] Manual feature extraction is performed on the first image data using at least two manual feature extraction operators to obtain at least two types of manual feature data.

[0145] In this embodiment, manual feature extraction is performed on the first image data using at least two manual feature extraction operators. Using multiple manual feature extraction operators can significantly enhance the diversity and comprehensiveness of feature representation, and can further improve the restoration quality.

[0146] The above-mentioned selection of at least two manual feature extraction operators refers to using different manual feature extraction operators to perform manual feature extraction on the damaged infrared image, performing dimension unification operation on the extracted features and saving them. For example, three manual feature extraction operators, namely, FAST operator, SIFT operator and DoG operator, are used to perform manual feature extraction on the first image data to obtain three corresponding manual feature data.

[0147] Specifically, the FAST operator is an efficient key point detection method, mainly used to detect corner features; the SIFT operator focuses on detecting and describing local invariant features in images, and is highly robust to rotation, scaling, and illumination changes; the DoG operator constructs a scale space through the Gaussian difference method, and can effectively extract multi-scale feature points. These operators capture the geometric and texture features of images from different angles.

[0148] Compared with using only a single manual feature extraction operator, using multiple manual feature extraction operators can significantly enhance the diversity and comprehensiveness of feature representation. For example, FAST is good at quickly detecting corners, while SIFT and DoG focus more on the stability and description ability of features. By combining the strengths of multiple operators, the details and global information of the image can be better captured in different scenarios, the robustness of feature expression can be improved, and the quality of restoration can be further improved.

[0149] In some embodiments, the manual feature data includes a manual feature map, and the deep feature data includes a deep feature map;

[0150] The manual feature data is fused with the deep feature data to obtain fused features, which may include:

[0151] At least two manual feature maps are fused with the deep feature map to obtain a fused feature.

[0152] In this implementation, at least two manual feature maps are fused with the deep feature map to obtain fused features, which can further improve the restoration quality.

[0153] Specifically, if the number of manual features is n, the n manual features are used as the input of the manual feature encoder, and the manual feature auxiliary classifier is used to obtain n manual feature mappings. , and then the deep feature map With n manual feature maps The fusion is performed to obtain the fusion features.

[0154] The fusion of multiple manual feature extraction operators and deep features can significantly enhance the diversity and comprehensiveness of feature representation. For example, FAST is good at quickly detecting corners, while SIFT and DoG focus more on the stability and description ability of features. By combining the strengths of multiple operators, it is possible to better capture the details and global information of the image in different scenarios, improve the robustness of feature expression, and thus further improve the quality of restoration.

[0155] In some implementations, inputting the fusion features into a convolutional neural network model to repair the first image data to obtain the second image data may include:

[0156] The fused features are processed through the spatial Dropout layer and attention layer of the convolutional neural network model to generate fused feature representation;

[0157] The fused feature representation is sent to the decoder of the convolutional neural network model to repair the first image data to obtain the second image data.

[0158] In this embodiment, the fused features are first processed by the spatial dropout layer and the attention layer of the convolutional neural network model to generate a fused feature representation, and then the fused feature representation is sent to the decoder of the convolutional neural network model to repair the first image data to obtain the second image data. This can improve the repair efficiency.

[0159] The above-mentioned passing the fusion features through the spatial Dropout layer and the attention layer of the convolutional neural network model means passing through the dropout layer and then passing through the attention layer.

[0160] The following describes the specific implementation process of this application in detail using the fusion of deep features and multiple manual features as an example, including the following steps:

[0161] The given damaged infrared image and real image are preprocessed and converted into grayscale images as image data. The three manual feature extraction operators FAST, SIFT and DoG are used to extract features from the damaged infrared image, and the extracted features are dimensionally unified and saved in the form of files.

[0162] The processed image data is input into the deep feature encoder, and the deep feature vector is obtained through the deep feature auxiliary classifier. , deep feature map and class probability output of deep feature-assisted classifier , and calculate the depth loss ;

[0163] The n manual feature data corresponding to the image data are used as the input of the manual feature encoder, and the n manual feature vectors are obtained through the manual feature auxiliary classifier. , manual feature mapping and the class probability output of the hand-crafted feature-assisted classifier , and calculate the manual loss ;

[0164] Perform knowledge collaborative learning calculations on deep features and manual features, and calculate the information matching loss function between each manual feature and deep feature through KL divergence , to encourage collaborative learning between different features;

[0165] Calculate the contribution distribution of deep features and multiple manual features. First, concatenate the deep feature vectors and handcrafted feature vectors Get feature map , the importance difference between deep features and manual features is weighted through the channel attention mechanism to generate the weight of deep features and the weights of the manual features ;

[0166] Mapping the deep features With n manual feature maps The fused features are processed by the spatial Dropout layer and the attention layer to generate a fused feature representation, and the fused features are sent to the U-Net decoder to restore the image details through step-by-step upsampling to obtain the second image data;

[0167] Measure the difference between the second image data and the real image, and use the L1 norm to calculate the reconstruction loss function ;

[0168] According to the depth feature weight and manual feature weights , depth loss function And the manual loss function , to form a feature-weighted loss function, thereby balancing the contribution of different features to model classification;

[0169] The reconstruction loss function , information matching loss function Add the feature weighted loss function to obtain the total loss function ;

[0170] Get the total loss function Finally, the gradient of the loss function to the model parameters is calculated through the back-propagation algorithm, and the model parameters are adjusted through the optimizer to minimize the loss function. Through continuous iterative optimization, the model gradually improves the quality of image restoration and makes the restoration result closer to the real image. This process continues to cycle until the predetermined number of iterations is reached or the convergence condition is met.

[0171] This application uses a deep self-supervised learning model to extract infrared ship target features, combines traditional feature operators and deep learning methods, combines manual features from traditional methods with deep self-supervised model features, and uses a collaborative loss mechanism to allow manual features and deep features to work together. The invention has strong restoration robustness in complex scenes and extreme conditions, can stably restore infrared image details and improve overall restoration quality.

[0172] Based on the multi-feature fusion infrared ship image restoration method provided in the above embodiment, the present application also provides a specific implementation of the multi-feature fusion infrared ship image restoration device. Figure 2 As shown, a multi-feature fusion infrared ship image restoration device 200 includes:

[0173] The data acquisition module 201 is used to acquire first image data and real image data, wherein the first image data is an infrared image of a damaged ship, and the real image data is a real image corresponding to the infrared image of the ship;

[0174] A manual feature extraction module 202, used to perform manual feature extraction on the first image data to obtain manual feature data;

[0175] A depth feature extraction module 203 is used to extract depth features from the first image data to obtain depth feature data;

[0176] A feature fusion module 204 is used to fuse the manual feature data with the deep feature data to obtain a fused feature;

[0177] A first restoration module 205 is used to input the fusion feature into the convolutional neural network model to restore the first image data to obtain second image data;

[0178] A reconstruction loss module 206, used to obtain a reconstruction loss function according to the second image data and the real image data, wherein the reconstruction loss function is used to represent the difference between the restored image and the real image;

[0179] A model optimization module 207 is used to optimize the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model;

[0180] The second restoration module 208 is used to restore the infrared ship image according to the infrared ship image restoration model.

[0181] The multi-feature fusion infrared ship image restoration device 200 of the embodiment of the present application is used to execute the multi-feature fusion infrared ship image restoration method of the above embodiment. Its specific processing process is the same as the multi-feature fusion infrared ship image restoration method of the above embodiment, and will not be repeated here.

[0182] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0183] The electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0184] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0185] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 302 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 302 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 302 is a non-volatile solid-state memory.

[0186] In some embodiments, the memory 302 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0187] The processor 301 reads and executes the computer program instructions stored in the memory 302 to implement any one of the infrared ship image restoration methods with multi-feature fusion in the above embodiments.

[0188] In one example, the electronic device may further include a communication interface 303 and a bus 310. Figure 3 As shown, the processor 301, the memory 302, and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0189] The communication interface 303 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0190] Bus 310 includes hardware, software or both, and couples the components of online data traffic billing equipment to each other. For example, but not limitation, the bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnect (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. Where appropriate, bus 310 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0191] In addition, in combination with the infrared ship image restoration method of multi-feature fusion in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the infrared ship image restoration methods of multi-feature fusion in the above embodiment is implemented.

[0192] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0193] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0194] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0195] Aspects of the present disclosure are described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0196] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A multi-feature fusion infrared ship image restoration method, characterized in that: The following steps are involved: Acquire first image data and real image data, wherein the first image data is a damaged infrared ship image, and the real image data is a real image corresponding to the infrared ship image; Performing manual feature extraction on the first image data to obtain manual feature data; Performing depth feature extraction on the first image data to obtain depth feature data; Fusing the manual feature data with the deep feature data to obtain fused features; Inputting the fused features into a convolutional neural network model to repair the first image data to obtain second image data; Obtaining a reconstruction loss function according to the second image data and the real image data, wherein the reconstruction loss function is used to represent the difference between the restored image and the real image; Optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model; The infrared ship image is repaired according to the infrared ship image repair model.

2. The multi-feature fusion infrared ship image restoration method according to claim 1 is characterized in that: The manual feature data includes a manual feature vector, and the depth feature data includes a depth feature vector; The step of optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model includes: Perform collaborative learning calculation based on the manual feature vector and the deep feature vector to obtain an information matching loss function; Obtaining a total loss function according to the reconstruction loss function and the information matching loss function; The convolutional neural network model is optimized according to the total loss function to obtain an infrared ship image restoration model.

3. The multi-feature fusion infrared ship image restoration method according to claim 1 is characterized in that: The manual feature data includes a manual feature vector and a manual loss function, and the depth feature data includes a depth feature vector and a depth loss function; The step of optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model includes: Perform contribution allocation calculation according to the manual feature vector and the deep feature vector to obtain a manual feature weight value and a deep feature weight value; Obtaining a feature weighted loss function according to the manual feature weight value, the deep feature weight value, the manual loss function and the deep loss function; Obtaining a total loss function according to the reconstruction loss function and the feature weighted loss function; The convolutional neural network model is optimized according to the total loss function to obtain an infrared ship image restoration model.

4. The method for repairing infrared ship images by multi-feature fusion according to claim 1 is characterized in that: The manual feature data includes a manual feature vector and a manual loss function, and the deep feature data includes a deep feature vector and a deep loss function; The step of optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model includes: Perform collaborative learning calculation based on the manual feature vector and the deep feature vector to obtain an information matching loss function; Perform contribution allocation calculation according to the manual feature vector and the deep feature vector to obtain a manual feature weight value and a deep feature weight value; Obtaining a feature weighted loss function according to the manual feature weight value, the deep feature weight value, the manual loss function and the deep loss function; Obtaining a total loss function according to the reconstruction loss function, the information matching loss function and the feature weighted loss function; The convolutional neural network model is optimized according to the total loss function to obtain an infrared ship image restoration model.

5. The method for repairing infrared ship images by multi-feature fusion according to claim 1, characterized in that: The step of manually extracting features from the first image data to obtain manual feature data includes: Select at least two manual feature extraction operators; Manual feature extraction is performed on the first image data using at least two manual feature extraction operators to obtain at least two types of manual feature data.

6. The method for repairing infrared ship images by multi-feature fusion according to claim 5 is characterized in that: The manual feature data includes manual feature mapping, and the deep feature data includes at least two deep feature mappings corresponding one-to-one to the manual feature extraction operators; The fusing the manual feature data with the deep feature data to obtain a fused feature includes: At least two of the manual feature maps are fused with the deep feature map to obtain a fused feature.

7. The method for repairing infrared ship images by multi-feature fusion according to claim 1 is characterized in that: Inputting the fusion features into a convolutional neural network model to repair the first image data to obtain second image data, including: The fused features are processed through the spatial Dropout layer and attention layer of the convolutional neural network model to generate fused feature representation; The fused feature representation is sent to the decoder of the convolutional neural network model to repair the first image data to obtain the second image data.

8. A multi-feature fusion infrared ship image restoration device, characterized in that: include: A data acquisition module, used to acquire first image data and real image data, wherein the first image data is a damaged infrared ship image, and the real image data is a real image corresponding to the infrared ship image; A manual feature extraction module, used for performing manual feature extraction on the first image data to obtain manual feature data; A depth feature extraction module, used to extract depth features from the first image data to obtain depth feature data; A feature fusion module, used to fuse the manual feature data with the deep feature data to obtain a fused feature; A first restoration module, used for inputting the fusion features into a convolutional neural network model to restore the first image data to obtain second image data; A reconstruction loss module, used to obtain a reconstruction loss function according to the second image data and the real image data, wherein the reconstruction loss function is used to represent the difference between the restored image and the real image; A model optimization module, used for optimizing the convolutional neural network model according to the reconstruction loss function to obtain an infrared ship image restoration model; The second restoration module is used to restore the infrared ship image according to the infrared ship image restoration model.

9. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the infrared ship image restoration method of multi-feature fusion as described in any one of claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the infrared ship image restoration method with multi-feature fusion as described in any one of claims 1 to 7 is implemented.

Images