ISAR image quality evaluation method and system based on visual attention model

Abstract

The invention provides an ISAR image quality evaluation method and system based on a visual attention model, and relates to the technical field of image quality evaluation. According to the method, an ISAR image to be evaluated is processed through a pre-trained ISAR image quality evaluation neural network to obtain an evaluation result, and the ISAR image quality evaluation neural network comprises a Gramin Transformer model, a double-layer attention layer, an MLP layer and a sigmoid layer; wherein each layer of the double attention layers comprises a channel attention module and a region attention module. The ISAR image quality evaluation neural network designed by the invention focuses on the feature identifiability in the encoding process by using the feature similarity matrix, and better pays attention to the global and local features of the ISAR image by using the attention mechanism, and compared with the traditional features, the extracted features have the advantages of diversified expression dimensions, richer expression content, and higher accuracy. And finally, carrying out feature fusion on the channel dimension and the space dimension to obtain a robustness feature so as to evaluate the image quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image quality assessment, and in particular to an ISAR image quality assessment method and system based on a visual attention model. Background Art

[0002] With the development of radar imaging technology, ISAR imaging has been more widely used in target recognition. However, the quality of ISAR images seriously affects the ability to classify and recognize targets. Therefore, ISAR image performance evaluation is the premise and primary link of ISAR image application, and is one of the key technologies in the process of ISAR image target classification and recognition.

[0003] There are two main methods for evaluating the performance of ISAR images. One is the traditional evaluation method, which relies on extracting classic features. The second method simply uses deep neural networks, such as Alexnet. Traditional ISAR quality evaluation methods are mostly manual calibration or use pixel statistics to extract classic features such as mean, equivalent view count, contrast, etc. for performance evaluation. Manual calibration methods are labor-intensive and have accumulated errors in subjective evaluation. The evaluation indicators of traditional feature evaluation methods mainly focus on image resolution, noise, and contrast, while there is no good evaluation method for other important image features such as detail preservation and target shape accuracy, resulting in low accuracy of evaluation results. Simple convolutional neural networks cannot capture cross-regional connections in images, nor can they evaluate images at a global level, and do not analyze the particularity of ISAR images to make corresponding network designs. Due to the limited number of parameters, the fitting power is not strong, and the features between regions and channels are not further mined, the evaluation results are less accurate.

[0004] From the above description, it can be seen that the existing ISAR image performance evaluation method has low accuracy. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides an ISAR image quality evaluation method and system based on a visual attention model, which solves the technical problem of low accuracy of the existing ISAR image performance evaluation methods.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present invention provides an ISAR image quality assessment method based on a visual attention model, comprising:

[0010] Obtain an ISAR image to be evaluated, and divide the ISAR image to be evaluated into a number of patches;

[0011] Using several patches as input data of the pre-trained ISAR image quality assessment neural network, and outputting the assessment score of the ISAR image;

[0012] The pre-trained ISAR image quality assessment neural network includes: Gramin Transformer model, double-layer attention layer, MLP layer and sigmoid layer; each layer of the double-layer attention layer includes a channel attention module and a regional attention module; Gramin The Transformer model includes an Embedding layer, a multi-layer encoding layer and an extraction layer. The processing process of the ISAR image quality assessment neural network on the ISAR image includes: converting several patches into the format of the input data of the encoding layer through the Embedding layer; the multi-layer encoding layer encodes the data and outputs the image features. The extraction layer extracts B feature maps of different levels in the multi-layer encoding layer, and each feature map uses the feature similarity matrix to extract the score [token], and the score [token] is retained separately; the B feature maps are spliced ​​and input into the double-layer attention layer, and the attention mechanism is applied in the channel and spatial dimensions to capture the attention features across channels and regions; the attention features and score [token] are spliced ​​and input into the MLP layer, and the MLP layer outputs the prediction score of each patch and the total prediction score of the ISAR image. The prediction score of each patch and the total prediction score of the ISAR image are passed through the sigmoid layer to obtain the evaluation score of the ISAR image.

[0013] Preferably, the encoding layer includes layer normalization, multi-head attention mechanism, Dropout, and MLP modules;

[0014] In the encoding layer, each token input to the encoding layer is normalized through layer normalization to obtain a tokens sequence; the information learned from different head parts in the joint tokens sequence is combined through the multi-head attention mechanism to obtain fused features, which are processed by Dropout and MLP modules in turn to obtain feature maps.

[0015] Preferably, before segmenting the ISAR image to be evaluated, the method further includes: performing a cropping operation on the ISAR image to form an ISAR image with a dimension of H*W*C, wherein C represents a channel, H represents a height of the ISAR image input into the ISAR image quality evaluation neural network, and W represents a width of the ISAR image input into the ISAR image quality evaluation neural network.

[0016] Preferably, dividing the ISAR image to be evaluated into a plurality of patches comprises:

[0017] Cut the image with dimensions H*W*C into patches of size 8*8*C, forming (H / 8)*(W / 8) patches.

[0018] Preferably, the extraction layer includes a Gram matrix and Attention.

[0019] Preferably, the loss function of the pre-trained ISAR image quality assessment neural network during the training process includes:

[0020]

[0021]

[0022] Among them, MSE LOSS represents the square error loss; TSSC LOSS represents the texture and structure similarity constraint loss; N is the number of samples, is the predicted quality score, y i is the true label of the i-th ISAR image; μ ij is the predicted quality score of the jth patch of the i-th image, is the mean prediction quality score of all patches in the i-th ISAR image; L is the total number of patches.

[0023] In a second aspect, the present invention provides an ISAR image quality assessment system based on a visual attention model, comprising:

[0024] A data acquisition module is used to acquire the ISAR image to be evaluated and divide the ISAR image to be evaluated into a number of patches;

[0025] An evaluation module, which is used to use several patches as input data of a pre-trained ISAR image quality evaluation neural network and output an evaluation score of the ISAR image;

[0026] The pre-trained ISAR image quality assessment neural network includes: Gramin Transformer model, double-layer attention layer, MLP layer and sigmoid layer; each layer of the double-layer attention layer includes a channel attention module and a regional attention module; Gramin The Transformer model includes an Embedding layer, a multi-layer encoding layer and an extraction layer. The processing process of the ISAR image quality assessment neural network on the ISAR image includes: converting several patches into the format of the input data of the encoding layer through the Embedding layer; the multi-layer encoding layer encodes the data and outputs the image features. The extraction layer extracts B feature maps of different levels in the multi-layer encoding layer, and each feature map uses the feature similarity matrix to extract the score [token], and the score [token] is retained separately; the B feature maps are spliced ​​and input into the double-layer attention layer, and the attention mechanism is applied in the channel and spatial dimensions to capture the attention features across channels and regions; the attention features and score [token] are spliced ​​and input into the MLP layer, and the MLP layer outputs the prediction score of each patch and the total prediction score of the ISAR image. The prediction score of each patch and the total prediction score of the ISAR image are passed through the sigmoid layer to obtain the evaluation score of the ISAR image.

[0027] Preferably, the loss function of the pre-trained ISAR image quality assessment neural network during the training process includes:

[0028]

[0029] Among them, MSE LOSS represents the square error loss; TSSC LOSS represents the texture and structure similarity constraint loss; N is the number of samples, is the predicted quality score, y i is the true label of the i-th ISAR image; μ ij is the predicted quality score of the jth patch of the i-th image, is the mean prediction quality score of all patches in the i-th ISAR image; L is the total number of patches.

[0030] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program for ISAR image quality assessment based on a visual attention model, wherein the computer program enables a computer to execute the ISAR image quality assessment method based on a visual attention model as described above.

[0031] In a fourth aspect, the present invention provides an electronic device, comprising:

[0032] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the ISAR image quality assessment method based on the visual attention model as described above.

[0033] (III) Beneficial effects

[0034] The present invention provides an ISAR image quality assessment method and system based on a visual attention model. Compared with the prior art, the method has the following beneficial effects:

[0035] The ISAR image quality assessment neural network of the present invention comprises: a Gramin Transformer model, a double-layer attention layer, an MLP layer and a sigmoid layer; wherein each layer of the double-layer attention layer comprises a channel attention module and a regional attention module; the Gramin The Transformer model includes an Embedding layer, a multi-layer encoding layer and an extraction layer. The processing process of the ISAR image quality assessment neural network on the ISAR image includes: converting several patches into the format of the input data of the encoding layer through the Embedding layer; the multi-layer encoding layer encodes the data and outputs the image features. The extraction layer extracts B feature maps of different levels in the multi-layer encoding layer, and each feature map uses the feature similarity matrix to extract the score [token], and the score [token] is retained separately; the B feature maps are spliced ​​and input into the double-layer attention layer, and the attention mechanism is applied in the channel and spatial dimensions to capture the attention features across channels and regions; the attention features and score [token] are spliced ​​and input into the MLP layer, and the MLP layer outputs the prediction score of each patch and the total prediction score of the ISAR image. The prediction score of each patch and the total prediction score of the ISAR image are passed through the sigmoid layer to obtain the evaluation score of the ISAR image. The ISAR image quality evaluation neural network designed in the present invention uses the feature similarity matrix to focus on the feature identifiability in the encoding process, and uses the attention mechanism to better focus on the global and local features of the ISAR image. Compared with traditional features, the extracted features have diversified expression dimensions and richer expression content. Finally, the channel dimension and the spatial dimension are fused to obtain robust features to evaluate the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 The overall structure diagram of the neural network for ISAR image quality evaluation;

[0038] Figure 2 This is the specific structure diagram of the transformer encoder;

[0039] Figure 3 Schematic diagram of the dropout module in the transformer encoder;

[0040] Figure 4 Schematic diagram of the MLP module in the transformer encoder;

[0041] Figure 5 It is a schematic diagram of the structure of the channel attention module;

[0042] Figure 6 Schematic diagram of the structure of the regional attention module. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The embodiments of the present application provide an ISAR image quality evaluation method and system based on a visual attention model, thereby solving the technical problem of low accuracy of existing ISAR image performance evaluation methods, achieving better attention to the global and local features of ISAR images and improving the accuracy of evaluation.

[0045] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0046] The ISAR image quality assessment neural network designed in the embodiment of the present invention focuses on the contribution of coding features to quality scores during feature extraction, uses feature similarity to distinguish scores of different qualities, fully explores the relationship between quality scores and image features, pays better attention to the global and local features of ISAR images, and the extracted features have diversified expression dimensions and richer expression content compared with traditional features. The attention mechanism is further used to fuse features from the channel dimension and the spatial dimension to obtain robust features to accurately evaluate the image quality.

[0047] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0048] An embodiment of the present invention provides an ISAR image quality assessment method based on a visual attention model, the method comprising:

[0049] Obtain an ISAR image to be evaluated, and divide the ISAR image to be evaluated into a number of patches;

[0050] Using several patches as input data of the pre-trained ISAR image quality assessment neural network, and outputting the assessment score of the ISAR image;

[0051] The pre-trained ISAR image quality assessment neural network includes: Gramin Transformer model, double-layer attention layer, MLP layer and sigmoid layer; each layer of the double-layer attention layer includes a channel attention module and a regional attention module; Gramin The Transformer model includes an Embedding layer, a multi-layer encoding layer and an extraction layer. The processing process of the ISAR image quality assessment neural network on the ISAR image includes: converting several patches into the format of the input data of the encoding layer through the Embedding layer; the multi-layer encoding layer encodes the data and outputs the image features. The extraction layer extracts B feature maps of different levels in the multi-layer encoding layer, and each feature map uses the feature similarity matrix to extract the score [token], and the score [token] is retained separately; the B feature maps are spliced ​​and input into the double-layer attention layer, and the attention mechanism is applied in the channel and spatial dimensions to capture the attention features across channels and regions; the attention features and score [token] are spliced ​​and input into the MLP layer, and the MLP layer outputs the prediction score of each patch and the total prediction score of the ISAR image. The prediction score of each patch and the total prediction score of the ISAR image are passed through the sigmoid layer to obtain the evaluation score of the ISAR image.

[0052] In the specific implementation process, the specific structure of the pre-trained ISAR image quality evaluation neural network is as follows: Figure 1As shown, it includes a Gram-T Transformer model, a double-layer attention layer, an MLP layer, and a sigmoid layer; the Gram-T Transformer model includes three parts, among which the first part is the Linear Projection of FlattenedPatches, also known as the Embedding layer, which is mainly used to convert the input image data into a form suitable for Transformer structure processing. The second part is the Transformer Encoder (encoding layer), which is the core of the entire Gram-T model. This part mainly includes layer normalization (Layer Norm), multi-head attention mechanism (Multi-Head Attention), Dropout, MLP Block, which is used to learn the features of the input image data. The third part is the extraction layer, including the Gram matrix and Attention, which is used to extract the score [token] in each feature map. Each layer of the double-layer attention layer includes a channel attention module and a regional attention module.

[0053] The Gram-T Transformer model includes multiple encoding layers (the embodiment of the present invention takes 12 layers as an example), extracts the feature graphs of the {7th, 8th, 9th, 10th}th layers of Gram-T, that is, B=4, and extracts score[token] from each feature graph using the feature similarity matrix, and directly adds it to the position token of the original feature graph to obtain a new feature graph. After Concat, F∈R B ×C×HW As output, it enters the double-layer attention layer.

[0054] The training process of the ISAR image quality evaluation neural network is as follows:

[0055] A total of 324 ISAR images were obtained by using simulation software and image collection. A total of 324 ISAR images were obtained by using simulation software and image collection. The training set and the test set were divided into a ratio of 8:2, with about 260 images in the training set and about 64 images in the test set. Preferably, the processing of training data also includes obtaining images and corresponding label content, and the processing of training data to a size that meets the requirements of the network structure model is to first scale the image to a size slightly larger than the size required by the network structure model, and then randomly crop it to the size required by the ISAR image quality evaluation neural network, and then randomly flip it to enhance the data. The ISAR image quality evaluation neural network is trained and optimized using the training set and the test set. Among them, in the ISAR image quality evaluation neural network, the specific processing process of the ISAR image is as follows:

[0056] S1. Crop the ISAR image to form a feature map of dimension 224×224×3, representing 3 channels. The height and width of the feature map are 224. Cut it into patches of size 8*8*3. After division, (224 / 8)*(224 / 8)=784 Patches will be obtained.

[0057] S2. Use a convolution layer with a kernel size of 8×8, a stride of 8, and 768 kernels to implement linear mapping for each segment. The shape of the convolution operation changes to [224, 224, 3]→[28, 28, 768]. Then flatten the two dimensions H and W, and the shape changes to ([28, 28, 768]→[784, 768]). At this time, it becomes a two-dimensional matrix called Patches Embedding, which meets the requirements of Transformer input. This vector of length 768 is called a token.

[0058] S3. Insert a one-dimensional position code into the token. The shape of the Patches Embedding after concatenation is [785, 768] (where Patches Embedding includes position embedding, and the position embedding dimension is 1×768). Send the Patches Embedding into the transformer encoder, pass through the normalization layer, normalize each token, calculate the mean and variance of 785*768, and process the feature points of each sample as follows Where γ and β are affine transformation parameters, and ε is the value added to the denominator to ensure numerical stability. The specific structure of transformer encoder is as follows Figure 2 As shown in the figure, it mainly consists of four parts: layer normalization (Layer Norm), multi-head attention mechanism (Multi-Head Attention), Dropout, and MLP Block, which are used to learn the features of input image data.

[0059] S4. Sequence R of tokens obtained in the previous step N×L×S Input into the multi-head attention mechanism, and pass the token through three change matrices W q , W k , W v (These three parameters are trainable and shared, with a size of S×S), and the corresponding q, k, and v vector sequences are recorded as Q, K, and V, and the size of each vector sequence is R L×S, each query in Q will be matched with each key in K one by one, and the correlation between the two will be calculated. The greater the correlation, the greater the weight of v. For the weight of each v, the final result is then weighted.

[0060]

[0061] For Multi-Head Attention, the multi-head attention mechanism can combine the information learned from different heads. The same method as in Self-Attention can be used for each head to get the corresponding result. The W used by each head q , W k , W v The matrices are different.

[0062] head i =Attention(Q i , K i , V i )

[0063] Where i = 1...h, h is the number of heads (h = 6 in the embodiment of the present invention). Finally, the results obtained from each head are concat-joined, and then the concatenated results are passed through W O (learnable parameters) are fused to obtain the final result, i.e., the fusion feature, formula:

[0064] Multihead(Q,K,V)=Concat(head1,head2,...head h )W o

[0065] S5. When learning deep learning models, the model is overfitted due to too many parameters of the transformer encoder and too few samples. Dropout can alleviate overfitting to a certain extent and can achieve regularization to a certain extent, such as Figure 3 The basic principle is: in the process of forward propagation, the activation value of a neuron is made inoperative with a probability of p, which can be seen in the figure below. The neuron that stops working is represented by a dotted line, and the corresponding propagation process connected to the neuron will no longer exist. This makes the model more general because it does not depend on some local features.

[0066] The output of S6 and Dropout passes through the MLP Block. The specific structure of the MLP Block is as follows: Figure 4As shown in the figure, the MLP Block consists of a fully connected layer, a GELU activation function, and a Dropout. The first fully connected layer quadruples the number of input nodes (785, 768) → (785, 3072), and the second fully connected layer restores the original number of nodes (785, 3072) → (785, 768).

[0067] S7. Set the number of data batches to B (in the embodiment of the present invention, B=4), extract the feature graphs of the {7, 8, 9, 10}th layers of Gram-T, and in the extraction layer, use the feature similarity matrix to extract the score [token] of each feature graph. This is a token specifically used for quality scoring. This [score] token is a trainable parameter. The data format is a vector of length 768, which also represents that the scoring criteria are divided into 768 levels. Feature graph X∈R 4×785×768 Mapped to V X , V X =XW C , W C ∈R 768×768 . Gram matrix Gram X =Vec(V X T V X )W g , Vec represents instance vectorization, Wg g ∈R 768×768 .

[0068] S8. After concat four feature maps, we get a feature map of dimension 4×785×3072. We temporarily retain score[token] and take the feature map of 4×784×3072 and input it into the double-layer attention layer (each layer of the double-layer attention layer includes a channel attention module and a regional attention module) to perform channel and spatial attention mining. The feature map of 4×784×3072 is reranged to 4×3072×785. In conventional Self-Attention (SA), the key-query dot product operation can make the patches in the image interact with each other, thereby extracting the features of the spatial dimension. However, the information contained in the dimension of C in C×HW, that is, the channel, is ignored. The function of the Channel Attention Block (CAB) in the embodiment of the present invention is to apply SA to the dimension of the channel (C), and transform K into HW×C through Reshaping so as to perform matrix multiplication with Q. Then, the correlation between channels can be obtained through Softmax (which can be understood as a covariance matrix with a size of C×C), and finally weighted with V for output. At this time, the feature dimension is 4×3072×784, and then it is realigned to 4×3072×28×28, and then it is transformed into 4×768×28×28 through a layer of convolutional layer and sent to the next module. The structural diagram of the Channel Attention Block is shown in the figure below. Figure 5 shown.

[0069] The feature maps obtained by S9 and S8 are convolved to a dimension of 4×768×28×28, and the range is 4×784×768, which are input to the Inter Region Attention Block (IRAB). Since ISAR images are radar images of strong scattering points of targets, they are mostly sparse distributions of strong scattering centers. The evaluation of their quality does not only rely on a single block but also needs to consider whether their global distribution fully describes the target structure and characteristics. Therefore, the IRAB module is designed to extract cross-region features and obtain the global scattering center distribution characteristics. The structure of the Inter Region Attention Block is shown in the figure. Figure 6 As shown in the figure, the IRAB module consists of three submodules, namely IRL, Conv, and Coef. IRL is the Inter Region Layer, which adopts a cross-region attention enhancement mechanism for the 4×784×768-dimensional features. Each patch interacts with the other patches across spatial dimensions to enhance the feature extractor's cross-region collaborative perception of ISAR images. Conv is convolution, and Coef is a scaling parameter α. The formula is as follows. It represents the output feature of the i-th stage after two IRLs, with a feature dimension of 4×784×768, which is then transformed into 4×768×28×28. After a convolution layer and a relu layer, the feature 4×768×28×28 is obtained, which is multiplied by the Coef coefficient of 0.8 and the initial Add together to get the cross-spatial feature F representing the ISAR image out .

[0070]

[0071] In the embodiment of the present invention, the 4×784×3072 feature map passes through the CAB and IRAB modules twice. When it is input to the CAB module for the second time, the output feature map of CAB Figure 4 ×768×28×28, after a convolution layer with an output channel number of 768, a feature map of 4×768×28×28 is obtained, which is input to the IRAB module. The final feature map is 4×768×28×28, and the rerange is 4×784×768. 4×784×768 is the attention feature. The attention feature and score[token] are concatenated to obtain a feature dimension of 4×785×768, which is then input to the MLP layer and outputs a feature dimension of 4×785×1, which represents the prediction score of each patch and the total prediction score of the ISAR image. The prediction score of each patch and the total prediction score of the ISAR image are obtained through the sigmoid layer to obtain the evaluation score of the ISAR image. This method refines the evaluation of the image patch and makes the overall evaluation of the image more accurate.

[0072] In the embodiment of the present invention, it is considered that the ISAR image is a high-resolution radar image obtained by using microwave data, and is a two-dimensional projection of a three-dimensional target on the range-Doppler plane. The targets in the ISAR image exist in the form of isolated, sparse strong scattering center distribution, and each strong scattering center reflects the electromagnetic scattering structure of the target. Therefore, the ISAR image has structural similarity. At the same time, the imaging effect of the target in the ISAR image within a certain angle range can remain relatively stable, so it has texture similarity. Texture and structural similarity constrains different patches of an image to have similar quality scores. Therefore, in view of the characteristics of ISAR image quality evaluation, a loss function of the ISAR image quality evaluation neural network in the training process is specially designed, which specifically includes:

[0073] ① MSE LOSS, N is the number of samples, is the predicted quality score, y i is the true label of the i-th ISAR image:

[0074]

[0075] ② Texture and Structure Similarity Constraints, N is the number of samples, μ ij is the predicted quality score of the jth patch of the i-th image, The mean quality score of all patches predicted for the i-th ISAR image is:

[0076]

[0077] The final loss function is:

[0078]

[0079] Where L is the total number of patches.

[0080] The embodiment of the present invention further provides an ISAR image quality evaluation system based on a visual attention model, comprising:

[0081] A data acquisition module is used to acquire the ISAR image to be evaluated and divide the ISAR image to be evaluated into a number of patches;

[0082] An evaluation module, which is used to use several patches as input data of a pre-trained ISAR image quality evaluation neural network and output an evaluation score of the ISAR image;

[0083] The pre-trained ISAR image quality assessment neural network includes: Gramin Transformer model, double-layer attention layer, MLP layer and sigmoid layer; each layer of the double-layer attention layer includes a channel attention module and a regional attention module; Gramin The Transformer model includes an Embedding layer, a multi-layer encoding layer and an extraction layer. The processing process of the ISAR image quality assessment neural network on the ISAR image includes: converting several patches into the format of the input data of the encoding layer through the Embedding layer; the multi-layer encoding layer encodes the data and outputs the image features. The extraction layer extracts B feature maps of different levels in the multi-layer encoding layer, and each feature map uses the feature similarity matrix to extract the score [token], and the score [token] is retained separately; the B feature maps are spliced ​​and input into the double-layer attention layer, and the attention mechanism is applied in the channel and spatial dimensions to capture the attention features across channels and regions; the attention features and score [token] are spliced ​​and input into the MLP layer, and the MLP layer outputs the prediction score of each patch and the total prediction score of the ISAR image. The prediction score of each patch and the total prediction score of the ISAR image are passed through the sigmoid layer to obtain the evaluation score of the ISAR image.

[0084] It can be understood that the ISAR image quality assessment system based on the visual attention model provided in the embodiment of the present invention corresponds to the above-mentioned ISAR image quality assessment method based on the visual attention model. The explanations, examples, beneficial effects and other parts of the relevant contents can refer to the corresponding contents in the ISAR image quality assessment method based on the visual attention model, and will not be repeated here.

[0085] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program for ISAR image quality assessment based on a visual attention model, wherein the computer program enables a computer to execute the ISAR image quality assessment method based on a visual attention model as described above.

[0086] An embodiment of the present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the ISAR image quality assessment method based on the visual attention model as described above.

[0087] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0088] 1. The ISAR image quality evaluation neural network designed in the embodiment of the present invention uses the feature similarity matrix to focus on the feature identifiability in the encoding process, and uses the attention mechanism to better focus on the global and local features of the ISAR image. Compared with traditional features, the extracted features have diversified expression dimensions and richer expression content. Finally, the channel dimension and the spatial dimension are feature fused to obtain robust features to evaluate the image quality.

[0089] 2. In the embodiment of the present invention, a Grammi matrix is ​​designed on the basis of a general transformer to calculate feature similarity. The correlation between image quality information and features in the feature extraction process is strengthened through the feature similarity matrix, rather than indirect optimization through the final fully connected layer. This is beneficial to improving the representation of image quality by features and enhancing the identifiability and expressiveness of features.

[0090] 3. The attention modules are Channel Attention Block (CAB) and Inter Region Attention Block (IRAB). These two modules apply attention mechanisms in the channel and spatial dimensions respectively: the attention module for the spatial dimension is used to capture long semantic association information on the position; the attention module for the channel dimension is used to capture long semantic association information on the channel. In this multi-dimensional way, the modules can synergistically enhance the interaction between the global and local regions, thereby improving the accuracy of the algorithm.

[0091] 4. The predicted score of the image is the sum of the predicted scores of each patch and the score predicted by the model encoding process, and the normalized score is obtained through the sigmoid layer. This method refines the evaluation of the image patch and makes the overall evaluation of the image more accurate.

[0092] 5. Based on the texture and structural similarity of ISAR image imaging, the corresponding loss function was designed to obtain the robustness characteristics, and finally the ISAR image quality was evaluated.

[0093] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating ISAR image quality based on a visual attention model, characterized in that: include: Obtain an ISAR image to be evaluated, and divide the ISAR image to be evaluated into a number of patches; Using several patches as input data of the pre-trained ISAR image quality assessment neural network, and outputting the assessment score of the ISAR image; The pre-trained ISAR image quality assessment neural network includes: Gramin Transformer model, double-layer attention layer, MLP layer and sigmoid layer; each layer of the double-layer attention layer includes a channel attention module and a regional attention module; Gramin The Transformer model includes an Embedding layer, a multi-layer encoding layer and an extraction layer. The processing process of the ISAR image quality assessment neural network on the ISAR image includes: converting several patches into the format of the input data of the encoding layer through the Embedding layer; the multi-layer encoding layer encodes the data and outputs the image features. The extraction layer extracts B feature maps of different levels in the multi-layer encoding layer, and each feature map uses the feature similarity matrix to extract the score [token], and the score [token] is retained separately; the B feature maps are spliced ​​and input into the double-layer attention layer, and the attention mechanism is applied in the channel and spatial dimensions to capture the attention features across channels and regions; the attention features and score [token] are spliced ​​and input into the MLP layer, and the MLP layer outputs the prediction score of each patch and the total prediction score of the ISAR image. The prediction score of each patch and the total prediction score of the ISAR image are passed through the sigmoid layer to obtain the evaluation score of the ISAR image.

2. The ISAR image quality assessment method based on the visual attention model according to claim 1, characterized in that: The encoding layer includes layer normalization, multi-head attention mechanism, Dropout, and MLP modules; In the encoding layer, each token input to the encoding layer is normalized through layer normalization to obtain a tokens sequence; the information learned from different head parts in the joint tokens sequence is combined through the multi-head attention mechanism to obtain fused features, which are processed by Dropout and MLP modules in turn to obtain feature maps.

3. The ISAR image quality assessment method based on the visual attention model according to claim 1, characterized in that: Before segmenting the ISAR image to be evaluated, the method further includes: performing a cropping operation on the ISAR image to form an ISAR image with a dimension of H*W*C, wherein C represents a channel, H represents a height of the ISAR image input into the ISAR image quality evaluation neural network, and W represents a width of the ISAR image input into the ISAR image quality evaluation neural network.

4. The ISAR image quality assessment method based on the visual attention model according to claim 3, characterized in that: The ISAR image to be evaluated is divided into several patches, including: Cut the image with dimensions H*W*C into patches of size 8*8*C, forming (H / 8)*(W / 8) patches.

5. The ISAR image quality assessment method based on visual attention model according to claim 1, characterized in that: The extraction layer includes a Gram matrix and Attention.

6. The ISAR image quality assessment method based on a visual attention model according to any one of claims 1 to 5, characterized in that: The loss function of the pre-trained ISAR image quality assessment neural network during the training process includes: Among them, MSE LOSS represents the square error loss; TSSC LOSS represents the texture and structure similarity constraint loss; N is the number of samples, is the predicted quality score, y i is the true label of the i-th ISAR image; μ ij is the predicted quality score of the jth patch of the i-th image, is the mean prediction quality score of all patches in the i-th ISAR image; L is the total number of patches.

7. An ISAR image quality assessment system based on a visual attention model, characterized in that: include: A data acquisition module is used to acquire the ISAR image to be evaluated and divide the ISAR image to be evaluated into a number of patches; An evaluation module, which is used to use several patches as input data of a pre-trained ISAR image quality evaluation neural network and output an evaluation score of the ISAR image; The pre-trained ISAR image quality assessment neural network includes: Gramin Transformer model, double-layer attention layer, MLP layer and sigmoid layer; each layer of the double-layer attention layer includes a channel attention module and a regional attention module; Gramin The Transformer model includes an Embedding layer, a multi-layer encoding layer and an extraction layer. The processing process of the ISAR image quality assessment neural network on the ISAR image includes: converting several patches into the format of the input data of the encoding layer through the Embedding layer; the multi-layer encoding layer encodes the data and outputs the image features. The extraction layer extracts B feature maps of different levels in the multi-layer encoding layer, and each feature map uses the feature similarity matrix to extract the score [token], and the score [token] is retained separately; the B feature maps are spliced ​​and input into the double-layer attention layer, and the attention mechanism is applied in the channel and spatial dimensions to capture the attention features across channels and regions; the attention features and score [token] are spliced ​​and input into the MLP layer, and the MLP layer outputs the prediction score of each patch and the total prediction score of the ISAR image. The prediction score of each patch and the total prediction score of the ISAR image are passed through the sigmoid layer to obtain the evaluation score of the ISAR image.

8. The ISAR image quality assessment system based on visual attention model according to claim 7, characterized in that: The loss function of the pre-trained ISAR image quality assessment neural network during the training process includes: Among them, MSE LOSS represents the square error loss; TSSC LOSS represents the texture and structure similarity constraint loss; N is the number of samples, is the predicted quality score, y i is the true label of the i-th ISAR image; μ ij is the predicted quality score of the jth patch of the i-th image, is the mean prediction quality score of all patches in the i-th ISAR image; L is the total number of patches.

9. A computer-readable storage medium, characterized in that: The computer program for ISAR image quality assessment based on a visual attention model is stored therein, wherein the computer program enables a computer to execute the ISAR image quality assessment method based on a visual attention model as claimed in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the ISAR image quality assessment method based on the visual attention model according to any one of claims 1 to 6.

Images