A deep network feature improvement method based on global similarity

Abstract

The present invention provides a method for improving deep network features based on global similarity, comprising: dividing an input image into grid-like image blocks, obtaining a feature tensor of the image block based on an input feature tensor; simplifying the number of two feature channels; transforming the dimensions of the two feature tensors to obtain a feature matrix; calculating the similarity between each pixel feature and each image block feature based on the feature matrix; normalizing the similarity to obtain a similarity weight; calculating weighted features for each pixel, transforming the matrix dimensions to obtain a weighted feature tensor; converting the simplified feature channel number in the weighted feature tensor into the initial feature channel number; and calculating the improved feature tensor. The present invention improves the input features of the deep network based on the pairwise similarity between pixels and image blocks, fully considering the dependency between features in different regions on the entire image, and is helpful to improve the expressiveness of deep network features and improve the performance of deep networks in image recognition.

Description

Technical Field

[0001] The invention relates to a deep network feature improvement method based on global similarity, and belongs to the technical field of image processing. Background Art

[0002] Convolutional neural networks have been widely used in the fields of computer vision and image processing, and have greatly improved the performance of image recognition. Deep convolutional neural networks can more effectively extract high-level semantic features by increasing the depth of the network, thereby further improving the performance of image recognition. There are many research works based on deep convolutional neural networks for computer vision and image processing.

[0003] In deep networks, the basic module is the convolution module. The convolution operation is usually calculated based on a small convolution kernel and a small local image sliding window. Therefore, the long-distance dependencies in the global range of the image can only be obtained indirectly through repeated convolution operations. Repeated local operations can easily blur the deep features of the image, causing the features of some slender objects to disappear in the deep layer. Summary of the invention

[0004] To solve the above problems, the present invention provides a deep network feature improvement method based on global similarity, which improves the input features of the deep network based on the pairwise similarity between pixels and image blocks. It can fully consider the dependency between features in different regions of the entire image, help to improve the expressiveness of deep network features, and improve the performance of deep networks in image recognition.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for improving deep network features based on global similarity includes the following steps:

[0007] S1: Divide the input image into grid-shaped image blocks, and obtain the feature tensor of the image block based on the input feature tensor; simplify the feature channels of the feature tensor of the input image and the feature tensor of the image block; transform the dimension of the feature tensor of the input image and the feature tensor of the image block to obtain the feature matrix of the input image and the image block;

[0008] S2: Based on the feature matrix of the input image and the image block, the similarity is calculated between each pixel feature and each image block feature;

[0009] S3: normalize the similarity to obtain the similarity weight;

[0010] S4: Calculate the weighted features for each pixel point, with the weight being the similarity weight calculated in the previous step; perform matrix dimensionality transformation on the weighted features to obtain a weighted feature tensor; change the reduced number of feature channels in the weighted feature tensor back to the original number of feature channels.

[0011] S5: The improved feature tensor is equal to the sum of the transformed weighted feature tensor and the original input feature tensor.

[0012] Further, in step S1, the feature tensor of the input image is F ∈ R CxHxW , where C represents the number of feature channels, H represents the height of the input image, W represents the width of the input image, and R CxHxW represents a real - valued feature tensor with three dimensions; the feature tensor of the image patch is P ∈ R CxGxG , where G represents the number of image patches in terms of height and width, G ≤ H and G ≤ W; the reduction of the number of feature channels uses a 1x1 convolutional kernel, and the number of feature channels is reduced from C to C', where C' < C; in the feature matrices of the image and the image patch, the feature of an element in the input image is represented by F i , 1 ≤ i ≤ HW, and the feature of an element in the image patch is represented by P j , 1 ≤ j ≤ G 2 .

[0013] Further, in step S2, the similarity is calculated through the following formula:

[0014] Sim(F i , P j ) = F i P j T

[0015] From this, the similarity matrix is obtained

[0016] Further, in step S3, the softmax function is used for normalization, and the obtained similarity weight matrix M is expressed as:

[0017] M = softmax(Sim).

[0018] Further, in step S4, the weighted feature of a pixel point is equal to the weighted sum of the features of all similar image patches, and the calculation formula is as follows:

[0019]

[0020] The process of changing back to the original number of feature channels uses a 1x1 convolutional kernel, and the number of feature channels C' is changed back to the original number of feature channels C.

[0021] Further, step S5 is calculated through the following formula:

[0022] F refined =F weighted +F

[0023] Among them, F refined is the improved feature tensor, F weighted is the transformed weighted feature tensor, and F is the initial input feature tensor.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] The method of the present invention explicitly represents the long-distance dependency relationship in the global scope of the image, calculates the global similarity, and uses the features of all similar image blocks in the global scope to improve the features of a certain image pixel. The present invention fully considers the global feature dependency relationship, improves the feature expression ability of the deep network, and helps to improve the image recognition performance based on the deep network. When the deep network feature improvement method provided by the present invention is applied to the research on the hidden crack defect recognition algorithm in the image, it can effectively improve the average precision (Average Precision, AP) of hidden crack defect recognition, the area under the ROC curve (Area Under Curve, AUC), F1 value (an indicator that comprehensively considers the precision and recall rate), balanced accuracy (Balanced Accuracy) and accuracy (Accuracy). BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the overall flow of an embodiment of the present invention.

[0027] Figure 2 This is a performance comparison diagram of PR curves and ROC curves on four benchmark networks when the present invention is applied to the problem of image hidden crack defect recognition. DETAILED DESCRIPTION

[0028] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0029] This embodiment provides a method for improving deep network features based on global similarity, the process of which is as follows: Figure 1 As shown, the following steps are included:

[0030] S1: Divide the input image into grid-like image blocks and calculate the feature tensor F∈R based on the input image. CxHxW (C represents the number of feature channels, H represents the height of the input image, W represents the width of the input image, R CxHxWObtain the feature tensor \(P\in\mathbb{R}\) of the image patch, which represents the real-valued feature tensor with three dimensions CxGxG (where \(G\) represents the number of image patches in height and width, \(G\leq H\) and \(G\leq W\)); use a \(1\times1\) convolutional kernel to reduce the number of feature channels of the feature tensor of the input image and the feature tensor of the image patch, so that the number of feature channels changes from \(C\) to \(C'\) (\(C'<C\)); reshape the feature tensors of the input image and the image patch to obtain the feature matrices of the input image and the image patch: the feature of an element in the input image is represented by \(F\) i (\(1\leq i\leq HW\)), and the feature of an element in the image patch is represented by \(P\) j (\(1\leq j\leq G\) 2 ).

[0031] S2: Based on the feature matrices of the input image and the image patch, calculate the similarity between each pixel feature and each image patch feature through the following formula:

[0032] \(\text{Sim}(F\) i , \(P\) j ) \(=F\) i \(P\) j T

[0033] According to the above formula, we will obtain a similarity matrix

[0034] S3: Use the softmax function to normalize the similarity to obtain the similarity weight matrix \(M\);

[0035] \(M = \text{softmax}(\text{Sim})\)

[0036] S4: Calculate the weighted feature for each pixel point. The weighted feature of a pixel point is equal to the weighted sum of all similar image patch features. The calculation formula is as follows:

[0037]

[0038] The weight is the similarity weight calculated in the previous step; reshape the weighted feature to obtain the weighted feature tensor; use a \(1\times1\) convolutional kernel to change the reduced number of feature channels \(C'\) in the weighted feature tensor back to the initial number of feature channels \(C\);

[0039] S5: The improved feature tensor \(F\) refined is equal to the sum of the transformed weighted feature tensor \(F\) weighted and the initial input feature tensor \(F\). The formula is as follows:

[0040] \(F\) refined \(=F\) weighted \(+F\)

[0041] In order to verify the effectiveness of the deep network feature improvement method based on global similarity provided by the present invention, the method is used to improve the deep network features and applied to the hidden crack defect recognition algorithm in the image.

[0042] Here, experiments were conducted on four benchmark networks (VGG16, ResNet56, DenseNet40 and GoogleNet) using Average Precision (AP), Area Under Curve (AUC), F1 value (an indicator that comprehensively considers precision and recall), Balanced Accuracy and Accuracy. The method of the present invention is referred to as GSN. The results are shown in Figure 2. Figure 2 As can be seen from Table 1 below, it is obvious that the deep network feature improvement method provided by the present invention further improves the performance of hidden crack defect recognition, indicating that the present invention can improve the feature expression ability of the deep network.

[0043] Table 1 AP, AUC, F1, balanced accuracy (B-Accuracy) and accuracy performance of different models

[0044] Model AP AUC F1 B-Accuracy Accuracy VGG16 90.86％ 93.58％ 80.66％ 85.85％ 87.98％ VGG16+proposed GSN 92.09％ 94.57％ 82.45％ 87.31％ 89.01％ ResNet56 88.97％ 92.22％ 70.72％ 80.17％ 76.43％ ResNet56+proposed GSN 92.37％ 94.69％ 81.14％ 87.43％ 87.07％ DenseNet40 85.34％ 89.63％ 69.15％ 78.57％ 76.17％ DenseNet40+proposed GSN 92.98％ 95.37％ 82.76％ 88.46％ 88.47％ GoogleNet 89.40％ 92.73％ 78.06％ 83.57％ 86.89％ GoogleNet+proposed GSN 92.62％ 95.02％ 81.92％ 87.78％ 87.92％

[0045] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A method for improving deep network features based on global similarity, characterized in that: The following steps are involved: S1: Divide the input image into grid-shaped image blocks, and obtain the feature tensor of the image block based on the feature tensor of the input image; simplify the feature channels of the feature tensor of the input image and the feature tensor of the image block; transform the dimension of the feature tensor of the input image and the feature tensor of the image block to obtain the feature matrix of the input image and the image block; S2: Based on the feature matrix of the input image and the image block, the similarity is calculated between each pixel feature and each image block feature; S3: normalize the similarity to obtain the similarity weight; S4: Calculate weighted features for each pixel, where the weight is the similarity weight calculated in the previous step; the weighted feature of a pixel is equal to the weighted sum of the features of all similar image blocks, and the calculation formula is as follows: The process of converting the initial number of feature channels uses a 1x1 convolution kernel, and the number of feature channels C' becomes the initial number of feature channels C; Perform matrix transformation on the weighted features to obtain a weighted feature tensor; convert the simplified number of feature channels in the weighted feature tensor into the initial number of feature channels; S5: The improved feature tensor is equal to the sum of the transformed weighted feature tensor and the initial input feature tensor.

2. According to the method for improving deep network features based on global similarity in claim 1, it is characterized in that: In the step S1, the feature tensor of the input image is F ∈ R CxHxW , where C represents the number of feature channels, H represents the height of the input image, W represents the width of the input image, and R CxHxW represents a real-valued feature tensor with three dimensions; the feature tensor of the image patch is P ∈ R CxGxG , where G represents the number of image patches in terms of height and width, G ≤ H and G ≤ W; the reduction of the number of feature channels uses a 1x1 convolutional kernel, and after reduction, the number of feature channels changes from C to C', where C' < C; in the feature matrices of the image and the image patch, the feature of an element in the input image is represented by F i , where 1 ≤ i ≤ HW, and the feature of an element in the image patch is represented by P j , where 1 ≤ j ≤ G 2 .

3. The method for improving deep network features based on global similarity according to claim 1, characterized in that: In step S2, the similarity is calculated by the following formula: Sim(F i ,P j )=F i P j T This gives us the similarity matrix 4. The method for improving deep network features based on global similarity according to claim 1, characterized in that: In step S3, the softmax function is used for normalization, and the obtained similarity weight matrix M is expressed as: M = softmax(Sim).

5. The method for improving deep network features based on global similarity according to claim 1, characterized in that: The step S5 is calculated by the following formula: F refined =F weighted +F Among them, F refined is the improved feature tensor, F weighted is the transformed weighted feature tensor, and F is the initial input feature tensor.

Images