A deep representation algorithm based on adaptive fusion of first-order and second-order lossless statistics

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a deep characterization algorithm based on first-order and second-order lossless statistical adaptive fusion, including: performing coarse-grained and fine-grained lossless operations on tensor features extracted using a backbone network in a deep convolutional network. Then, for the tensor features that undergo two-stage information lossless operations, first-order and second-order statistical features are calculated, first-order and second-order self-compensation transformation matrices are constructed, and compensation operations are performed on the first-order and second-order statistical features. Finally, for the first-order statistical features and second-order statistical features that have completed the self-compensation operation, an adaptation matrix is ​​constructed to complete the adaptive fusion of the first-order and second-order lossless statistical features. The present invention can reduce the loss of a large amount of limited information contained in the tensor features through information lossless operations on tensor features, compensation calculations of first-order and second-order statistical features, and adaptive fusion, thereby improving the deep characterization performance.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a deep characterization algorithm based on first-order and second-order lossless statistical adaptive fusion. Background Art

[0002] With the advent of the big data era, deep convolutional neural networks have made unprecedented progress in visual classification tasks. In terms of model structure, the backbone network of a deep convolutional network is usually stacked with several convolutional layers, whose main function is to gradually transform the input three-channel image into multi-channel tensor features. Because traditional classifiers cannot use tensor features to complete classification decisions. In this case, some kind of pooling operation is needed to convert tensor features into vector features of fixed length. The pooling layer in the deep learning network plays a vital role in deep representation learning and is one of the research hotspots in the field of deep learning research.

[0003] In the field of deep learning, the most popular pooling method is the standard global average pooling operation. This pooling method can only capture the first-order statistical characteristics of tensor features. Variations of the global pooling operation include global maximum pooling, weighted sum pooling, and general global pooling. Compared with the global average pooling operation, global variance pooling can obtain more information about changes in tensor features. Due to the high-dimensional small sample problem inside the tensor features, how to robustly calculate the global variance is a very difficult task. To this end, SVD-Truc (Y. Song, N. Sebe, and W. Wang, Why approximate matrix square root outperforms accurate svd in global covariance pooling? Proceedings of the IEEE International Conference on Computer Vision, 2021, pp. 1115–1123.) applies truncation to the covariance matrix to calculate the smoothed singular value decomposition for gradient backpropagation. In addition, the global variance pooling method also has the problem of high-dimensional feature representation. To this end, Compact B-CNN (Y. Gao, O. Beijbom, N. Zhang, and T. Darrell, "Compact bilinear pooling," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 317–326.) and Low-rank B-CNN (S. Kong and C. Fowlkes, Low-rank bilinear pooling for fine-grained classification, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 365–374.) use main city analysis and low-rank representation to calculate compact global variance pooling features respectively. In fact, tensor features obey a certain distribution, and the current pooling method only calculates a part of the statistical characteristics of the distribution. It can be seen that the current pooling method will lose a lot of limited information contained in the tensor features, thereby reducing the performance of deep representation. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings existing in the prior art, and to propose a deep characterization algorithm based on first-order and second-order lossless statistical adaptive fusion, which can reduce the loss of a large amount of limited information contained in tensor features and improve the deep characterization performance.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A deep characterization algorithm based on first-order and second-order lossless statistical adaptive fusion includes the following steps:

[0007] Step 1: Two-stage information lossless operation of tensor features: For the tensor features extracted using the backbone network in the deep convolutional network, first perform coarse-grained lossless operation, and then perform fine-grained lossless operation;

[0008] Step 2, self-compensation calculation of first-order statistical features: for the tensor features that perform two-stage information lossless operations, calculate the first-order statistical features, construct a first-order self-compensation transformation matrix, and perform compensation operations on the first-order statistical features;

[0009] Step 3, self-compensation calculation of second-order statistical features: for the tensor features that perform two-stage information lossless operations, calculate the second-order statistical features, construct a second-order self-compensation transformation matrix, and perform compensation operations on the second-order statistical features;

[0010] Step 4: Adaptive fusion of first-order and second-order lossless statistical features: For the first-order statistical features and second-order statistical features that complete the self-compensation operation, an adaptation matrix is ​​constructed to complete the adaptive fusion of the first-order and second-order lossless statistical features.

[0011] Preferably, the specific method of step 1 is as follows:

[0012] S11: For any image I, input it into the backbone network of the deep convolutional neural network to obtain the tensor feature T∈R H×W×C , transform T into a matrix T∈R HW×C , each row vector in the matrix is ​​regarded as an observation value of the tensor variable t, that is

[0013] S12: Perform a coarse-grained lossless operation on each row vector in the matrix T. The calculation expression is:

[0014]

[0015] In formula (1), γ>0 is an adjustable parameter;

[0016] S13: For each row vector in the matrix T after the coarse-grained lossless operation, calculate its average vector feature, denoted as m, and the Euclidean distance calculation formula between the jth row vector and m is:

[0017]

[0018] Then a fine-grained lossless operation is performed on each row vector in the matrix T, and the calculation expression is:

[0019]

[0020] In formula (3), γ>0 is an adjustable parameter, and L is an adjustable threshold parameter.

[0021] Preferably, the specific method of step 2 is as follows:

[0022] S21: For each row vector in the matrix T after the two-stage information lossless operation, the first-order statistical feature is calculated. The calculation formula is:

[0023]

[0024] S22: Construct a first-order self-compensation conversion matrix P = [p 1 , p 2 , ...p K ]∈R C×K , then the calculation formula for the compensation operation of the first-order statistical characteristics is:

[0025]

[0026] The first-order lossless feature is obtained by formula (5):

[0027] Preferably, the specific method of step 3 is as follows:

[0028] S31: For each row vector in the matrix T after the two-stage information lossless operation, the second-order statistical feature is calculated, and the calculation formula is:

[0029]

[0030] S32: Constructing a second-order self-compensating transformation matrix Then the calculation formula for the compensation operation of the second-order statistical characteristics is:

[0031]

[0032] The second-order lossless feature is obtained by formula (7):

[0033] Preferably, the specific method of step 4 is as follows:

[0034] S41: For the first-order statistical characteristics that complete the self-compensation operation, K response matrices are constructed, and the kth response matrix is ​​represented by R k =[r1 , r 2 ,...r K ]∈R K×K , the matching matrix S is calculated based on the second-order lossless feature, and the calculation formula of its kth component is:

[0035]

[0036] Using the matching matrix S, the first-order lossless features are matched:

[0037]

[0038] S42: For the second-order statistical characteristics of the self-compensation operation, K response matrices are constructed, and the kth response matrix is ​​expressed as The matching matrix is ​​obtained based on the second-order lossless feature calculation The calculation formula of its kth component is:

[0039]

[0040] Using the matching matrix Perform matching operation on second-order lossless features:

[0041]

[0042] S43: Complete the matching operation and Put it together, that is:

[0043]

[0044] In formula ((12), Cat(·) is the concatenation operator, and the adaptive matrix W = [w 1 , w 2 , ...w K ]∈R 2K×K , perform adaptive fusion operation on the concatenated features:

[0045]

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention can reduce the loss of a large amount of limited information contained in tensor features and improve the depth characterization performance through information lossless calculation of tensor features, compensation calculation of first-order and second-order statistical features, and adaptive fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the present invention;

[0049] Figure 2 A schematic diagram of visualization of image features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the advantages and features of the present invention, thereby making a clearer definition of the protection scope of the present invention. The embodiments described in the present invention are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0051] A deep characterization algorithm based on first-order and second-order lossless statistical adaptive fusion includes the following steps:

[0052] Step 1: Two-stage information lossless operation of tensor features: For the tensor features extracted using the backbone network in the deep convolutional network, first perform coarse-grained lossless operation, and then perform fine-grained lossless operation;

[0053] Step 2, self-compensation calculation of first-order statistical features: for the tensor features that perform two-stage information lossless operations, calculate the first-order statistical features, construct a first-order self-compensation transformation matrix, and perform compensation operations on the first-order statistical features;

[0054] Step 3, self-compensation calculation of second-order statistical features: for the tensor features that perform two-stage information lossless operations, calculate the second-order statistical features, construct a second-order self-compensation transformation matrix, and perform compensation operations on the second-order statistical features;

[0055] Step 4: Adaptive fusion of first-order and second-order lossless statistical features: For the first-order statistical features and second-order statistical features that complete the self-compensation operation, an adaptation matrix is ​​constructed to complete the adaptive fusion of the first-order and second-order lossless statistical features.

[0056] Specifically, the specific method of step 1 is as follows:

[0057] S11: For any image I, input it into the backbone network of the deep convolutional neural network to obtain the tensor feature T∈R H×W×C , transform T into a matrix T∈R HW×C , each row vector in the matrix is ​​regarded as an observation value of the tensor variable t, that is

[0058] S12: Perform a coarse-grained lossless operation on each row vector in the matrix T. The calculation expression is:

[0059]

[0060] In formula (1), γ>0 is an adjustable parameter;

[0061] S13: For each row vector in the matrix T after the coarse-grained lossless operation, calculate its average vector feature, denoted as m, and the Euclidean distance calculation formula between the jth row vector and m is:

[0062]

[0063] Then a fine-grained lossless operation is performed on each row vector in the matrix T, and the calculation expression is:

[0064]

[0065] In formula (3), γ>0 is an adjustable parameter, and L is an adjustable threshold parameter.

[0066] Specifically, the specific method of step 2 is as follows:

[0067] S21: For each row vector in the matrix T after the two-stage information lossless operation, the first-order statistical feature is calculated. The calculation formula is:

[0068]

[0069] S22: Construct a first-order self-compensation conversion matrix P = [p 1 , p 2 , ...p K ]∈R C×K , then the calculation formula for the compensation operation of the first-order statistical characteristics is:

[0070]

[0071] The first-order lossless feature is obtained by formula (5):

[0072] Specifically, the specific method of step 3 is as follows:

[0073] S31: For each row vector in the matrix T after the two-stage information lossless operation, the second-order statistical feature is calculated, and the calculation formula is:

[0074]

[0075] S32: Constructing a second-order self-compensating transformation matrix Then the calculation formula for the compensation operation of the second-order statistical characteristics is:

[0076]

[0077] The second-order lossless feature is obtained by formula (7):

[0078] Specifically, the specific method of step 4 is as follows:

[0079] S41: For the first-order statistical characteristics that complete the self-compensation operation, K response matrices are constructed, and the kth response matrix is ​​represented by R k =[r 1 , r 2 ,...r K ]∈R K×K , the matching matrix S is calculated based on the second-order lossless feature, and the calculation formula of its kth component is:

[0080]

[0081] Using the matching matrix S, the first-order lossless features are matched:

[0082]

[0083] S42: For the second-order statistical characteristics of the self-compensation operation, K response matrices are constructed, and the kth response matrix is ​​expressed as The matching matrix is ​​obtained based on the second-order lossless feature calculation The calculation formula of its kth component is:

[0084]

[0085] Using the matching matrix Perform matching operation on second-order lossless features:

[0086]

[0087] S43: Complete the matching operation and Put it together, that is:

[0088]

[0089] In formula ((12), Cat(·) is the concatenation operator, and the adaptive matrix W = [w 1 , w 2 , ...w K ]∈R 2K×K , perform adaptive fusion operation on the concatenated features:

[0090]

[0091] Example:

[0092] This embodiment takes the image classification task as an example to illustrate the implementation method of the present invention. Figure 1 The present invention is based on the first-order and second-order lossless statistics adaptive fusion of the deep characterization algorithm, such as Figure 1 As shown, the specific steps are as follows:

[0093] Step 1: For a small sample image classification dataset, sample any image I from its base class dataset and input it into the backbone network in the deep convolutional neural network to obtain the tensor feature T∈R H×W×C ,Will

[0094] Transformed into matrix T∈R HW×C , each row vector in the matrix is ​​regarded as an observation value of the tensor variable t, that is Perform a two-stage lossless operation on each row vector in the matrix T to obtain

[0095] Step 2: For each row vector in the matrix T after the two-stage information lossless operation, calculate the first-order statistical characteristics and construct the first-order self-compensation transformation matrix P. Then the compensation operation of the first-order statistical characteristics is obtained:

[0096] Step 3: For each row vector in the matrix T after the two-stage information lossless operation, calculate the second-order statistical characteristics and construct the er-order self-compensation transformation matrix O. Then the compensation operation of the second-order statistical characteristics is obtained

[0097] Step 4: For the first-order statistical features that complete the self-compensation operation, construct K response matrices, calculate the matching matrix S based on the second-order lossless features, and perform matching operations on the first-order lossless features to obtain For the second-order statistical features that complete the self-compensation operation, K response matrices are constructed, and the matching matrix is ​​calculated based on the second-order lossless features. Perform matching operation on the second-order lossless features to obtain Will and Splice them together to construct the adaptive matrix W, and perform adaptive fusion operation on the spliced ​​features to obtain based on A cross entropy loss function is established to optimize the parameters of the backbone network, self-compensation transformation matrix, response matrix, and adaptive matrix in the network. The optimized network is tested for small sample image classification.

[0098] like Figure 2 As shown, the image feature performance is excellent using the deep representation algorithm based on first-order and second-order lossless statistical adaptive fusion, and various image samples can be clearly separated in the feature space.

[0099] In summary, the present invention can reduce the loss of a large amount of limited information contained in tensor features and improve the deep characterization performance through information lossless calculation of tensor features, compensation calculation of first-order and second-order statistical features, and adaptive fusion.

[0100] The description and practice disclosed in the present invention are easy to think and understand for ordinary technicians in the technical field, and several improvements and modifications can be made without departing from the principles of the present invention. Therefore, modifications or improvements made without departing from the spirit of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A deep characterization method based on adaptive fusion of first-order and second-order lossless statistics, characterized in that: The steps include: Step 1: Two-stage information lossless operation of tensor features: For the image tensor features extracted by the backbone network in the deep convolutional network, first perform coarse-grained lossless operation, and then perform fine-grained lossless operation; Step 2, self-compensation calculation of first-order statistical features: for the tensor features that perform two-stage information lossless operations, calculate the first-order statistical features, construct a first-order self-compensation transformation matrix, and perform compensation operations on the first-order statistical features; Step 3, self-compensation calculation of second-order statistical features: for the tensor features that perform two-stage information lossless operations, calculate the second-order statistical features, construct a second-order self-compensation transformation matrix, and perform compensation operations on the second-order statistical features; Step 4, adaptive fusion of first-order and second-order lossless statistical features: for the first-order statistical features and second-order statistical features that complete the self-compensation operation, construct an adaptation matrix to complete the adaptive fusion of the first-order and second-order lossless statistical features; The specific method of step 2 is as follows: S21: For each row vector in the matrix T after the two-stage information lossless operation, the first-order statistical feature is calculated. The calculation formula is: S22: Construct a first-order self-compensation transformation matrix P = [p1, p2, ... p K ]∈R C×K , then the calculation formula for the compensation operation of the first-order statistical characteristics is: The first-order lossless feature is obtained by formula (5): The specific method of step 3 is as follows: S31: For each row vector in the matrix T after the two-stage information lossless operation, the second-order statistical feature is calculated, and the calculation formula is: S32: Constructing a second-order self-compensating transformation matrix Then the calculation formula for the compensation operation of the second-order statistical characteristics is: The second-order lossless feature is obtained by formula (7): The specific method of step 4 is as follows: S41: For the first-order statistical characteristics that complete the self-compensation operation, K response matrices are constructed, and the kth response matrix is ​​represented by R k =[r1, r2, ...r K ]∈R K×K , the matching matrix S is calculated based on the second-order lossless feature, and the calculation formula of its kth component is: Using the matching matrix S, the first-order lossless features are matched: S42: For the second-order statistical characteristics of the self-compensation operation, K response matrices are constructed, and the kth response matrix is ​​expressed as The matching matrix is ​​obtained based on the second-order lossless feature calculation The calculation formula of its kth component is: Using the matching matrix Perform matching operation on second-order lossless features: S43: Complete the matching operation and Put it together, that is: In formula ((12), cat(·) is the concatenation operator, and the adaptive matrix W = [w1, w2, ... w K ]∈R 2K×K , perform adaptive fusion operation on the concatenated features:

2. The method for deep characterization based on adaptive fusion of first-order and second-order lossless statistics according to claim 1, characterized in that: The specific method of step 1 is as follows: S11: For any image I, input it into the backbone network of the deep convolutional neural network to obtain the tensor feature T∈R H×W×C , transform T into a matrix T∈R HW×C , each row vector in the matrix is ​​regarded as an observation value of the tensor variable t, that is t j ∈R 1×C ; S12: Perform a coarse-grained lossless operation on each row vector in the matrix T. The calculation expression is: In formula (1), γ>0 is an adjustable parameter; S13: For each row vector in the matrix T after the coarse-grained lossless operation, calculate its average vector feature, denoted as m, and the Euclidean distance calculation formula between the jth row vector and m is: Then a fine-grained lossless operation is performed on each row vector in the matrix T, and the calculation expression is: In formula (3), γ>0 is an adjustable parameter, and L is an adjustable threshold parameter.

Images