Image depth representation method based on adaptive feature view fusion

Abstract

The invention discloses an image depth representation method based on adaptive feature view fusion. The method comprises the following steps: firstly, extracting a feature view by using a multi-stage encoder; then, a self-adaptive feature view weight framework is provided by utilizing a Lagrange optimization method and a KKT condition, nonlinear inverse weight is introduced, key views are strengthened, and meanwhile views which are not high in importance but still related are reserved; finally, an adaptive feature view fusion algorithm based on the structured depth map is provided, and after the initially extracted feature view is used for the fusion algorithm, the generated unified sparse depth map is more significant for distinguishing different categories. The method can solve the problems that a convolutional neural network cannot fully utilize low-level features, and a traditional method only pays attention to reduction of the weight of unimportant feature views and seldom adaptively establishes an internal structure and a potential relation through different feature views during feature fusion. Meanwhile, the result of the method can be used for data analysis tasks such as clustering analysis and depth map construction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image depth representation method based on adaptive feature view fusion. Background Art

[0002] Convolutional neural network (CNN) is one of the representative algorithms in the field of artificial intelligence. CNN has the ability to learn representations and can perform translation-invariant classification of input information according to its hierarchical structure. Traditional CNN focuses on deepening the network to improve performance. However, only obtaining deep representations may lead to the loss of low-level features, such as global and edge information, which are critical for some specific tasks. Effective feature extraction, fusion, and representation of key information in data are of great significance to improving model performance. Making full use of low-level features in deep networks is a common method. Many studies focus on better extracting effective information from low-level representations. The literature "D.Haase, M.Amthor, Rethinking depthwise separable convolutions: How intra-kernel correlations lead to improved mobilenets, in: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 14600–14609." and "ZYKhan, Z.Niu, Cnn with depthwiseseparable convolutions and combined kernels for rating prediction, ExpertSystems with Applications 170 (2021) 114528." uses a deep separable convolution network (DSC). DSC uses a convolution kernel for each channel of the input feature map, and then concatenates the outputs of all convolution kernels to obtain its final output. Its computational efficiency is much better than that of ordinary convolution, but it may cause gradient disappearance and low accuracy, which will have an adverse effect on training stability and performance. In addition, the paper "Q. Zhao, T. Sheng, Y. Wang, Z. Tang, Y. Chen, L. Cai, H. Ling, M2det: A single-shot object detector based on multi-level feature pyramid network, in: Proceedings of the AAAI conference on artificial intelligence, volume 33, 2019, pp. 9259–9266." uses a feature pyramid network to extract features from various spatial resolutions by using different convolution and pooling scales, but it significantly increases the computational and storage costs.

[0003] In order to better achieve feature fusion, the paper "P.Chao, X.Zhang, Y.Gang, G.Luo, S.Jian, Largekernel matters—improve semantic segmentation by global convolutionalnetwork, in: 2017IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017." proposed a global convolutional network (GCN), which fuses low-level features and high-level features through skip connections to obtain features that are more semantic than a single feature map. However, GCN uses a lot of residual connection structures for boundary refinement, which greatly increases the amount of calculation. "J.Wen, Z.Zhang, Z.Zhang, Z.Wu, L.Fei, Y.Xu, B.Zhang, Dimc-net: Deepincomplete multi-view clustering network, in: Proceedings of the 28th ACMInternational Conference on Multimedia, 2020, pp.3753–3761." uses multiple encoders to extract depth information from multiple views, and designs a weighted fusion layer to reduce the negative impact of missing views. Existing feature view fusion strategies often eliminate relatively unimportant feature views by reducing their weights, and rarely pay attention to adaptively building internal structures and potential connections through different feature views. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides an image depth representation method based on adaptive feature view fusion. First, a multi-level encoder is used to extract feature views. Then, an adaptive feature view weight framework is proposed using the Lagrangian optimization method and the KKT condition, which introduces nonlinear inverse weights, strengthens key views, and retains views that are not important but still relevant. Finally, an adaptive feature view fusion algorithm based on a structured depth map is proposed. After the initially extracted feature views are used in the fusion algorithm, the generated unified sparse depth map is more significant for distinguishing different categories. The present invention can solve the problems of insufficient utilization of low-level features by convolutional neural networks, traditional methods only focusing on reducing the weights of unimportant feature views during feature fusion, and rarely adaptively establishing internal structures and potential connections through different feature views. At the same time, the results of the present invention can be used for data analysis tasks such as clustering analysis and depth map construction.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] Step 1: Multi-level feature view extraction;

[0007] All images in the input image dataset are resized, standardized and normalized so that their pixel values ​​are proportionally normalized to the interval [0,1]; the input image is then preprocessed through a convolutional layer and a pooling layer; the dataset is input to the encoder and trained using a contrastive learning strategy. The encoder consists of v continuous residual connections;

[0008] Step 2: Adaptive feature view weight framework;

[0009] set up Where v = 1, 2, ..., l is the feature view of the vth stage, l is the total number of feature views, is a metric function to evaluate the loss of each feature view. The view weights at different exponential levels m, m = 0, 1, 2, ..., ∞ are integrated and simplified to obtain:

[0010]

[0011] Among them, α v represents the weight corresponding to the vth stage, Represents the loss corresponding to the vth stage;

[0012] Apply the Lagrange multiplier method to optimize formula (2) and adaptively obtain the optimal α v :

[0013]

[0014] Among them, p is a hyperparameter, represents the loss corresponding to the jth stage, (·) + =max(0,·), and p satisfies:

[0015]

[0016] in, represents the loss corresponding to the pth stage, Represents the loss corresponding to the p+1th stage;

[0017] So we get the weight α of each view v ;

[0018] Step 3: Adaptive feature view fusion based on structured depth map;

[0019] set up The dimension is d v , the feature view of the vth stage with n data; the fusion model is expressed as:

[0020]

[0021] Among them, G ij represents the connectivity between the i-th sample and the j-th sample, λ represents the hyperparameter of the regularization term, and G represents the depth map representation;

[0022] Suppose that the data is to be divided into c clusters. The optimal depth graph representation graph G contains c connected components. Using the Laplacian matrix The connection between the connected components of the depth map G, where the degree matrix D is a diagonal matrix, the i-th entry

[0023] Adding implicit constraints to the model Get a structured graph, where σ i (L) is the i-th smallest eigenvalue of the semi-positive definite matrix L, and then we get in is the spectral matrix, that is, the eigenvector matrix of the Laplace matrix L;

[0024] Therefore, the duality of the structured model in equation (5) can be rewritten as:

[0025]

[0026] When β→∞, the third term tr(F T LF) tends to That is, the low-rank constraint rank(L) = nc is satisfied, so equation (6) is regarded as an approximate form of equation (5);

[0027] Step 4: Optimization process;

[0028] The structured model proposed in formula (6) is used for optimization;

[0029] Fixed alpha v Solve F with G, and equation (6) is written as The solution F is the c smallest eigenvectors of the Laplacian matrix L;

[0030] Fix G and F to solve for α v :Formula (6) is written as in This is equivalent to solving it using the adaptive feature view weight framework in step 2;

[0031] Fixed alpha v and F to solve G: Formula (6) is written as Among them G i is the i-th column of G, F i is the i-th row of the spectral matrix F, λ i yes The trade-off parameter of

[0032] When e i ∈R n×1 is a column vector whose jth element is e ij , the i-th item is

[0033] For any i, assuming e i1 ≤e i2 ≤···≤e in After optimizing the problem using the Lagrange multiplier method, the closed-form solution G is derived. i , the jth element is represented as:

[0034]

[0035] The closed-form solution based on equation (7) can solve the proposed structured adaptive multi-feature view fusion problem and obtain the joint unified depth map G.

[0036] Preferably, the encoder comprises a start block 1, a residual block 2, a residual block 3, a residual block 4, a residual block 5 and a fully connected layer 6;

[0037] The number of input channels of the starting block 1 is 1, and the number of output channels is 64. It consists of a convolution layer with a convolution kernel of 3×3 and a stride of 1, a batch normalization BatchNorm2d layer, and a nonlinear activation ReLU layer; the number of input channels of the residual block 2 is 64, and the number of output channels is 128. It includes two branches. Branch 1 consists of a convolution layer with a convolution kernel of 3×3 and a stride of 1, a batch normalization BatchNorm2d layer, a nonlinear activation ReLU layer, a convolution layer with a convolution kernel of 3×3 and a stride of 1, and a batch normalization BatchNorm2d layer connected sequentially. Branch 2 consists of a convolution layer with a convolution kernel of 1×1 and a stride of 1, and a batch normalization BatchNorm2d layer. The output of branch 1 is added to the output of branch 2, and then connected to a nonlinear activation ReLU layer. The input channel number of residual block 3 is 64, and the output channel number is 128, including a branch 1, the output of branch 1 is added to the residual, and then connected to a nonlinear activation ReLU layer; the input channel number of residual block 4 is 128, and the output channel number is 256; the input channel number of residual block 5 is 256, and the output channel number is 512; the fully connected layer 6 is an adaptive average pooling layer, the input is the features of each residual block output, the input channel number is 64, 128, 256, 512, and the output channel number is 64, 128, 256, 512, and a flattening layer, and the output sizes are 2×64, 2×128, 2×256, 2×512 respectively; the input of the encoder is an image, and the output is the shallow and deep features corresponding to the image.

[0038] Preferably, the process of training by contrastive learning strategy in step 1 is as follows:

[0039] Update the encoder parameters by minimizing the contrast loss through the gradient descent method, contrast loss loss v The calculation formula is as follows:

[0040]

[0041] in represents the distance between two representations, yes The only negative sample and n is the number of samples and τ is a hyperparameter.

[0042] The beneficial effects of the present invention are as follows:

[0043] (1) We design a multi-stage encoder guided by contrastive learning to continuously obtain multiple feature views containing complementary semantic information;

[0044] (2) A novel feature view weighting strategy with negative inversion technology is proposed, which highlights the key views while retaining the remaining feature views, and can effectively utilize the low-level semantic information in the convolutional neural network;

[0045] (3) Through embedding graph optimization and feature view fusion adaptive learning, a unified structured depth map is constructed to represent the depth map representation. Equipped with the structured depth map and adaptive feature view weight framework, an adaptive feature view fusion algorithm based on structured depth map is developed for clustering tasks under low-rank constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of an image depth characterization method based on adaptive feature view fusion of the present invention;

[0047] Figure 2 The distance heat map and the final depth map obtained by setting different numbers of feature views in the COIL20 image dataset according to the embodiment of the present invention are: (a) the distance heat map using 1 feature view, (b) the distance heat map using 2 feature views, (c) the distance heat map using 4 feature views, and (d) the depth map matrix obtained after adaptive fusion of multiple feature views;

[0048] Figure 3These are the training results of the embodiments of the present invention on the CIFAR-10 and CIFAR-100 image datasets, (a) ACC heat map result of the CIFAR-10 dataset, (b) NMI result of the CIFAR-10 dataset, (c) ACC result of the CIFAR-100 dataset, and (d) NMI result of the CIFAR-10 dataset. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0050] In order to overcome the problems that the existing methods do not fully utilize low-level features when using convolutional neural networks to implement clustering tasks, the traditional methods only focus on reducing the weights of unimportant feature views during feature fusion, and rarely adaptively establish internal structures and potential connections through different feature views, the present invention provides an image depth representation method for adaptive feature view fusion. First, a multi-level encoder is used to extract feature views. Specifically, the input image is first preprocessed through a convolution layer and a maximum pooling layer. The subsequent core feature view extraction module consists of v continuous residual connections. Through the multi-level encoder, v feature view data H can be obtained. (v) , in which the loss function uses the contrastive loss function in contrastive learning. Then, using the Lagrangian optimization method and the KKT condition, an adaptive feature view weight framework is proposed, which introduces nonlinear inverse weights, strengthens the key views, and retains views that are not very important but still relevant. In order to effectively fuse the feature views, an adaptive feature view fusion algorithm based on structured depth maps is finally proposed. After the initially extracted feature views are used in the fusion algorithm, the generated unified sparse depth map is more effective in distinguishing different categories, so it is more effective to use it as a depth map representation for clustering, and significantly improves the accuracy of convolutional neural networks in clustering tasks. The results of the present invention can be used for data analysis tasks such as clustering analysis and depth map construction.

[0051] An image depth representation method for adaptive feature view fusion, the steps are as follows:

[0052] Step 1, multi-level feature view extraction;

[0053] All images in the input image dataset are resized, standardized and normalized so that their pixel values ​​are proportionally normalized to the interval [0,1]. The input image is then preprocessed through a convolutional layer and a pooling layer. The dataset is then input into the core feature view extraction module and trained using a contrastive learning strategy. The encoder consists of v continuous residual connections.

[0054] The encoder includes a start block 1, a residual block 2, a residual block 3, a residual block 4, a residual block 5 and a fully connected layer 6. The number of input channels of the start block 1 is 1, the number of output channels is 64, and it is composed of a convolution layer with a convolution kernel of 3×3 and a step size of 1, a batch normalization BatchNorm2d layer and a nonlinear activation ReLU layer; the number of input channels of the residual block 2 is 64, the number of output channels is 128, and it includes two branches, branch 1 is composed of a convolution layer with a convolution kernel of 3×3 and a step size of 1, a batch normalization BatchNorm2d layer, a nonlinear activation ReLU layer, a convolution layer with a convolution kernel of 3×3 and a step size of 1, and a batch normalization BatchNorm2d layer connected in sequence, and branch 2 is composed of a convolution layer with a convolution kernel of 1×1 and a step size of 1 and a batch normalization BatchNorm2d layer, the output of branch 1 is added to the output of branch 2, and then connected to a nonlinear activation ReLU layer. The input channel number of residual block 3 is 64, and the output channel number is 128, including a branch 1, the output of branch 1 is added to the residual, and then connected to a nonlinear activation ReLU layer; the input channel number of residual block 4 is 128, and the output channel number is 256; the input channel number of residual block 5 is 256, and the output channel number is 512; the fully connected layer 6 is an adaptive average pooling layer, the input is the features of each residual block output, the input channel number is 64, 128, 256, 512, and the output channel number is 64, 128, 256, 512, and a flattening layer, and the output size is 2×64, 2×128, 2×256, 2×512 respectively; the input of the encoder is an image, and the output is the shallow and deep features corresponding to the image;

[0055] The training described above is to update the encoder parameters by minimizing the contrast loss through the gradient descent method. v The calculation formula is as follows:

[0056]

[0057] in yes The only negative sample and

[0058] Step 2, adaptive feature view weight framework;

[0059] set up where v=1,2,…,l is the feature view of the vth stage, is a metric function for evaluating the loss of each feature view. By integrating the view weights at different exponential levels m (m = 0, 1, 2, ..., ∞) and simplifying, we can get:

[0060]

[0061] Applying the Lagrange multiplier method to optimize the formula, the optimal α can be adaptively obtained. v , and finally can be transformed into:

[0062]

[0063] in(·) + =max(0,·), and p needs to satisfy:

[0064]

[0065] By using the adaptive feature view weight framework, the weight α of each view can be obtained v .

[0066] Step 3: Adaptive feature view fusion based on structured depth map;

[0067] Based on the adaptive feature view weight learning framework, adaptive multi-feature view weight fusion is developed. The dimension is d v , the feature view of the vth stage with n data. The fusion model can be expressed as:

[0068]

[0069] However, G in equation (5) is difficult to calculate. Assuming that the data is to be divided into c clusters, the optimal depth graph representation graph G contains exactly c connected components, which can be calculated using the Laplacian matrix The connection between the connected components of the depth map G, where the degree matrix D is a diagonal matrix, the i-th entry

[0070] Adding implicit constraints to the model We can get a structured graph, where σ i (L) is the i-th smallest eigenvalue of the semi-positive definite matrix L, and then we can get in is the spectral matrix, that is, the eigenvector matrix of the Laplace matrix L. Therefore, the duality of the structured model in equation (5) can be rewritten as:

[0071]

[0072] When β→∞, the third term tr(F T LF) tends to That is to say, the low-rank constraint rank(L) = nc is satisfied.

[0073] Therefore, formula (6) can be regarded as an approximate form of formula (5).

[0074] Step 4, optimization process;

[0075] The optimization is performed using the structured model proposed in formula (6). v Solving F with G, equation (6) can be written as The solution F is the c smallest eigenvectors of the Laplace matrix L; fix G and F to solve α v :Equation (6) can be written as in This is equivalent to solving the problem using the adaptive feature view weight framework in step 2; fix α v and F to solve G: Formula (6) can be written as Among them G i is the i-th column of G, F i is the i-th row of the spectral matrix F, λ i yes The trade-off parameter of i ∈R n×1 is a column vector whose jth element is e ij , the i-th item is

[0076] For any i, assuming e i1 ≤e i2 ≤···≤e in , in order to find a sparse depth map G, only k adjacent data are considered. After optimizing the problem using the Lagrange multiplier method, its closed-form solution G can be derived i , its j-th element is represented by

[0077]

[0078] The closed-form solution based on equation (7) can solve the proposed structured adaptive multi-feature view fusion problem and obtain the joint unified depth map G.

[0079] According to the above four steps, the algorithm can be summarized as follows: First, initialize the feature view weight α v and the spectrum matrix F respectively satisfy ∑ v α v =1 and F T F = I, and initialize the compromise parameter e according to the number of data n ij , then update the depth map G and construct the Laplacian matrix Then update the feature view loss function according to the number of views v The adaptive feature view weight framework is used to update the weight of each feature view, and finally the spectrum matrix F is updated and the above process is repeated. -3 When , the algorithm converges and the training ends. The output results have feature view weights αv , spectral matrix F and joint depth map G.

[0080] In summary, the image depth representation obtained by the adaptive feature view fusion algorithm based on structured depth graph can be used for data analysis tasks such as K-means clustering and spectral clustering analysis based on graph theory, as well as constructing deep graphs for graph representation.

[0081] Example:

[0082] like Figure 1 As shown, the present invention provides an image depth representation method based on adaptive feature view fusion, and its specific implementation process is as follows:

[0083] Step 1, multi-level feature view extraction;

[0084] Step 1.1, crop all images in the original image dataset to a size of 32×32, and then perform data standardization and normalization to normalize the pixel values ​​to the [0,1] interval. Then preprocess the input image through a convolutional layer with a kernel size of 7×7 and a maximum pooling layer for downsampling;

[0085] Step 1.2, the preprocessed data set is input into the core feature view extraction module, and trained by contrastive learning strategy. The encoder consists of v continuous residual connections. The detailed parameters and design are as follows:

[0086] Starting block 1: The number of input channels is 1, and the number of output channels is 64. A convolution layer with a kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer + a nonlinear activation ReLU layer;

[0087] Residual block 2: The number of input channels is 64, and the number of output channels is 64. It consists of branch 1 (a convolution layer with a convolution kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer + a nonlinear activation ReLU layer, a convolution layer with a convolution kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer), and branch 2 (a convolution layer with a convolution kernel of 1×1 and a stride of 1 for receiving residuals + a batch normalization BatchNorm2d layer). Then, the output of branch 1 and the output of branch 2 are added together, and then passed through a nonlinear activation ReLU layer;

[0088] Residual block 3: The number of input channels is 64, and the number of output channels is 128. It consists of branch 1 (a convolution layer with a convolution kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer + a non-linear activation ReLU layer, a convolution layer with a convolution kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer), and branch 2 (a convolution layer with a convolution kernel of 1×1 and a stride of 1 for receiving residuals + a batch normalization BatchNorm2d layer). Then, the output of branch 1 and the output of branch 2 are added together, and then passed through a non-linear activation ReLU layer;

[0089] Residual block 4: The number of input channels is 128, and the number of output channels is 256. It consists of branch 1 (a convolution layer with a convolution kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer + a nonlinear activation ReLU layer, a convolution layer with a convolution kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer), and branch 2 (a convolution layer with a convolution kernel of 1×1 and a stride of 1 for receiving residuals + a batch normalization BatchNorm2d layer). Then, the output of branch 1 and the output of branch 2 are added together, and then passed through a nonlinear activation ReLU layer;

[0090] Residual block 5: The number of input channels is 256, and the number of output channels is 512. It consists of branch 1 (a convolution layer with a convolution kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer + a non-linear activation ReLU layer, a convolution layer with a convolution kernel of 3×3 and a stride of 1 + a batch normalization BatchNorm2d layer), and branch 2 (a convolution layer with a convolution kernel of 1×1 and a stride of 1 for receiving residuals + a batch normalization BatchNorm2d layer). Then, the output of branch 1 and the output of branch 2 are added together, and then passed through a non-linear activation ReLU layer;

[0091] Fully connected layer 6: It is an adaptive average pooling layer. The input is the features of each residual block output. The number of input channels is 64, 128, 256, and 512 respectively. The number of output channels is 64, 128, 256, 512 and a flattening layer. The output sizes are 2×64, 2×128, 2×256, and 2×512 respectively.

[0092] The encoder input is an image, which is read into the network in the form of a two-dimensional matrix. The output is the shallow and deep features corresponding to the image. The data form is 4 feature vectors with dimensions of 64, 128, 256, and 512 respectively. The encoding process can be expressed as Where X = H (0) represents the original input, H (v) The dimension of the i-th stage is d vFinally, the obtained feature views are used for contrastive learning in unsupervised tasks.

[0093] The training described above is to update the encoder parameters by minimizing the contrast loss through the gradient descent method. v The calculation formula is as follows:

[0094]

[0095] in yes The only negative sample and

[0096] Step 2, adaptive feature view weight framework;

[0097] set up where v=1,2,…,l is the feature view of the vth stage, is a metric function for evaluating the loss of each feature view. By integrating the view weights at different exponential levels m (m = 0, 1, 2, ..., ∞), we can obtain:

[0098]

[0099] in The smaller it is, the more important the v-th view is. Each item in the m-th index layer As an adaptive feature view weight learning with exponential regularization. Therefore, formula (9) can be regarded as the fusion of feature view weight learning under different indicator levels m. Formula (9) can be simplified as:

[0100]

[0101] in use This formula introduces nonlinear inverse weights on different views, strengthening the key views while retaining less important but still relevant views. Applying the Lagrange multiplier method to optimize the formula can adaptively obtain the optimal α v , and finally can be transformed into:

[0102]

[0103] in(·) + =max(0,·), and p needs to satisfy the following constraints:

[0104]

[0105] Therefore, when When , its vth element is α vThe optimal α has only p non-zero entries.

[0106] The algorithm can be summarized as follows: Input is the loss function of each view Initialize p = 0, repeat p = p + 1 and update α using equation (11) v , until equation (12) is satisfied. The output is the weight α of each view v .

[0107] Step 3: Adaptive feature view fusion based on structured depth map;

[0108] Based on the adaptive feature view weight learning framework, adaptive multi-feature view weight fusion is developed. The dimension is d v , the feature view of the vth stage with n data, the fusion model can be expressed as

[0109]

[0110] Where λ>0 is to prevent the depth map The regularization parameters of the trivial solution are obtained and G can be adaptively constructed by fusion of multiple feature views.

[0111] Assuming that the data is to be divided into c clusters, the optimal structure of the deep graph representation graph G contains exactly c connected components. However, it is difficult to solve G using formula (13). Using the Laplacian matrix The connection between the connected components of the depth map G, where the degree matrix D is a diagonal matrix, the i-th entry

[0112] According to the theorem proposed in the literature "K. Fan, "On a theorem of weyl concerning eigenvalues ​​of linear transformations," Proceedings of the National Academy of Sciences, vol. 36, no. 1, pp. 31–35, 1950." and "C. Fan, Spectral Graph Theory, Conference Board of the mathematical sciences by the American Mathematical Society, 2015.", implicit constraints are added to the model We can get a structured graph, where σ i (L) is the i-th smallest eigenvalue of the semi-positive definite matrix L, and then we can get in is the spectral matrix, that is, the eigenvector matrix of the Laplace matrix L. Therefore, the duality of the structured model in equation (13) can be rewritten as:

[0113]

[0114] When β→∞, the third term tr(F T LF) tends to That is, the low-rank constraint rank(L)=nc is satisfied.

[0115] Step 4, optimization process;

[0116] Step 4.1, fix α v and G to solve F: Formula (14) can be written as:

[0117]

[0118] The solution F is the c smallest eigenvectors of the Laplacian matrix L.

[0119] Step 4.2, fix G and F and solve for α v :Equation (14) can be written as:

[0120]

[0121] in This is equivalent to solving it using the adaptive feature view weight framework in step 2.

[0122] Step 4.3, fix α v and F to solve G: Equation (14) can be written as

[0123]

[0124] Among them G i is the i-th column of G, F i is the i-th row of the spectral matrix F, λ i yes The trade-off parameter of i ∈R n×1 is a column vector whose jth element is e ij When , the i-th term of formula (17) is:

[0125]

[0126] For any i, assuming e i1 ≤e i2 ≤···≤e in , in order to find a sparse depth map G, only k adjacent data are considered. After optimizing the problem using the Lagrange multiplier method, its closed-form solution G can be derivedi , its j-th element is represented by

[0127]

[0128] Based on this closed-form solution, the proposed structured adaptive multi-feature view fusion problem can be solved. The algorithm can be summarized as follows: First, initialize the feature view weight α v and the spectrum matrix F respectively satisfy ∑ v α v =1 and F T F = I, and initialize the compromise parameter e according to the number of data n ij , then update the depth map G and construct the Laplacian matrix Then update the feature view loss function according to the number of views v The adaptive feature view weight framework is used to update the weight of each feature view, and finally the spectrum matrix F is updated and the above process is repeated. -3 When , the algorithm converges and the training ends. The output results have feature view weights α v , spectral matrix F and joint depth map G.

[0129] The present invention proposes an adaptive multi-feature view fusion method under low-rank constraint, which adopts a multi-stage feature view encoder to preprocess and extract core feature views with complementary information, and proposes a new feature view weight strategy, which uses the negative inversion technique in the adaptive non-parametric process to highlight the key feature views, while retaining the remaining feature views to achieve effective multi-feature view fusion. In addition to the proposed weight framework, a unified depth map is constructed in the optimization process, and multi-feature view data is adaptively fused. In order to obtain the optimal structured depth map, the low-rank constraint of c-connected components is also applied. The unified sparse depth map generated after the fusion of multiple feature views is more significant for distinguishing different categories, and it is more effective to use it as a deep map representation for clustering, which significantly improves the accuracy of convolutional neural networks in clustering tasks. The deep representation obtained by the present invention can be used for K-means clustering and spectral clustering (SpectralClustering) clustering analysis based on graph theory, as well as constructing a deep graph (Deep Graph) for graph representation (GraphRepresentation) and other data analysis tasks.

[0130] To verify the effectiveness of the method of the present invention, a NVIDIA GeForce GTX 1080Ti GPU graphics card with 24GB RAM was used on the Windows 11 operating system, and the PyTorch 1.10 and Python 3.7 development environments were used to perform simulation experiments. Figure 2 and Figure 3As shown in Table 1, four image datasets containing different scenes are used as benchmark datasets in the experiment, namely: COIL20 object image dataset, cat-dog war dataset, CIFAR-10 dataset, and CIFAR-100 dataset. The specific parameters of each dataset are shown in Table 1.

[0131] Table 1 Simulation experiment benchmark dataset information

[0132]

[0133] K-means (K-Means) clustering method, spectral clustering (SC) method, deep secure incomplete multi-view clustering (DSIMVC) method, collaborative regularized multi-view spectral clustering (CRMVSC) method, consensus graph learning (CGL) method for multi-view clustering, robust multi-view K-means clustering (RMKMC) method, contrastive clustering (CC) method, efficient deep embedding subspace clustering (EDESC) method, multi-view subspace clustering on topological manifold (MSC) method were selected as comparison methods, and feature extraction was performed on four image datasets respectively. Then, the accuracy (ACC) and Normalized Mutual Information (NMI) values ​​obtained by clustering the surface features of the original data or the deep features learned by each method were calculated, and the results are shown in Table 2. It can be seen that the overall performance of the method of the present invention is excellent, and its performance on the four benchmark datasets is better than that of other methods.

[0134] Table 2 Performance comparison of various feature analysis methods for clustering features on four benchmark datasets

[0135]

Claims

1. An image depth representation method based on adaptive feature view fusion, characterized in that: The steps include: Step 1: Multi-level feature view extraction; All images in the input image dataset are resized, standardized, and normalized so that their pixel values ​​are proportionally normalized to the interval [0,1]; The input image is then preprocessed through a convolutional layer and a pooling layer; the dataset is input to the encoder and trained using a contrastive learning strategy. The encoder consists of v continuous residual connections; Step 2: Adaptive feature view weight framework; set up Where v = 1, 2, ..., l is the feature view of the vth stage, l is the total number of feature views, is a metric function to evaluate the loss of each feature view. The view weights at different exponential levels m, m = 0, 1, 2, ..., ∞ are integrated and simplified to obtain: Among them, α v represents the weight corresponding to the vth stage, Represents the loss corresponding to the vth stage; Apply the Lagrange multiplier method to optimize formula (2) and adaptively obtain the optimal α v : Among them, p is a hyperparameter, represents the loss corresponding to the jth stage, (·) + =max(0,·), and p satisfies: in, represents the loss corresponding to the pth stage, Represents the loss corresponding to the p+1th stage; So we get the weight α of each view v ; Step 3: Adaptive feature view fusion based on structured depth map; set up The dimension is d v , the feature view of the vth stage with n data; the fusion model is expressed as: Among them, G ij represents the connectivity between the i-th sample and the j-th sample, λ represents the hyperparameter of the regularization term, and G represents the depth map representation; Suppose that the data is to be divided into c clusters. The optimal depth graph representation graph G contains c connected components. Using the Laplacian matrix The connection between the connected components of the depth map G, where the degree matrix D is a diagonal matrix, the i-th entry Adding implicit constraints to the model Get a structured graph, where σ i (L) is the i-th smallest eigenvalue of the semi-positive definite matrix L, and then we get in is the spectral matrix, that is, the eigenvector matrix of the Laplace matrix L; Therefore, the duality of the structured model in equation (5) can be rewritten as: When β→∞, the third term tr(F T LF) tends to That is, the low-rank constraint rank(L) = nc is satisfied, so equation (6) is regarded as an approximate form of equation (5); Step 4: Optimization process; The structured model proposed in formula (6) is used for optimization; Fixed alpha v Solve F with G, and equation (6) is written as The solution F is the c smallest eigenvectors of the Laplacian matrix L; Fix G and F to solve for α v :Formula (6) is written as in This is equivalent to solving it using the adaptive feature view weight framework in step 2; Fixed alpha v and F to solve G: Formula (6) is written as Among them G i is the i-th column of G, F i is the i-th row of the spectral matrix F, λ i yes The trade-off parameter of When e i ∈R n×1 is a column vector whose jth element is e ij , the i-th item is For any i, assuming e i1 ≤e i2 ≤···≤e in After optimizing the problem using the Lagrange multiplier method, the closed-form solution G is derived. i , the jth element is represented as: The closed-form solution based on equation (7) can solve the proposed structured adaptive multi-feature view fusion problem and obtain the joint unified depth map G.

2. The image depth representation method based on adaptive feature view fusion according to claim 1, characterized in that: The encoder includes a start block 1, a residual block 2, a residual block 3, a residual block 4, a residual block 5 and a fully connected layer 6; The number of input channels of the starting block 1 is 1, and the number of output channels is 64. It consists of a convolution layer with a convolution kernel of 3×3 and a stride of 1, a batch normalization BatchNorm2d layer, and a nonlinear activation ReLU layer; the number of input channels of the residual block 2 is 64, and the number of output channels is 128. It includes two branches. Branch 1 consists of a convolution layer with a convolution kernel of 3×3 and a stride of 1, a batch normalization BatchNorm2d layer, a nonlinear activation ReLU layer, a convolution layer with a convolution kernel of 3×3 and a stride of 1, and a batch normalization BatchNorm2d layer connected sequentially. Branch 2 consists of a convolution layer with a convolution kernel of 1×1 and a stride of 1, and a batch normalization BatchNorm2d layer. The output of branch 1 is added to the output of branch 2, and then connected to a nonlinear activation ReLU layer. The input channel number of residual block 3 is 64, and the output channel number is 128, including a branch 1, the output of branch 1 is added to the residual, and then connected to a nonlinear activation ReLU layer; the input channel number of residual block 4 is 128, and the output channel number is 256; the input channel number of residual block 5 is 256, and the output channel number is 512; the fully connected layer 6 is an adaptive average pooling layer, the input is the features of each residual block output, the input channel number is 64, 128, 256, 512, and the output channel number is 64, 128, 256, 512, and a flattening layer, and the output sizes are 2×64, 2×128, 2×256, 2×512 respectively; the input of the encoder is an image, and the output is the shallow and deep features corresponding to the image.

3. The image depth representation method based on adaptive feature view fusion according to claim 1, characterized in that: The process of training by contrastive learning strategy in step 1 is as follows: Update the encoder parameters by minimizing the contrast loss through the gradient descent method, contrast loss loss v The calculation formula is as follows: in represents the distance between two representations, yes The only negative sample and n is the number of samples and τ is a hyperparameter.

Images