Progressive multi-view depth integrated representation learning method

Abstract

The present application relates to a progressive multi-view depth comprehensive representation learning method, and belongs to the field of machine learning and computer vision, and is used for solving the problem of how to mine the consistency information and complementary information between views in multi-view representation under the condition of ensuring that the accuracy is not reduced. The method of the present application firstly establishes an easier pre-training task to better initialize the consistency and complementary dimensions and can process multiple views. Secondly, in the fine-tuning stage, the pre-established sparse batch normalization layer is used to fully mine the complementary information of multi-view data and pre-align the representation of a specific view to solve the problem of missing views. The learning method of the present application can process any multi-view data and is suitable for different data sizes.

Description

Technical Field

[0001] The present disclosure relates to the fields of machine learning and computer vision, and in particular to a progressive multi-view deep comprehensive representation learning method. Background Art

[0002] Many real-world applications involve multiple perspectives. For example, doctors diagnose a patient's illness based on blood tests, radiographs, and other data. These different perspectives, comprised of multiple different tests, often reveal consistent and complementary information about the same data. Fusion of multi-perspective data can improve performance in many tasks. The rapid development of deep learning research in recent years has given rise to many popular multi-perspective deep learning research topics.

[0003] Multi-view Comprehensive Representation Learning (MCRL) aims to learn comprehensive representations of data from multiple viewpoints. Currently popular MCRL methods based on deep learning usually concatenate or average-pool the representations of each viewpoint during the fusion phase. However, when some viewpoints are missing in practical applications, the performance of multi-view representation learning based on concatenation will inevitably degrade or even fail. Fusion methods based on average pooling usually cannot fully utilize the complementarity of multi-view data. Therefore, in the multi-view representation problem, how to mine the consistency and complementary information between viewpoints while ensuring that the accuracy is not reduced remains a great challenge.

[0004] Therefore, how to provide a multi-perspective characterization method based on deep learning is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] In view of the above-mentioned prior art, the present invention aims to propose a Progressive Deep Multi-view Fusion (PDMF) method for learning multi-view data that explicitly considers the consistency and complementarity between different views. The present invention solves at least the following technical problems:

[0006] (1) The problem of missing perspective: How to ensure that the performance of the model will not be degraded when the perspective is missing.

[0007] (2) Multiple multimodal data processing issues: How to enable the model to process arbitrary multimodal data.

[0008] (3) The problem of mining the complementarity and consistency information of multi-perspective data: How to make the model fully utilize the complementarity and consistency of multi-perspective data.

[0009] In order to achieve the above objectives, the technical solutions of the present invention are as follows.

[0010] In a first aspect, a progressive multi-view deep comprehensive representation learning method is provided, wherein the learning method comprises a pre-training stage and a fine-tuning stage;

[0011] In the pre-training phase, the following steps are included:

[0012] Build a relationship matrix and decoder neural network for each view;

[0013] The first comprehensive representation is represented as a high-level representation using a relationship matrix, and each high-level representation is decoded into each specific viewpoint data based on a decoder neural network;

[0014] At the end of training, the first comprehensive representation and the sparse relationship matrix corresponding to each perspective are obtained;

[0015] During the fine-tuning training phase, the following steps are included:

[0016] Establish corresponding encoder neural network and multi-view sparse batch normalization layer for each view;

[0017] Encode the specific viewpoint data using an encoder neural network to obtain a second comprehensive representation;

[0018] The second comprehensive representation is scaled and translated to achieve alignment through a multi-view sparse batch normalization layer, and the scaling factor is achieved through a sparsified relation matrix constraint;

[0019] The aligned second comprehensive representation is averaged to obtain a third comprehensive representation, and the first comprehensive representation is used to guide the learning of the third comprehensive representation during training;

[0020] At the end of training, the trained decoder neural network is obtained.

[0021] The above technical solution establishes an easier pre-training task, in which a relationship matrix is ​​established for each viewpoint. Based on the relationship matrix and the first comprehensive representation, the high-level representation corresponding to each viewpoint is obtained, so that the corresponding viewpoint data can be reconstructed based on the high-level representation through the decoding neural network. A sparse relationship matrix is ​​obtained by adding a structured sparse regularizer to the relationship matrix, making some basis vectors of the matrix zero:

[0022]

[0023] W v is the sparse relationship matrix of the v-th perspective, D is the dimension of comprehensive representation, is the dimension of each perspective. The sparse relationship matrix makes the v-th perspective independent of the zero-value vector corresponding to If If a basis vector in the sparse relationship matrix corresponding to a certain dimension is 0, then this dimension represents the complementary information of multi-view data; if If the columns of all perspective relationship matrices corresponding to a certain dimension of are not zero, then this dimension is associated with all perspectives. This also indicates that this dimension captures the consistency of all perspectives.

[0024] In the above technical solution, the pre-training stage obtains the first comprehensive representation and the sparse relationship matrix corresponding to each perspective. One implementation method is to solve the following optimization problem:

[0025]

[0026] Where:

[0027] g v is the decoder neural network corresponding to the vth view, and V is the total number of views;

[0028] is the first comprehensive representation corresponding to the nth sample, and N is the total number of samples;

[0029] The first comprehensive representation corresponding to the nth sample is used to reconstruct the reconstruction error of the view data using the relationship matrix and the decoder neural network;

[0030] is the classification loss based on the first comprehensive representation;

[0031] α1 and α2 are hyperparameters.

[0032] As an improvement to the above technical solution, the classification loss This not only penalizes incorrect classifications but also ensures structured representation and makes the comprehensive representations of similar samples similar. A specific calculation method is as follows:

[0033]

[0034]

[0035] Where:

[0036] Represents the entire set of tags, is labeled Auxiliary comprehensive representation A collection of is an auxiliary comprehensive representation labeled y A collection of is the feature mapping function,

[0037] when hour, otherwise

[0038] In the above technical solution, the second comprehensive representation is scaled and translated by a multi-view sparse batch normalization layer to achieve alignment. In one embodiment, the specific calculation formula is:

[0039]

[0040] Where:

[0041] is the comprehensive representation of the output of the cth node of the lth layer of the network corresponding to the vth perspective of the nth sample;

[0042] It is the comprehensive representation of the output of the cth node of the l+1th layer of the network corresponding to the vth perspective of the nth sample;

[0043] and Represents the mean and standard deviation of all representation dimensions of the current batch of data respectively;

[0044] ∈ is the minimum value introduced to prevent the denominator from being zero;

[0045] is the scale factor, are offsets, all of which are values ​​to be optimized.

[0046] In the above technical solution, the scaling factor is implemented by sparse relation matrix constraints. A specific implementation method includes the following:

[0047] Based on the sparse relationship matrix, we obtain the basis vector of the c-th dimension representation of the v-th perspective, and obtain the relationship value after the c-th dimension comprehensive representation of the v-th perspective is fused:

[0048]

[0049] Create a function Among them, σ(.) is the sigmoid activation function, is the Lth encoder neural network corresponding to the vth view v The scaling factor of the c-th dimension comprehensive representation of the layer, 1≤c≤D, D represents the dimension of the comprehensive representation, 1≤v≤V, V is the total number of viewing angles, a and b are auxiliary parameters, a and b are the values ​​to be optimized;

[0050] By making the function Minimize gain a and b.

[0051] In the above technical solution, the trained decoder neural network is obtained in the fine-tuning stage. One implementation method is to solve the following optimization problem:

[0052]

[0053] Where:

[0054] f v is the decoder neural network corresponding to the vth view, and V is the total number of views;

[0055] N b is the number of samples in a batch in the batch normalization layer;

[0056] λ1 and λ2 are hyperparameters;

[0057] L t (z n ) is based on the third comprehensive representation z n Classification loss for classification;

[0058] t is the index of the current training batch;

[0059] δ t =max(1,20 / t) is the annealing coefficient;

[0060] The auxiliary loss includes the loss generated by using the sparsified relationship matrix to constrain the scaling factor for scaling and the loss generated by using the first comprehensive representation to guide the learning of the third comprehensive representation, as follows:

[0061]

[0062] z n For the third comprehensive representation, is the first comprehensive representation, ζ>0, is a hyperparameter, V is the total number of viewpoints, D is the dimension of comprehensive representation, and σ(·) is the sigmoid activation function;

[0063] is the Lth encoder neural network corresponding to the vth view v The scaling factor of the c-th dimension comprehensive representation of the layer, 1≤c≤D, where D represents the dimension of the comprehensive representation.

[0064] In the above technical solution, the aligned second comprehensive representation is averaged to obtain the third comprehensive representation. A specific implementation method is as follows:

[0065]

[0066] z n For the third comprehensive representation, is the second comprehensive representation of the alignment corresponding to the vth perspective, where V is the total number of perspectives.

[0067] In a second aspect, a progressive multi-view deep comprehensive representation learning and application method is provided, the method comprising the following contents:

[0068] Each view corresponds to a trained encoder deep neural network and a multi-view sparse batch normalization layer;

[0069] Input the multi-view data into the trained encoder deep neural network to obtain the aligned second comprehensive representation;

[0070] The second comprehensive representation is scaled and translated to achieve alignment through a multi-view sparse batch normalization layer;

[0071] The aligned second comprehensive representation is averaged to obtain the third comprehensive representation;

[0072] in:

[0073] The scaling factor for achieving scaling is constrained by the sparsified relation matrix during the training phase, and the sparsified relation matrix is ​​obtained through pre-training.

[0074] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program capable of being loaded by a processor and executing any one of the above-described progressive multi-view deep comprehensive representation learning methods;

[0075] And / or stores a computer program that can be loaded by a processor and executes a progressive multi-view deep comprehensive representation learning application method. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 、 one A schematic diagram of learning and training of a progressive multi-view comprehensive representation learning model in an embodiment. DETAILED DESCRIPTION

[0077] One object of the present invention is to propose a Progressive Deep Multi-view Fusion (PDMF) method for learning multi-view deep representations. This progressive multi-view comprehensive learning strategy can explicitly consider the consistency and complementarity between different views of multi-view data.

[0078] Splicing-based multi-view representation learning (MSRL) typically assumes that all viewpoints are complete. However, in the real world, some data instances may lack some viewpoints. In the absence of viewpoints, the performance of most MSRL methods inevitably degrades or even fails. Because MSRL is based on splicing and fusion, it is not applicable to the case of missing viewpoints. To address the issue of missing viewpoints, the PDMF method pre-aligns representations for specific viewpoints.

[0079] Most existing feature alignment-based multi-view representation learning (MARL) methods fail to fully exploit the complementary information in multi-view data, i.e., some views may contain information that others do not. Certain dimensions of the representation of a particular aligned view should not be relevant for all views, but only for certain dimensions. The PDMF method pre-establishes a sparse batch normalization layer to fully exploit the complementary information in multi-view data.

[0080] To alleviate the MARL problem, Wang et al. (Wang, Y.; Huang, W.: Sun, F.; Xu, T.; Rong, Y.; and Huang, J. 2020. Deep multimodal fusion by channel exchanging. Advances in Neural Information Processing Systems, 33.) proposed a new MARL dimension exchange strategy. However, the training of this model is greatly affected by the initial value of the parameters, so this model is not very stable. Nagrani et al. (Nagrani, A.; Yang, S.; Arnab, A.; Jansen, A.; Schmid, C.; and Sun, C. 2021. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34.) proposed a fusion method based on common and unique semantic features of perspectives. Then, this method only defines the association between a pair of perspectives. Therefore, its complexity will increase exponentially with the increase in the number of perspectives, so this model only contains two perspectives. Compared to the aforementioned methods, this invention establishes a simpler pre-training task, better initializes consistent and complementary dimensions, and can handle multiple viewpoints. Furthermore, this invention defines learnable associations between multi-view data and comprehensive representations, rather than simply establishing associations between any two views. Therefore, it is applicable to multi-view scenarios with any number of viewpoints.

[0081] In terms of multi-view pre-training, the PDMF of the present invention is superior to other models of multimodal pre-training. Most multimodal pre-training methods can only process two specific modalities, for example, visual modality data (images or videos) and language modality data. The PDMF method of the present invention can process arbitrary multi-view data. In addition, multimodal pre-training requires large-scale training data, while the PDMF of the present invention is applicable to different data sizes.

[0082] The following will clearly and completely describe the technical solutions in the embodiments of this application in conjunction with the accompanying drawings. The terms "first," "second," and "third" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features specified as "first," "second," and "third" may explicitly or implicitly include one or more of such features.

[0083] In a specific implementation of the progressive multi-view deep comprehensive representation learning method, the following steps are included:

[0084] S1. Sample definition, assuming that the data set has N samples and each sample has V perspectives.

[0085] In the sample definition, the defined sample parameters are shown in Table 1.

[0086] Table 1

[0087]

[0088]

[0089] S2, pre-training stage, establish the relationship matrix And build the decoder neural network.

[0090] S2-1. Establish a relationship matrix It is used to measure the strength of the relationship between each perspective and the first comprehensive representation.

[0091] Building the decoder neural network Used to learn specific view data and high-level representation Nonlinear mapping relationship.

[0092] The goals of the decoding process are:

[0093]

[0094] is the reconstruction error function. is the square of the L2 norm. N is the number of samples, V is the number of viewpoints, g v(·) is the decoding neural network corresponding to the vth neuron, is the real number field of the first comprehensive representation, D is the dimension of the first comprehensive representation, is the real field of the relation matrix, Dimensions for each perspective.

[0095] For each W v Add a structured sparse regularizer so that W v Some columns of are zero vectors, so that the v-th perspective is independent of the zero-value vector corresponding to Dimensions:

[0096]

[0097] is the sparse relationship matrix W v The jth element in the i-th row of .

[0098] like Figure 1 As shown, W 1 The third column of is zero, which means The third dimension of has nothing to do with perspective 1 and reflects the complementarity between perspectives. If the columns of all perspective relationship matrices corresponding to a certain dimension of are not zero, it means that the dimension is associated with all perspectives and the dimension captures the consistency of all perspectives.

[0099] S2-2, based on The classification function of Capturing the conceptual structure of multi-perspective data:

[0100]

[0101] in: A conceptual structure for multi-perspective data. Represents the entire set of tags, is an auxiliary comprehensive representation labeled y A collection of is the feature mapping function,

[0102] Based on comprehensive consideration of classification and concept structure learning, the classification loss function should not only penalize incorrect classification, but also ensure structured representation and make the comprehensive representation of similar samples similar. In a specific embodiment, the classification loss function is specified as:

[0103]

[0104] in, is the classification loss function. hour, otherwise, is labeled Auxiliary comprehensive representation A collection of is labeled y n Auxiliary comprehensive representation A collection of .

[0105] S2-3, the overall optimization problem in the pre-training stage is:

[0106]

[0107] in: is the reconstruction error function, α1 and α2 are hyperparameters. In one embodiment, α1 = 0.01, α2 = 0.05.

[0108] In the fine-tuning stage, a multi-view sparse batch normalization layer (MSBN) is established and a comprehensive view representation is learned.

[0109] S3-1: Create an encoder for each view To learn aligned view-specific representations

[0110]

[0111] In one embodiment, the encoder It is a fully connected neural network. As an improvement to the technical solution, a batch normalization layer (BN) is added to each encoder layer:

[0112]

[0113]

[0114]

[0115] in, is the c-th dimension representation of the output of the l-th layer, and are respectively the trainable scale factor and offset, which are used to scale and translate the normalized data. ∈ is the minimum value introduced to prevent the denominator from being zero. and Represents the mean and standard deviation of all representation dimensions of the current batch of data. b is the number of samples in a batch. is the number of nodes in the lth layer of the network corresponding to the vth perspective. l ≥ 2.

[0116] S3-2: Due to the complementarity of multi-view data, Some dimensions in the top layer should be related to z n Therefore, in one embodiment, a multi-view sparse batch normalization layer (MSBN) is designed to model sparse connections. And this sparsity constraint is added to the scale factor In the figure above, Lv is the layer marked as L in the multi-view sparse batch normalization layer corresponding to the v-th view.

[0117] pass To evaluate and The correlation between It loses its influence on the vth perspective, and The value of It has nothing to do with it; The larger the value, the higher the correlation between the two.

[0118] because and z n are of the same dimension, when the vth perspective and the comprehensive representation z n When the cth feature of is irrelevant (for example, the patient's blood test perspective does not include the feature of the lesion location), we hope The cth eigenvalue of is 0, so The c-th dimension of z n The calculation of the cth dimension has no contribution.

[0119] First, the relationship matrix W of each perspective v There is a sparse relationship between perspective and comprehensive representation, using the relationship matrix To guide Specifically, from W v Extract z n Relationship with perspective v:

[0120] represents the dimension of the v-th view, or the number of features. Then, by minimizing You can get a better Where a and b are trainable parameters. σ(·) is the sigmoid activation function, which forces Distributed in (0, 1).

[0121] In S3-2 above, a multi-view sparse batch normalization layer is designed to model sparse connections, which can be used in most popular MARL methods. This layer facilitates the fusion model to flexibly integrate inter-view information while preserving intra-view information. The invention imposes this sparsity constraint on the scaling factor. and use the relationship matrix obtained in part S2 To guide Specifically, from W v Extract z n Relationship with perspective v:

[0122]

[0123] Then, by minimizing You can get a better like Figure 1 As shown in the fine-tuning stage, the multi-view sparse batch normalization is obtained The third dimension of is 0, The third dimension of is not zero, which means that the first perspective does not contain information about the third dimension of comprehensive representation, while the vth perspective contains the corresponding information. This just shows that the complementarity between multiple perspectives can be fully exploited under this implementation.

[0124] S3-3: Through the above steps S3-1 and S3-2, the representation of a specific perspective can be well aligned, directly Take the average and get the multi-view comprehensive representation z n :

[0125]

[0126] Use the auxiliary comprehensive representation obtained in step S2 Guidance n Learning, the entire auxiliary loss is defined as:

[0127]

[0128] Where ζ>0 is a hyperparameter, for example, ζ=0.1. The overall optimization problem of the entire fine-tuning phase is:

[0129]

[0130] Among them, the first one is the loss function of the classification task of comprehensive expression, namely L t (z n ) is based on the z of each sample n The loss function of the downstream tasks (classification, clustering, etc.) is performed. λ1 and λ2 are hyperparameters, such as λ1 = 0.1 and λ2 = 0.01. δ t=max(1, 20 / t) is the annealing coefficient, t is the index of the current training batch. Training stops when the specified training batch is reached.

[0131] In another embodiment, the above-mentioned progressive multi-view deep comprehensive representation learning method is applied, and the application mainly includes the following contents:

[0132] Each view corresponds to a trained encoder deep neural network and a multi-view sparse batch normalization layer;

[0133] Input the multi-view data into the trained encoder deep neural network to obtain the aligned second comprehensive representation;

[0134] The second comprehensive representation is scaled and translated to achieve alignment through a multi-view sparse batch normalization layer;

[0135] The aligned second comprehensive representation is averaged to obtain the third comprehensive representation;

[0136] in:

[0137] The scaling factor for achieving scaling is constrained by the sparsified relation matrix during the training phase, and the sparsified relation matrix is ​​obtained through pre-training.

[0138] Through the above description of the embodiments, those skilled in the art will clearly understand that the present disclosure can be implemented using software plus necessary general-purpose hardware. Of course, it can also be implemented using dedicated hardware, including application-specific integrated circuits, dedicated CPUs, dedicated memories, and dedicated components. Generally speaking, any function performed by a computer program can be easily implemented using corresponding hardware. Moreover, the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present disclosure, software implementation is often the preferred embodiment.

[0139] In summary, the present invention has the following beneficial technical effects:

[0140] (1) A progressive multi-view comprehensive learning strategy is proposed, which is divided into two stages: pre-training and fine-tuning, and explicitly considers the consistency and complementary information of multi-view data.

[0141] (2) A relationship matrix was established to measure the relationship strength between each perspective and the comprehensive representation. v Add a structured sparse regularizer so that W v Some columns of are 0 vectors:

[0142]

[0143] This makes the vThe perspective is independent of the zero-valued vector corresponding to dimension.

[0144] (3) In order to Capturing the conceptual structure of multi-perspective data ensures Being structurally separable, we built a predictive classification scheme similar to a clustering algorithm:

[0145]

[0146] Among them, Y represents the entire set of labels, and T(y) is the auxiliary comprehensive representation of the label y A collection of is the feature mapping function, By comprehensively considering classification and concept structure learning, the classification loss function is specified as:

[0147]

[0148] Among them, when hour, otherwise This classification loss function not only penalizes incorrect classifications, but also ensures structured representation and makes the comprehensive representations of similar samples similar.

[0149] (4) We establish an MSBN layer to fuse the aligned view-specific representations, which can be used in most popular MARL methods. This layer facilitates the fusion model to flexibly integrate inter-view information while preserving intra-view information.

[0150] (5) It is not limited to processing 2-view data, but can process arbitrary multi-view data. The dimension of multi-view data can be 3, 4, 5, 6, 10, 15... When the dimension is very large, it can be achieved by selecting hardware with good performance or through distributed computing.

[0151] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. A progressive multi-view deep comprehensive representation learning method, characterized by: The learning method includes a pre-training phase and a fine-tuning phase; In the pre-training phase, the multi-view data processed includes visual modality data and language modality data, including the following steps: Build a relationship matrix and decoder neural network for each view; The first comprehensive representation is represented as a high-level representation using a relationship matrix, and each high-level representation is decoded into each specific viewpoint data based on a decoder neural network; At the end of training, the first comprehensive representation and the sparse relationship matrix corresponding to each perspective are obtained; In the pre-training phase, the first comprehensive representation and the sparse relationship matrix corresponding to each perspective are obtained by solving the following optimization problem: Where: is the decoder neural network corresponding to the vth view, and V is the total number of views; is the first comprehensive representation corresponding to the nth sample, and N is the total number of samples; The first comprehensive representation corresponding to the nth sample is used to reconstruct the reconstruction error of the view data using the relationship matrix and the decoder neural network; is the classification loss based on the first comprehensive representation; is a hyperparameter; The sparse relationship matrix is ​​obtained by adding a structured sparse regularizer to the relationship matrix so that some basis vectors are 0: For the The sparse relationship matrix of perspectives, D is the dimension of comprehensive representation, Dimensions for each perspective; During the fine-tuning training phase, the following steps are included: Establish corresponding encoder neural network and multi-view sparse batch normalization layer for each view; Encode the specific viewpoint data using an encoder neural network to obtain a second comprehensive representation; The second comprehensive representation is scaled and translated to achieve alignment through a multi-view sparse batch normalization layer, and the scaling factor is achieved through a sparsified relation matrix constraint; The aligned second comprehensive representation is averaged to obtain a third comprehensive representation, and the first comprehensive representation is used to guide the learning of the third comprehensive representation during training; At the end of training, the trained decoder neural network is obtained.

2. The method according to claim 1, characterized in that Classification loss The calculation is as follows: Where: Represents the entire set of tags, is labeled Auxiliary comprehensive representation A collection of is labeled Auxiliary comprehensive representation A collection of is the feature mapping function, when hour, ,otherwise .

3. The method according to claim 1, characterized in that The second comprehensive representation is scaled and translated to achieve alignment through the multi-view sparse batch normalization layer. The specific calculation formula is: Where: For the The first sample The first perspective of the network Layer Comprehensive representation of node output; For the The first sample The first perspective of the network Layer Comprehensive representation of node output; and Represents the mean and standard deviation of all representation dimensions of the current batch of data respectively; It is a minimum value introduced to prevent the denominator from being zero; is the scale factor, are offsets, all of which are values ​​to be optimized.

4. The method according to claim 3, characterized in that The scaling factor is achieved by sparsifying the relation matrix constraints, including: Based on the sparse relationship matrix, we can obtain the The c-th dimension representation basis vector of the perspective is obtained The c-th dimension of each perspective comprehensively represents the fused relationship value: Create a function ,in, is the sigmoid activation function, is the first encoder neural network corresponding to the vth view The scaling factor of the c-th dimension comprehensive representation of the layer, , represents the dimension of comprehensive representation, , V is the total number of viewing angles, a and b are auxiliary parameters, , a and b are the values ​​to be optimized; By making the function Minimize gain , a and b.

5. The method according to claim 1, wherein The fine-tuning phase obtains a trained decoder neural network by solving the following optimization problem: Where: is the decoder neural network corresponding to the vth view, and V is the total number of views; is the number of samples in a batch in the batch normalization layer; 、 is a hyperparameter; Based on the third comprehensive representation Classification loss for classification; is the index of the current training batch; is the annealing coefficient; The auxiliary loss includes the loss generated by using the sparsified relationship matrix to constrain the scaling factor for scaling and the loss generated by using the first comprehensive representation to guide the learning of the third comprehensive representation, as follows: For the third comprehensive representation, For the first comprehensive characterization, , is a hyperparameter, V is the total number of viewpoints, D is the comprehensive representation dimension, is the sigmoid activation function; is the first encoder neural network corresponding to the vth view The scaling factor of the c-th dimension comprehensive representation of the layer, , Represents the dimension of comprehensive representation.

6. The method according to claim 1, characterized in that The aligned second comprehensive representation is averaged to obtain the third comprehensive representation, which is: For the third comprehensive representation, is the second comprehensive representation of the alignment corresponding to the vth perspective, where V is the total number of perspectives.

7. A progressive multi-view deep comprehensive representation learning application method, applied to the progressive multi-view deep comprehensive representation learning method according to claim 1, characterized in that: The method includes the following: Each view corresponds to a trained encoder deep neural network and a multi-view sparse batch normalization layer; Input the multi-view data into the trained encoder deep neural network to obtain the aligned second comprehensive representation; The second comprehensive representation is scaled and translated to achieve alignment through a multi-view sparse batch normalization layer; The aligned second comprehensive representation is averaged to obtain the third comprehensive representation; in: The scaling factor for achieving scaling is constrained by the sparsified relation matrix during the training phase, and the sparsified relation matrix is ​​obtained through pre-training.

8. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 6; and / or stores a computer program that can be loaded by a processor and execute the method according to claim 7.

Images