Warpage-based multi-view fusion classification system and method

Abstract

A kind of multi-view fusion classification system and method based on morpher, comprising: image coding module, fusion module, morpher module and view classifier module, image coding module is respectively encoded after pre-processing multi-view data, obtains feature vector, fusion module is merged to vector, obtains high-dimensional feature vector and is input to morpher module, morpher module is learned to high-dimensional feature by attention mechanism, forward feedback, normalization operation and obtains high-order feature expression, view classifier module respectively multi-view fusion classification is carried out to high-order feature expression, and single view classification is carried out to the low-dimensional high-order feature after disassembling, obtains fusion result and single view result.The present application adopts the morpher based on self-attention, utilizes the correlation and complementarity existing between multi-view, discards time sequence position coding to speed up training speed, and simultaneously, using self-attention mechanism in morpher automatically learns the weight of each feature, so that network focuses on more effective feature to classification result, significantly improves classification performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a multi-view fusion classification method based on a transformer. BACKGROUND

[0002] In recent years, great progress has been made in solving the multi-view classification problem in an unsupervised and semi-supervised manner. Some methods perform multi-view learning based on classic co-training and co-regularization methods, which ignore the problems such as overfitting and high computational cost caused by high-dimensional multi-view data. Considering the influence of high-dimensional multi-view data, other methods perform multi-view learning based on canonical correlation analysis to maximize the correlation between different views in the latent subspace. Although these methods can effectively project high-dimensional features into a low-dimensional subspace, they only focus on the correlation between different views and ignore the complementary information of each view, which may lead to inefficient fusion of the original multi-view data. SUMMARY

[0003] The present application aims at the defects that the prior art cannot match the positions of the current search image and the target image and cannot simultaneously perform target recognition and classification, and proposes a multi-view fusion classification system and method based on a transformer. The multi-view fusion classification system and method based on the transformer uses the correlation and complementarity between multi-views, discards the time sequence position coding to speed up the training speed, and uses the self-attention mechanism in the transformer to automatically learn the weight of each feature, so that the network focuses on the features more effective for the classification result, thereby significantly improving the classification performance.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to a multi-view fusion classification system based on a transformer, which comprises an image encoding module, a fusion module, a transformer module and a view classifier module. The image encoding module encodes the preprocessed multi-view data to obtain a plurality of one-dimensional feature vectors. The fusion module merges the plurality of one-dimensional feature vectors to obtain a high-dimensional feature vector and inputs the high-dimensional feature vector into the transformer module. The transformer module learns the high-dimensional feature through an attention mechanism, forward feedback and normalization operation and obtains a high-order feature expression. The view classifier module performs multi-view fusion classification on the high-order feature expression and single-view classification on the disassembled low-dimensional high-order feature to obtain a fusion result and a single-view result.

[0006] This invention relates to a multi-view fusion classification method based on deformers. The method connects the encoded output of multiple views through an image encoding module to a high-dimensional feature vector. Then, a high-order matrix containing the global dependencies of all views is obtained through a cascaded scaling dot product attention technique and input to a multi-view fusion classification model composed of a multilayer perceptron. The vector is then decomposed into low-dimensional high-order vectors and input to a single-view classification model composed of fully connected layers and Softmax layers. This results in a multi-view fusion classification result that comprehensively considers the information of multiple views and a single-view classification result that represents the importance between views.

[0007] Technical effect

[0008] This invention obtains high-dimensional features by fusing the encoded one-dimensional feature vectors of multiple views, and then inputs these high-dimensional features into the deformer module, enabling the model to learn information from multiple views to a certain extent. Furthermore, decomposing the high-order features allows the model to learn and adjust for each view, increasing the model's classification performance. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the system of the present invention;

[0010] Figure 2 This is a schematic diagram of the deformer module;

[0011] Figure 3 This is a flowchart of an implementation example;

[0012] Figure 4 This is a schematic diagram of the system operation. Detailed Implementation

[0013] like Figure 1 As shown in this embodiment, a multi-view fusion classification system based on a deformer is included, comprising: an image encoding module, a fusion module, a deformer module, and a view classifier module. The image encoding module encodes the preprocessed multi-view data to obtain multiple one-dimensional feature vectors. The fusion module merges these one-dimensional feature vectors to obtain a high-dimensional feature vector, which is then input to the deformer module. The deformer module learns the high-dimensional features through attention mechanisms, forward feedback, and normalization operations to obtain high-order feature representations. The view classifier module performs multi-view fusion classification on the high-order feature representations and single-view classification on the decomposed low-dimensional high-order features, obtaining fusion results and single-view results.

[0014] The image coding module learns features from original data in a matrix form, improves the expression ability of the features, and facilitates the morpher to better learn the features, the input picture size of the module is 224*224, and the output feature vector size is 1*n, and the module specifically comprises: a plurality of convolution layers and pooling layers in succession, wherein: the convolution layer respectively performs convolution processing on a plurality of view images and obtains local feature information of each view, and the pooling layer further performs down-sampling according to the local feature information output by the convolution layer.

[0015] The merging is specifically referring to: merging the k 1*n-dimensional feature rows to obtain a k*n-dimensional feature matrix including image feature information of multiple views.

[0016] As shown in Figure 2 The morpher module includes a multi-head attention mechanism unit, a feed-forward propagation layer and two addition normalization layers, wherein: the multi-head attention mechanism unit encodes the merged high-dimensional feature matrix and learns the global dependency relationship, obtains a plurality of weight values, and respectively multiplies the high-dimensional feature matrix to obtain a plurality of weighted feature matrices; the first addition normalization layer adds and normalizes the high-dimensional feature matrix and the plurality of weighted feature matrices, to obtain a plurality of residual normalized matrices; the feed-forward propagation layer combines the plurality of residual normalized matrices and multiplies a randomly initialized matrix to obtain a feed-forward matrix; and the second addition normalization layer adds and normalizes the feed-forward matrix and the plurality of weighted feature matrices to obtain high-order features.

[0017] The multi-head attention mechanism unit is a plurality of parallel scaled dot-product attention modules stacked and packaged, which outputs the results after parallel computing a plurality of Q, K, V attention values and splicing, wherein: each scaled dot-product attention module includes a Query vector (Q), a Key vector (K) and a Value vector (V), all of which have a length of 64, which calculates the similarity between the Query vector and the Key vector by multiplying the three different weight matrices WQ, WK and WV respectively, and normalizes the weight by using the Softmax function, and finally the Value vector obtains the weighted sum of the weight matrices WQ, WK and WV, and the output result does not change the size of the feature matrix, so the output high-order feature size is k*n.

[0018] The view classifier module includes a single view classifier and a multi-view fusion classifier, wherein: the single view classifier decomposes the output high-order features of the morpher module into k 1*n-dimensional low-dimensional high-order features corresponding to each view, respectively obtains the predicted class probability value of each view, then calculates the loss difference value of each view probability value and the class label value, iterates the model multiple times and continuously updates the parameters so that the loss difference value can reach the optimal value; and the multi-view fusion classifier classifies multiple views according to the high-order features output by the morpher module.

[0019] The single-view classifier includes a fully connected layer and a softmax layer. The fully connected layer maps the learned distributed feature representation to the sample label space to obtain a one-dimensional feature vector. The softmax layer enables the real number domain of the model output to be mapped to the effective real number space [0,1] representing the probability distribution to obtain the class probability value. Finally, the average method is used to calculate the average of the class probability values ​​of each view, and an appropriate probability threshold is selected to divide the predicted class.

[0020] The multi-view fusion classifier includes a multilayer perceptron and a softmax layer. The multilayer perceptron continuously learns higher-order feature vectors and simultaneously adjusts the weights and biases of its nodes. The softmax layer maps the real-number domain of the model output to the effective real-number space [0,1] representing the probability distribution, thus obtaining the class probability value. A threshold is set for the probability value to obtain the predicted class.

[0021] The multilayer perceptron includes two sets of batch normalized layers and dense layers, wherein: the first dense layer includes activation of a Gaussian error linear unit (GELU) based on 120 neurons, and the second dense layer has Softmax activation and uses L2 regularization to minimize overfitting as much as possible.

[0022] like Figure 3 As shown in the figure, this embodiment relates to a multi-view fusion classification method based on the above system. The high-dimensional feature vectors of the encoded output of multiple views are connected by an image encoding module. Then, a high-order matrix containing the global dependencies of all views is obtained through a cascaded scaling dot product attention technique and input into a multi-view fusion classification model composed of a multilayer perceptron. After being decomposed into low-dimensional high-order vectors, the vectors are input into a single-view classification model composed of a fully connected layer and a Softmax layer. Thus, the multi-view fusion classification result that comprehensively considers the information of multiple views and the single-view classification result that represents the importance between views are obtained respectively.

[0023] The method specifically includes:

[0024] Step 1: Obtain multi-angle, multi-faceted, and continuous video frame views of the object as a dataset. Each training set sample has k views, and all views are processed to a size of 224×224 with 3 channels.

[0025] Step 2: The purpose of creating the dataset is to divide it into a training set and a test set. The training set is used to train the network model, while the test set is used to evaluate the model's performance. The training and test sets should be divided in a 4:1 ratio.

[0026] Step 3: Build a multi-view fusion classification model based on deformers. The specific steps are as follows:

[0027] Step 3.1, image encoding: first, the image is represented as R H*W*C wherein: H represents the image height, W represents the image width, and C represents the number of channels. Each view is encoded separately, and the output feature vector is 1 x n dimension. Then, the encoded output blocks of the k views are fused to obtain a high-dimensional feature vector space of k x n dimensions. That is, each group of k views is encoded, and the encoding convolution kernel size is 3 x 3. After encoding, the output feature vector is 1 x n; the feature space includes the feature information of all views.

[0028] The fusion is specifically: wherein A i is the encoded vector feature of the i-th view, is a row splicing operation, Z is a high-dimensional feature, and the output high-dimensional feature is k x n dimension.

[0029] Step 3.2, input the high-dimensional vector feature after image encoding into the morpher module for feature learning to obtain a high-order feature expression containing the global dependency relationship of all views.

[0030] Step 3.3, decompose the high-order feature expression output by the morpher module into a k x n high-order feature matrix, and then further decompose it into k 1 x n low-dimensional feature vectors and input them into the single-view classification model for class prediction to obtain the probability value of each low-dimensional feature vector.

[0031] Step 3.4, input the high-order feature expression output by the morpher module as a whole into the multi-view fusion classification model, calculate the loss function between the predicted class probability value and the label value, optimize the parameters to make the loss function optimal for training, and then obtain the probability value to determine the class of the sample.

[0032] Step 4, use the NovoGrad optimizer with the classification cross-entropy loss function and the training set of step 2 to train the multi-view fusion classification model, train the binary cross-entropy loss in the case of binary classification, and then use the test set to detect the performance of the model from the accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC).

[0033] Through specific experiments, under the hardware environment of Nvidia RTX2080 with 8GB display memory, the batch size is set to 16, the initial learning rate is set to 0.002, the Adam optimizer and the cross-entropy loss function are used, the iteration number is set to 200, 500 ultrasound video data are used, including two types of left ventricular hypertrophy diseases: hypertrophic cardiomyopathy (HCM) and hypertensive heart disease (HHD), and each type of disease has 250 cases. The experimental results are as follows: the area under the receiver operating characteristic curve (AUC) of the HCM and HHD classification experiment is 0.848, the accuracy is 85.3%, the sensitivity is 92.3%, and the specificity is 78.2%. For the single-view classifier, the AUC of the HCM and HHD classification experiment is 0.817, the accuracy is 86.3%, the sensitivity is 89.2%, and the specificity is 73.6%.

[0034] Compared with the prior art, the classification AUC, accuracy, sensitivity and specificity of the present application are all better than those of the prior art.

[0035] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the present application.

Claims

1. A morpher-based multi-view fusion classification system, comprising: The application relates to a multi-view image classification method and device. The image coding module codes the preprocessed multi-view data respectively to obtain a plurality of one-dimensional feature vectors, the fusion module combines the plurality of one-dimensional feature vectors to obtain a high-dimensional feature vector and inputs the high-dimensional feature vector into the morpher module, the morpher module learns the high-dimensional feature through an attention mechanism, forward feedback and normalization operation and obtains a high-dimensional high-order feature, and the view classifier module respectively performs multi-view fusion classification on the high-dimensional high-order feature and single-view classification on the decomposed low-dimensional high-order feature to obtain a fusion result and a single-view result. The combining specifically refers to that k one-dimensional feature rows are combined to obtain a k*n-dimensional feature matrix including image feature information of a plurality of views, that is, a high-dimensional feature vector.

2. The morpher-based multi-view fusion classification system of claim 1, wherein, The image coding module includes a plurality of continuous convolution layers and pooling layers, wherein the convolution layers respectively perform convolution processing on the inputted plurality of view images and obtain local feature information of each view, and the pooling layers further perform down-sampling according to the local feature information outputted by the convolution layers.

3. The morpher-based multi-view fusion classification system of claim 1, wherein, The morpher module includes a multi-head attention mechanism unit, a feedforward propagation layer and two addition normalization layers, wherein the multi-head attention mechanism unit encodes the combined high-dimensional feature matrix and learns a global dependency relationship, obtains a plurality of weight values, respectively multiplies the high-dimensional feature matrix by the plurality of weight values to obtain a plurality of weighted feature matrices, the first addition normalization layer adds and normalizes the high-dimensional feature matrix and the plurality of weighted feature matrices to obtain a plurality of residual normalized matrices, the feedforward propagation layer combines the plurality of residual normalized matrices, multiplies the plurality of residual normalized matrices by a randomly initialized matrix to obtain a feedforward matrix, and the second addition normalization layer adds and normalizes the feedforward matrix and the plurality of weighted feature matrices to obtain the high-dimensional high-order feature.

4. The morpher-based multi-view fusion classification system of claim 3, wherein, The multi-head attention mechanism unit is a plurality of parallel scaled dot-product attention modules stacked and packaged, parallel calculation of attention values of a plurality of Q, K and V is performed, and the result is spliced and outputted, wherein each scaled dot-product attention module includes a Query vector Q, a Key vector K and a Value vector V, the lengths of the vectors are all 64, the Query vector and the Key vector are calculated for similarity through multiplication by three different weight matrices WQ, WK and WV respectively, the weight is normalized by using a Softmax function, finally the Value vector obtains a weighted sum of the weight matrices WQ, WK and WV, and the high-dimensional high-order feature has a size of k*n.

5. The morpher-based multi-view fusion classification system of claim 1, wherein, The view classifier module includes a single-view classifier and a multi-view fusion classifier, wherein the single-view classifier decomposes the high-dimensional high-order feature outputted by the morpher module into k one-dimensional low-dimensional high-order features corresponding to each view, respectively obtains a prediction class probability value of each view, then calculates a loss difference value between each view probability value and a class label value, the model is iterated for multiple times and the parameters are continuously updated so that the loss difference value can reach an optimal value, and the multi-view fusion classifier performs multi-view classification according to the high-dimensional high-order feature outputted by the morpher module.

6. The morpher-based multi-view fusion classification system of claim 5, wherein, The single-view classifier comprises a full connection layer and a Softmax layer; and the multi-view fusion classifier comprises a multi-layer perception and a Softmax layer.

7. The morpher-based multi-view fusion classification system of claim 6, wherein, The multi-layer perception comprises two groups of batch normalization layers and dense layers, wherein: the first dense layer comprises a Gaussian error linear unit (GELU) based activation with 120 neurons, the latter dense layer has a Softmax activation, and L2 regularization is used to reduce overfitting as much as possible.

8. A multi-view fusion classification method based on the multi-view fusion classification system based on the morpher according to any one of claims 1-7, characterized in that, The multi-view coding output high-dimensional feature vectors are connected through an image coding module, and then a high-order matrix input containing global dependency relationships of all views is obtained through a concatenated scaled dot product attention technique and input into a multi-view fusion classifier composed of a multi-layer perception, and the decomposed low-dimensional high-order features are input into a single-view classifier composed of a full connection layer and a Softmax layer, so as to obtain a multi-view fusion classification result considering multi-view information and a single-view classification result representing the importance between views respectively.

Images