Multi-view image classification method based on deep cross entropy integration

Abstract

The invention discloses a multi-view image classification method based on deep cross entropy integration, and the method comprises the steps: constructing a data set based on an image, and dividing the data set into a training set and a test set; constructing a network architecture of a cross entropy integrated image classification model, wherein the cross entropy integrated image classification model comprises a backbone network, a prototype network and an integrated classification network; the processing process in the backbone network part is as follows: encoding an input image into a feature map # imgabs0 # prototype network part, and projecting the feature map # imgabs1 # to a prototype feature P to obtain a group of diversified feature representations V; the ensemble classification network performs weighted fusion on a plurality of view representations V = {V1, V2... VN} of the input image, and calculates ensemble cross entropy loss; training the cross entropy integrated image classification model by using the training set; according to the method, the requirements of different data sets and tasks are met, and the inherent problems of a traditional multi-view image classification method are solved.

Description

Technical Field

[0001] The present invention relates to image classification technology, in particular to a multi-view image classification method based on deep cross entropy integration. Background Art

[0002] At present, deep learning methods have made significant progress in image classification technology, especially the application of Convolutional Neural Networks (CNNs). However, traditional single-view image classification methods often face certain limitations when processing complex images. The information from a single perspective is often not enough to fully describe the features of the image, thus affecting the classification performance. In order to overcome this problem, multi-view image classification methods have emerged, which enrich the feature representation by constructing multiple views of the data, thereby improving the robustness and generalization ability of the classification.

[0003] Existing multi-view image classification methods mainly include two categories: feature fusion and decision fusion. The feature fusion method combines features from different perspectives to construct a high-dimensional feature space for classifiers. However, this method easily leads to excessive feature dimensions, increasing computational complexity and overfitting risks. The decision fusion method performs weighted averaging or voting on the output results of multiple independent classifiers to obtain the final classification result. Although it alleviates the feature dimension problem to a certain extent, the correlation between classifiers may lead to unsatisfactory fusion effects.

[0004] In addition, traditional multi-view image classification methods often adopt fixed strategies in feature extraction and classifier design, which lack flexibility and are difficult to adapt to the needs of different datasets and tasks. Summary of the invention

[0005] In order to address the deficiencies in the prior art, this application proposes a multi-view image classification method with deep cross entropy integration. By designing an attention prototype network, different prototype representations are learned, the input features are projected onto these prototypes, and encoded with attention prototype vectors to generate different feature sets. Then, classifiers are trained for each feature set, and these classifiers are combined into a nonlinear classifier. Finally, the classification results are optimized by calculating the integration loss. This method can effectively capture the features of different viewpoints of the image and perform feature fusion, thereby improving the performance of image classification.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A multi-view image classification method based on deep cross entropy integration comprises the following steps:

[0008] Step 1: Obtain images and label them, construct a data set based on images and labels, and divide the data set into a training set and a test set;

[0009] Step 2: construct a network architecture of a cross entropy integrated image classification model, wherein the cross entropy integrated image classification model includes three parts: a backbone network, a prototype network, and an integrated classification network;

[0010] The processing process in the backbone network part is: encoding the input image into a feature map in, Represents the image dimension, H, W, C represent the height, width and number of channels of the input image respectively;

[0011] The prototype network part maps the feature Project it onto the prototype feature P to obtain a set of diverse feature representations V;

[0012] In the integrated classification network, multiple views of the input image are represented as V = {V 1 ,V 2 …V N}Perform weighted fusion and calculate integrated cross entropy loss;

[0013] Step 3: Use the training set in step 1 to train the cross entropy ensemble image classification model constructed in step 2;

[0014] Step 4: Use the trained cross entropy ensemble image classification model to classify images.

[0015] Furthermore, the dataset is divided into training set and test set in a ratio of 6:4.

[0016] Furthermore, the acquired data set is preprocessed, and the preprocessing specifically includes: data enhancement, image conversion, and normalization.

[0017] Furthermore, the backbone network uses a ResNet-18 network framework to extract image features to form a feature map.

[0018] Further, the processing process in the prototype network part is:

[0019] S1. Map the feature encoded by the backbone network Converted to a vector, it is represented as follows:

[0020]

[0021] Among them, Q, K, V are query, key, and value respectively, and W Q , W K , W V are weight matrices corresponding to query, key, and value respectively;

[0022] S2. Apply the attention mechanism to obtain diverse prototype features, which are expressed as follows:

[0023]

[0024] in, is a scaling factor; Softmax is a normalization function that makes the weighted probability distribution and 1; P is composed of multiple prototype features P n The formed set, P = {P 1 ,P 2 ,…,P N}, n represents the nth prototype, N represents the total number of prototypes, It is the dimension of the archetype;

[0025] S3. For each feature in the feature map Find feature x i,j With prototype feature P n The relative difference between i,j =x i,j -P n ;

[0026] S4. Based on the relative difference, the input image is converted into a multi-view representation V = {V 1 ,V 2 …V N}, where N represents the total number of multi-view representations, which is consistent with the total number of prototypes, and is expressed as follows:

[0027]

[0028]

[0029] Among them, V n is the representation of the nth view, θ n Represents the feature map x i,j With the nth prototype feature P n The correlation coefficient between them, H and W are the height and width of the picture respectively;

[0030] Through the above steps, the feature map Projected onto the prototype feature P, a set of diverse feature representations V is obtained.

[0031] Further, the integrated cross entropy loss is expressed as follows:

[0032]

[0033] Among them, L is the integrated cross entropy loss, M is the total number of pictures, and y iis the true label of the i-th image, N is the number of base classifiers, which is consistent with the number of prototype features; π n is the weight of the nth base classifier, is the transpose of the nth view representation of the ith image, w j is the weight of the fully connected network, and R is the number of classification categories.

[0034] Further, the training process of step 3 is as follows:

[0035] Step 3.1: For the constructed cross entropy integrated image classification model, constraints need to be added during the training process, as shown below:

[0036]

[0037] Among them, η, ξ, τ are Lagrange multipliers, 1 T π means summing the weights;

[0038] Step 3.2: Back propagate to solve for w j , weight π n ;

[0039] For the parameter w in the previous step j Find the partial derivative to get the w of the nth base classifier j gradient:

[0040]

[0041] For the parameter π in the previous step n Taking partial derivatives we get:

[0042]

[0043] The obtained π n The integrated result is obtained by combining it with the cross entropy calculated by the base classifier, and then the parameters are continuously updated through back propagation until the loss function converges and the training is stopped.

[0044] Beneficial effects of the present invention:

[0045] 1. The present invention designs an attention prototype network, which obtains different prototypes by focusing on different local features, thereby converting a single view into a multi-view representation. This design can fully explore the potential features of the image and construct a semantically rich multi-view representation.

[0046] 2. The present invention adopts an integrated classification method in classifier design. Through weighted fusion, the individual cross entropy losses of multiple data views are unified into the overall integrated space, thereby improving the classification decision-making ability.

[0047] 3. The present invention automatically updates the weight distribution through the adaptive weight method, thereby improving the flexibility of the model to adapt to the needs of different data sets and tasks and solving the inherent problems of traditional multi-view image classification methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the training process of the cross entropy integrated image classification model in the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] like Figure 1 A multi-view image classification method based on deep cross entropy integration is shown, comprising the following steps:

[0051] Step 1: Prepare image data and mark the image to obtain the label data corresponding to the image. The image data and label data constitute a complete data set. The data set is divided into a training set and a test set in a 6:4 ratio.

[0052] More specifically, the acquired data set needs to be preprocessed to obtain an image format that is convenient for model processing. Preprocessing specifically includes: data enhancement, image conversion, and normalization.

[0053] Step 2: Build the network architecture of the cross entropy ensemble image classification model.

[0054] The cross entropy integrated image classification model includes three parts: backbone network, prototype network and integrated classification network. Among them, the backbone network uses the existing ResNet-18 network framework to extract image features, and the prototype network and integrated classification network are network architectures independently designed by the present invention. The working principles of each part in the classification model are as follows:

[0055] (1) The processing process in the backbone network part is as follows:

[0056] The input image is encoded into a feature map through the ResNet-18 network in, Represents the image dimension, H, W, and C represent the height, width, and number of channels of the input image, respectively.

[0057] (2) The processing process in the prototype network part is as follows:

[0058] S1. Map the feature encoded by the backbone network Converted to a vector, it is represented as follows:

[0059]

[0060]

[0061] Among them, Q, K, and V are query, key, and value respectively. is the feature map, W Q , W K , W V is the weight matrix corresponding to query, key, and value.

[0062] S2. Apply the attention mechanism to obtain diverse prototype features. It is expressed as follows:

[0063]

[0064] in, is a scaling factor; Softmax is a normalization function that makes the weighted probability distribution and 1; P is composed of multiple prototype features P n The set formed, P = {P 1 ,P 2 ,…,P N}, n represents the nth prototype, N represents the total number of prototypes, It is the dimension of the prototype.

[0065] S3. For each feature in the feature map Find feature x i,j With prototype feature P n The relative difference between i,j =x i,j -P n .

[0066] S4. Based on the relative difference, the input image is converted into a multi-view representation V = {V 1 ,V 2 …V N}, where N represents the total number of multi-view representations, which is consistent with the total number of prototypes. It is expressed as follows:

[0067]

[0068] Among them, V n is the representation of the nth view, θ n Represents the feature map x i,j With the nth prototype feature P n The correlation coefficient between them, H and W are the height and width of the picture respectively.

[0069] Through the above steps, the feature map Projected onto the prototype feature P, a set of diverse feature representations V is obtained.

[0070] (3) The processing process of the classification network is as follows:

[0071] The multiple views of the input image are represented by V = {V 1 ,V 2 …V N} performs weighted fusion and calculates the integrated cross entropy loss, which is expressed as follows:

[0072]

[0073] Among them, L is the integrated cross entropy loss, M is the total number of pictures, and y i is the true label of the i-th image, N is the number of base classifiers, which is consistent with the number of prototype features; π n is the weight of the nth base classifier, is the transpose of the nth view representation of the ith image, w j is the weight of the fully connected network, and R is the number of classification categories.

[0074] In the present invention, a more robust unified feature expression can be obtained by weighted fusion of multiple view representations.

[0075] Step 3: Combine with Figure 1 , use the training set in step 1 to train the cross entropy ensemble image classification model constructed in step 2. The training process is as follows:

[0076] Step 3.1: For the constructed cross entropy integrated image classification model, constraints need to be added during the training process, as shown below:

[0077]

[0078] Among them, η, ξ, τ are Lagrange multipliers, 1 T π represents the sum of the weights.

[0079] Step 3.2: Back propagate to solve for w j , weight π n .

[0080] For the parameter w in equation (8), j Find the partial derivative to get the w of the nth base classifier j gradient:

[0081]

[0082] For the parameter π in equation (8), n Taking partial derivatives we get:

[0083]

[0084] The obtained π n The result of the integration is obtained by combining it with the cross entropy calculated by the base classifier, and then the parameters are continuously updated through back propagation until the loss function of formula (8) converges and the training is stopped.

[0085] Step 4: Use the trained cross entropy ensemble image classification model to classify images.

[0086] In order to verify the reliability of the cross entropy integrated image classification model constructed by the present invention, the cross entropy integrated image classification model is evaluated and tested as follows:

[0087] The following image classification test and evaluation is performed based on the optimal parameters obtained in the previous step. Feature extraction is performed on the original image, and the single feature is further mapped to a multi-dimensional feature through the attention prototype network, and then the integration loss is calculated for the multi-dimensional feature, where π n The size of is related to the single loss. The smaller the loss, the greater the n The larger the value, the more important the feature is, and vice versa. Our method is compared with the most advanced image classification methods such as SCE, AMC, CCE and RBF on datasets such as FashionMNIST, CIFAR10, CIFAR100, SVHN and TinyImageNet. The experimental results show that the classification accuracy of our method on FashionMNIST, CIFAR10, CIFAR100, SVHN and TinyImageNet datasets is 1.46%, 1.19%, 0.83% and 1.52% higher than the results of all comparison methods on average. The comparison results clearly verify that the classification performance of our method is better than other algorithms.

[0088] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A multi-view image classification method based on deep cross entropy integration, characterized in that: The steps include: Step 1: Obtain images and label them, form a data set based on images and labels, and divide the data set into training set and test set; Step 2: Construct a network architecture of a cross entropy integrated image classification model, wherein the cross entropy integrated image classification model includes three parts: a backbone network, a prototype network, and an integrated classification network; The processing process in the backbone network part is: encoding the input image into a feature map in, Represents the image dimension, H, W, C represent the height, width and number of channels of the input image respectively; In the prototype network part, the feature map χ is projected onto the prototype feature P to obtain a set of diverse feature representations V; In the integrated classification network, multiple views of the input image are represented as V = {V1, V2...V N }Perform weighted fusion and calculate integrated cross entropy loss; Step 3: Use the training set in step 1 to train the cross entropy ensemble image classification model constructed in step 2; Step 4: Use the trained cross entropy ensemble image classification model to classify images.

2. The multi-view image classification method based on deep cross entropy integration according to claim 1, characterized in that: The processing process in the prototype network part is: S1. Map the feature encoded by the backbone network Converted to a vector, it is represented as follows: Among them, Q, K, V are query, key, and value respectively, and W Q , W K , W V are weight matrices corresponding to query, key, and value respectively; S2. Apply the attention mechanism to obtain diverse prototype features, which are expressed as follows: in, is a scaling factor; Softmax is a normalization function that makes the weighted probability distribution and 1; P is composed of multiple prototype features P n The collection formed, n represents the nth prototype, N represents the total number of prototypes, It is the dimension of the archetype; S3. For each feature in the feature map Find feature x i，j With prototype feature P n The relative difference between i,j =x i,j -P n ; S4. Based on the relative difference, the input image is converted into a multi-view representation V = {V1, V2…V N }, where N represents the total number of multi-view representations, which is consistent with the total number of prototypes, and is expressed as follows: Among them, V n is the representation of the nth view, θ n Represents the feature map x i,j With the nth prototype feature P n The correlation coefficient between them, H and W are the height and width of the picture respectively; Through the above steps, the feature map χ is projected onto the prototype feature P to obtain a set of diverse feature representations V.

3. The multi-view image classification method of deep cross entropy integration according to claim 1, characterized in that: The integrated cross entropy loss is expressed as follows: Among them, L is the integrated cross entropy loss, M is the total number of pictures, and y i is the true label of the i-th image, N is the number of base classifiers, which is consistent with the number of prototype features; π n is the weight of the nth base classifier, is the transpose of the nth view representation of the ith image, w j is the weight of the fully connected network, and R is the number of classification categories.

4. The multi-view image classification method of deep cross entropy integration according to claim 1, characterized in that: The training process of step 3 is as follows: Step 3.1: For the constructed cross entropy integrated image classification model, constraints need to be added during the training process, as shown below: Among them, η, ξ, τ are Lagrange multipliers, 1 T π means summing the weights; Step 3.2: Back propagate to solve for w j , weight π n ; For the parameter w in the previous step j Find the partial derivative to get the w of the nth base classifier j gradient: For the parameter π in the previous step n Taking partial derivatives we get: The obtained π n The integrated result is obtained by combining it with the cross entropy calculated by the base classifier, and then the parameters are continuously updated through back propagation until the loss function converges and the training is stopped.

5. The multi-view image classification method of deep cross entropy integration according to claim 1, characterized in that: The acquired data set is preprocessed, and the preprocessing specifically includes: data enhancement, image conversion, and normalization.

6. The multi-view image classification method of deep cross entropy integration according to claim 1, characterized in that: The backbone network uses the ResNet-18 network framework to extract image features to form a feature map.

7. The multi-view image classification method of deep cross entropy integration according to claim 1, characterized in that: The dataset is divided into training set and test set according to the ratio of 6:4.

Images