A multi-view image classification method based on deep cross-entropy ensemble

Abstract

This invention discloses a deep cross-entropy ensemble multi-view image classification method. It constructs a dataset based on images, divided into training and testing sets. The method builds a network architecture for a cross-entropy ensemble image classification model, comprising a backbone network, a prototype network, and an ensemble classification network. The backbone network encodes the input image into a feature map. The prototype network projects the feature map onto prototype features P to obtain a set of diverse feature representations V. The ensemble classification network represents multiple views of the input image as V = {V1, V2…V...}. N The method involves weighted fusion and calculation of ensemble cross-entropy loss; training the cross-entropy ensemble image classification model using the training set; and using the trained cross-entropy ensemble image classification model for image classification. This invention adapts to the needs of different datasets and tasks, and solves the inherent problems of traditional multi-view image classification methods.

Description

Technical Field

[0001] This invention relates to image classification technology, specifically a multi-view image classification method based on deep cross-entropy ensemble. Background Technology

[0002] Currently, deep learning methods, especially Convolutional Neural Networks (CNNs), have made significant progress in image classification. However, traditional single-view image classification methods often face limitations when processing complex images. Information from a single viewpoint is often insufficient to fully describe the image's features, thus affecting classification performance. To overcome this problem, multi-view image classification methods have emerged, enriching feature representations by constructing multiple views of the data, thereby improving the robustness and generalization ability of classification.

[0003] Existing multi-view image classification methods mainly fall into two categories: feature fusion and decision fusion. Feature fusion methods combine features from different viewpoints to construct a high-dimensional feature space for classifier use. However, this method can easily lead to excessively high feature dimensionality, increasing computational complexity and the risk of overfitting. Decision fusion methods, on the other hand, perform weighted averaging or voting on the outputs of multiple independent classifiers to obtain the final classification result. While this alleviates the feature dimensionality problem to some extent, the correlation between classifiers may result in unsatisfactory fusion performance.

[0004] Furthermore, traditional multi-view image classification methods often employ fixed strategies in feature extraction and classifier design, lacking flexibility and making it difficult to adapt to the needs of different datasets and tasks. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application proposes a multi-view image classification method using deep cross-entropy ensemble. By designing an attention prototype network, different prototype representations are learned. Input features are projected onto these prototypes and encoded using attention prototype vectors to generate different feature sets. Next, a classifier is trained for each feature set, and these classifiers are combined into a non-linear classifier. Finally, the classification result is optimized by calculating the ensemble loss. This method can effectively capture features from different viewpoints of an image and perform feature fusion, thereby improving the performance of image classification.

[0006] The technical solution adopted in this invention is as follows:

[0007] A multi-view image classification method based on deep cross-entropy ensemble includes the following steps:

[0008] Step 1: Acquire and label the images, construct a dataset based on the images and labels, and divide the dataset into a training set and a test set;

[0009] Step 2: Construct the network architecture of the cross-entropy ensemble image classification model, which includes three parts: a backbone network, a prototype network, and an ensemble classification network.

[0010] The processing procedure in the backbone network is as follows: the input image is encoded into a feature map. in, H, W, and C represent the image dimensions, respectively, representing the height, width, and number of channels of the input image.

[0011] Feature mapping will be implemented in the prototype network section. Projecting onto the prototype feature P yields a diverse set of feature representations V;

[0012] In the ensemble classification network, multiple views of the input image are represented as V = {V1, V2, ..., V...} N Perform weighted fusion and calculate the ensemble cross-entropy loss;

[0013] Step 3: Train the cross-entropy ensemble image classification model constructed in Step 2 using the training set from Step 1;

[0014] Step 4: Perform image classification using the trained cross-entropy ensemble image classification model.

[0015] Furthermore, the dataset is divided into training and testing sets in a 6:4 ratio.

[0016] Furthermore, the acquired dataset is preprocessed, including data augmentation, image transformation, and normalization.

[0017] Furthermore, the backbone network uses the ResNet-18 network framework to extract image features to form a feature map.

[0018] Furthermore, the processing procedure in the prototype network section is as follows:

[0019] S1. Map the features encoded by the backbone network. Converted to a vector, it is represented as follows:

[0020]

[0021] Where Q, K, and V are the query, key, and value, respectively, and W... Q W K W V These are weight matrices corresponding to the query, key, and value, respectively.

[0022] S2. Apply an attention mechanism to obtain diverse prototype features, as shown below:

[0023]

[0024] in, It is the scaling factor; Softmax is the normalization function that makes the sum of the weight probability distributions equal to 1; P is composed of multiple prototype features P n The resulting set, P = {P1, P2, ..., P...} N}, n represents the nth prototype, and N represents the total number of prototypes. It is the dimension of the prototype;

[0025] S3. For each feature in the feature map Find feature x i,j With prototype feature P n The relative difference d between i,j =x i,j -P n ;

[0026] S4. Based on the relative difference, transform the input image into a multi-view representation V = {V1, V2, ..., V...} N}, where N represents the total number of multi-view representations, consistent with the total number of prototypes, as shown below:

[0027]

[0028]

[0029] Among them, V n It is the representation of the nth view, θ n Represents the feature map x i,j With the nth prototype feature P n The correlation coefficient between them, where H and W are the height and width of the image, respectively;

[0030] The above steps will be used to map features. Projecting onto the prototype feature P yields a diverse set of feature representations V.

[0031] Furthermore, the integrated cross-entropy loss is expressed as follows:

[0032]

[0033] Where L is the ensemble cross-entropy loss, M is the total number of images, and y i π is the true label of the i-th image, N is the number of base classifiers, which is consistent with the number of prototype features; n It is the weight of the nth base classifier. w is the transpose of the nth view representation of the i-th image. j R represents the weights of the fully connected network, and R is the number of classification categories.

[0034] Furthermore, the training process in step 3 is as follows:

[0035] Step 3.1: For the constructed cross-entropy ensemble image classification model, constraints need to be added during the training process, as follows:

[0036]

[0037] Where η, ξ, τ are Lagrange multipliers, 1 T π represents the summation of the weights;

[0038] Step 3.2: Backpropagation to solve for w j Weight π n ;

[0039] Regarding the parameter w in the previous step j Taking the partial derivative yields the w of the nth base classifier. j gradient:

[0040]

[0041] For the parameter π in the previous step n Taking the partial derivative, we get:

[0042]

[0043] The obtained π n The ensemble result is obtained by combining the cross-entropy calculated from the base classifier and then continuously updating the parameters through backpropagation until the loss function converges, at which point training stops.

[0044] The beneficial effects of this invention are:

[0045] 1. This invention designs an attention-based prototype network, which obtains different prototypes by focusing on different local features, thereby transforming a single view into a multi-view representation. This design can fully exploit the latent features of an image and construct semantically rich multi-view representations.

[0046] 2. In the classifier design, the present invention adopts an ensemble classification approach, which unifies the individual cross-entropy losses of multiple data views into the overall ensemble space through weighted fusion, thereby improving the classification decision-making ability.

[0047] 3. This invention improves model flexibility by automatically updating the weight allocation through an adaptive weighting method, thereby adapting to the needs of different datasets and tasks and solving the inherent problems of traditional multi-view image classification methods. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the training process of the cross-entropy ensemble image classification model in this invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0050] like Figure 1 The multi-view image classification method based on deep cross-entropy ensemble, as shown, includes the following steps:

[0051] Step 1: Prepare image data and label the images to obtain corresponding label data. The image data and label data constitute a complete dataset. Divide the dataset into a training set and a test set in a 6:4 ratio.

[0052] More specifically, the acquired dataset needs to be preprocessed to obtain an image format suitable for model processing. Preprocessing includes data augmentation, image transformation, and normalization.

[0053] Step 2: Construct the network architecture of the cross-entropy ensemble image classification model.

[0054] The cross-entropy ensemble image classification model comprises three parts: a backbone network, a prototype network, and an ensemble classification network. The backbone network uses the existing ResNet-18 network framework to extract image features, while the prototype network and the ensemble classification network are network architectures independently designed in this invention. The working principles of each part of the classification model are as follows:

[0055] (1) The processing procedure in the backbone network part is as follows:

[0056] The input image is encoded into a feature map using a ResNet-18 network. in, H, W, and C represent the image dimensions, respectively, and represent the height, width, and number of channels of the input image.

[0057] (2) The processing procedure in the prototype network part is as follows:

[0058] S1. Map the features encoded by the backbone network. Converted to a vector, it is represented as follows:

[0059]

[0060]

[0061] Where Q, K, and V represent the query, key, and value, respectively. It is a feature map, W Q W K W V It is a weight matrix corresponding to the query, key, and value.

[0062] S2. Apply attention mechanisms to obtain diverse prototype features. This is represented as follows:

[0063]

[0064] in, It is the scaling factor; Softmax is the normalization function that makes the sum of the weight probability distributions equal to 1; P is composed of multiple prototype features P n The resulting set, P = {P1, P2, ..., P...} N}, n represents the nth prototype, and N represents the total number of prototypes. It is the dimension of the prototype.

[0065] S3. For each feature in the feature map Find feature x i,j With prototype feature P n The relative difference d between i,j =x i,j -P n .

[0066] S4. Based on the relative difference, transform the input image into a multi-view representation V = {V1, V2, ..., V...} N}, where N represents the total number of multi-view representations, consistent with the total number of prototypes. Represented as follows:

[0067]

[0068] Among them, V n It is the representation of the nth view, θ n Represents the feature map x i,j With the nth prototype feature P n The correlation coefficients between them, where H and W are the height and width of the image, respectively.

[0069] The above steps will be used to map features. Projecting onto the prototype feature P yields a diverse set of feature representations V.

[0070] (3) The processing procedure of the classification network is as follows:

[0071] Represent the multiple views of the input image as V = {V1, V2, ..., V...} N We perform weighted fusion and calculate the ensemble cross-entropy loss, as shown below:

[0072]

[0073] Where L is the ensemble cross-entropy loss, M is the total number of images, and y i π is the true label of the i-th image, N is the number of base classifiers, which is consistent with the number of prototype features; n It is the weight of the nth base classifier. w is the transpose of the nth view representation of the i-th image. j R represents the weights of the fully connected network, and R is the number of classification categories.

[0074] In this invention, a more robust unified feature representation can be obtained by weighted fusion of multiple view representations.

[0075] Step 3: Combine with the appendix Figure 1 The cross-entropy ensemble image classification model constructed in step 2 is trained using the training set from step 1. The training process is as follows:

[0076] Step 3.1: For the constructed cross-entropy ensemble image classification model, constraints need to be added during the training process, as follows:

[0077]

[0078] Where η, ξ, τ are Lagrange multipliers, 1 T π represents the summation of the weights.

[0079] Step 3.2: Backpropagation to solve for w j Weight π n .

[0080] For the parameter w in equation (8) j Taking the partial derivative yields the w of the nth base classifier. j gradient:

[0081]

[0082] For the parameter π in equation (8) n Taking the partial derivative, we get:

[0083]

[0084] The obtained π n The ensemble result is obtained by combining the cross-entropy calculated by the base classifier and then continuously updating the parameters through backpropagation until the loss function of equation (8) converges, at which point training stops.

[0085] Step 4: Perform image classification using the trained cross-entropy ensemble image classification model.

[0086] To verify the reliability of the cross-entropy ensemble image classification model constructed in this invention, the cross-entropy ensemble image classification model was evaluated and tested, as follows:

[0087] Based on the optimal parameters obtained in the previous step, the following image classification test and evaluation are performed. Feature extraction is performed on the original image, and a prototype attention network is used to further map single features to multi-dimensional features. Then, the ensemble loss is calculated on the multi-dimensional features, where π... n The magnitude of π is related to the individual loss; the smaller the loss, the greater the π. n The larger the value, the more important the feature, and vice versa. Our method is compared with state-of-the-art image classification methods such as SCE, AMC, CCE, and RBF on datasets including FashionMNIST, CIFAR10, CIFAR100, SVHN, and TinyImageNet. Experimental results show that our method achieves an average improvement of 1.46%, 1.19%, 0.83%, and 1.52% in classification accuracy on the FashionMNIST, CIFAR10, CIFAR100, SVHN, and TinyImageNet datasets, respectively. The comparison results clearly validate that our method outperforms other algorithms in classification performance.

[0088] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A multi-view image classification method based on deep cross-entropy ensemble, characterized in that, Includes the following steps: Step 1: Acquire and label the images, construct a dataset based on the images and labels, and divide the dataset into a training set and a test set; Step 2: Construct the network architecture of the cross-entropy ensemble image classification model, which includes three parts: a backbone network, a prototype network, and an ensemble classification network. The processing procedure in the backbone network is as follows: the input image is encoded into a feature map. , ,in, H, W, and C represent the image dimensions, respectively, representing the height, width, and number of channels of the input image. Feature mapping will be implemented in the prototype network section. Projected onto prototype features Above, a diverse set of feature representations is obtained. The prototype network obtains different prototypes by focusing on different local features, transforming a single view into a multi-view representation. The processing procedure in the prototype network part is as follows: S1. Map the features encoded by the backbone network. Converted to a vector, it is represented as follows: ; Where Q, K, and Z are the query, key, and value, respectively. , , These are weight matrices corresponding to the query, key, and value, respectively. S2. Apply an attention mechanism to obtain diverse prototype features, as shown below: ; in, It is the scaling factor; Softmax is the normalization function that makes the sum of the weight probability distributions equal to 1; P is composed of multiple prototype features. The formed set , , where n represents the nth prototype, and N represents the total number of prototypes. It is the dimension of the prototype; S3. For each feature in the feature map Find the features With prototype features relative difference between ; S4. Based on the relative difference, transform the input image into a multi-view representation. Since the total number of multi-view representations is the same as the total number of prototypes, the total number of multi-view representations is also N, as shown below: ； in, It is the representation of the nth view. Represents feature mapping With the nth prototype feature The correlation coefficient between them, where H and W are the height and width of the image, respectively; The above steps will be used to map features. Projected onto prototype features This allows us to obtain a diverse set of feature representations. ; In the ensemble classification network, multiple views of the input image are represented. Weighted fusion is performed, and the ensemble cross-entropy loss is calculated. An ensemble classification approach is adopted, and the weighted fusion unifies the individual cross-entropy losses of multiple data views into the overall ensemble space. The ensemble cross-entropy loss is expressed as follows: ； in, To integrate the cross-entropy loss, M is the total number of images. is the true label of the i-th image, and N is the number of base classifiers, which is consistent with the number of prototype features; It is the weight of the nth base classifier. It is the transpose of the nth view representation of the i-th image. These are the weights of the fully connected network, where R is the number of classification categories. Step 3: Train the cross-entropy ensemble image classification model constructed in Step 2 using the training set from Step 1; Step 4: Perform image classification using the trained cross-entropy ensemble image classification model.

2. The multi-view image classification method based on deep cross-entropy ensemble according to claim 1, characterized in that, The training process in step 3 is as follows: Step 3.1: For the constructed cross-entropy ensemble image classification model, constraints need to be added during the training process, as follows: ; in, , , For Lagrange multipliers, This indicates that the weights are summed. Step 3.2: Solve via backpropagation Weight ; For the parameters in the previous step The partial derivative is used to obtain the nth base classifier. gradient: ； For the parameters in the previous step Taking the partial derivative, we get: ； Received The ensemble result is obtained by combining the cross-entropy calculated from the base classifier and then continuously updating the parameters through backpropagation until the loss function converges, at which point training stops.

3. The multi-view image classification method based on deep cross-entropy ensemble according to claim 1, characterized in that, The acquired dataset is preprocessed, including data augmentation, image transformation, and normalization.

4. The multi-view image classification method based on deep cross-entropy ensemble according to claim 1, characterized in that, The backbone network uses the ResNet-18 network framework to extract image features to form a feature map.

5. The multi-view image classification method based on deep cross-entropy ensemble according to claim 1, characterized in that, The dataset was divided into training and test sets in a 6:4 ratio.

Images