An image classification method based on multimodal information fusion

Abstract

The present application discloses an image classification method based on multimodal information fusion, which relates to the field of image processing technology. The image classification method uses a multi-scale feature extraction module in a feature extraction network to perform multi-scale feature extraction on the original image of each modality. The multi-feature fusion pyramid is improved on the basis of the FPN framework combined with a hole convolution layer, which can effectively extract the context information of the original image. The multi-feature association module further enhances the multi-scale feature association, so that the feature extraction network can obtain a single-modal feature map with a stronger representation ability of the original image. After multi-modal information fusion is performed on two single-modal feature maps to obtain a fused feature map, in addition to using the fused feature map, the original single-modal feature map is retained and classified together using a classification network. The method can fully mine the image information within and between modalities, thereby effectively improving the accuracy and robustness of image classification.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image classification method based on multimodal information fusion. Background Art

[0002] Image classification is an important research direction in the field of machine vision. This technology extracts image features and uses machine learning models to distinguish different categories. It has a wide range of applications in multiple tasks in various fields.

[0003] Most traditional classical image classification methods use single-modality images to achieve feature extraction and classification. Single-modality images have poor feature representation capabilities and limited information expression, which leads to the unsatisfactory accuracy and precision of image classification based on single-modality images. In order to solve the above problems, the method of image classification after fusing multi-modal image information has gradually been applied to classification tasks in various fields. Since images of different modalities can provide different image information, this method can achieve the complementarity between image information of different modalities, which can improve the accuracy and precision of classification compared with the image classification method using single-modality images. However, the existing image classification methods based on multi-modal information fusion have the problem of insufficient utilization of multi-modal image information and fail to fully explore image information, resulting in the accuracy and precision of image classification based on multi-modal information fusion still difficult to achieve satisfactory results, especially in some scenes with low image quality, it is difficult to guarantee the classification performance. Summary of the invention

[0004] In response to the above problems and technical requirements, this application proposes an image classification method based on multimodal information fusion. The technical solution of this application is as follows:

[0005] An image classification method based on multimodal information fusion, the image classification method comprising:

[0006] Acquire an original image of a first modality and an original image of a second modality of an object to be classified;

[0007] The feature extraction network is used to extract features from the original image of each modality respectively, and a single-modal feature map of the original image of each modality is obtained; the feature extraction network includes a multi-scale feature extraction module, a multi-feature fusion pyramid and a multi-feature association module. The multi-scale feature extraction module is used to extract multi-scale features from the input original image and output K initial feature maps of different scales; the multi-feature fusion pyramid is designed based on the feature pyramid network framework combined with the hole convolution layer. The multi-feature fusion pyramid performs feature fusion on K initial feature maps of different scales and outputs K intermediate feature maps; the multi-feature association module performs multi-scale feature enhancement on the K intermediate feature maps and outputs a single-modal feature map, and the integer parameter K≥2;

[0008] Perform multimodal information fusion on the single-modal feature maps of the original images of the two modalities to obtain a fused feature map;

[0009] The classification network is used to obtain the classification result of the object to be classified based on the fused feature map and the two single-modal feature maps.

[0010] Its further technical solution is that the multi-feature fusion pyramid includes K sequentially connected dilated convolution layers, the convolution kernel sizes from the first dilated convolution layer to the Kth dilated convolution layer along the connection direction increase sequentially, and the output ends of the first K-1 dilated convolution layers along the connection direction are respectively connected to the input end of an upsampling layer;

[0011] The first initial feature map output by the multi-scale feature extraction module passes through a 1*1 convolution layer and is output to the first dilated convolution layer. For any integer parameter k∈[2, K], the kth initial feature map output by the multi-scale feature extraction module passes through a 1*1 convolution layer, and is summed with the output of the k-1th upsampling layer along the connection direction and is output to the kth dilated convolution layer. The sizes of the first to Kth initial feature maps output by the multi-scale feature extraction module increase in sequence.

[0012] The output of each hole convolution layer in the multi-feature fusion pyramid passes through a transition layer and outputs an intermediate feature map to the multi-feature association module.

[0013] A further technical solution is that, in the multi-feature fusion pyramid, a frequency channel attention mechanism layer is connected between the input end of each upsampling layer and the connected dilated convolution layer.

[0014] A further technical solution is that the multi-feature association module includes a fusion layer, a segmentation layer and an enhancement layer, and K intermediate feature maps input into the multi-feature association module are respectively input into the fusion layer and the enhancement layer;

[0015] After the fusion layer adds the corresponding elements of the K input intermediate feature maps, it passes through the global pooling layer, the first fully connected layer, the ReLU activation function, and the second fully connected layer in sequence to obtain the intermediate fusion features and output them to the segmentation layer;

[0016] The segmentation layer performs an average segmentation operation on the intermediate fusion features on the channel to obtain K segmentation feature maps and output them to the enhancement layer;

[0017] The enhancement layer performs multi-scale feature enhancement on the input K intermediate feature maps and K segmentation feature maps and outputs a single-modal feature map.

[0018] Its further technical solution is that in the multi-feature association module, the enhancement layer obtains weight parameters by respectively passing each segmentation feature map obtained by the average segmentation operation through a sigmoid activation function, and uses the K weight parameters to perform weighted addition of the corresponding elements of the input K intermediate feature maps to output a single-modal feature map.

[0019] A further technical solution is to perform multimodal information fusion on the single-modal feature maps of the original images of the two modalities to obtain a fused feature map including:

[0020] The low-rank multimodal fusion network LMF is used to fuse the multimodal information of two single-modal feature maps to obtain a fused feature map.

[0021] A further technical solution is to use a classification network to obtain a classification result of an object to be classified based on a fusion feature map and two single-modal feature maps, including:

[0022] The fused feature map and the two single-modal feature maps are input into the three base classifiers respectively;

[0023] The classification results of the three base classifiers are input into the meta-classifier to obtain the classification result of the object to be classified.

[0024] A further technical solution is that the image classification method further includes:

[0025] Acquire a sample data set, the sample data set comprising a plurality of training sample groups, each training sample group comprising a sample image of a first modality and a sample image of a second modality of a same sample object, and each training sample group comprising a classification annotation result for the sample object;

[0026] Build the network structure of the feature extraction network, and use the sample data set to perform model training based on the network structure of the feature extraction network to obtain a trained feature extraction network;

[0027] The trained feature extraction network is used to extract sample feature maps of two sample images in each training sample group and perform multimodal information fusion to obtain a sample fusion feature map of the training sample group;

[0028] The classification network is trained using the sample feature maps of the two sample images of each training sample group and the obtained sample fusion feature map combined with the classification annotation results.

[0029] A further technical solution is to use the sample data set to perform model training based on the network structure of the feature extraction network, including for each training sample group:

[0030] The two sample images in the training sample group are respectively input into the feature extraction network for feature extraction to obtain the feature maps of the two sample images. and feature map ;

[0031] For feature maps and feature map Subtract the corresponding pixels to get the feature map ;

[0032] The feature map Input the fully connected layer to get the vector , the feature map Input the fully connected layer to get the vector , the feature map Input the fully connected layer to get the vector ;

[0033] According to the vector ,vector and vector Calculate the hybrid loss function by combining the classification and annotation results of the training sample group And according to the mixed loss function Adjust the network parameters of the feature extraction network for model training.

[0034] Its further technical solution is, according to the vector ,vector and vector Calculate the hybrid loss function by combining the classification and annotation results of the training sample group include:

[0035] Using the binary cross entropy loss function according to the vector The loss function is calculated by the classification and labeling results of the training sample group ;

[0036] Using the binary cross entropy loss function according to the vector The loss function is calculated by the classification and labeling results of the training sample group ;

[0037] Using the binary cross entropy loss function according to the vector The loss function is calculated by the classification and labeling results of the training sample group ;

[0038] Calculate the mixed loss function ,in, and are coefficients respectively.

[0039] The beneficial technical effects of this application are:

[0040] The present application discloses an image classification method based on multimodal information fusion, which utilizes a feature extraction network to extract a single-modal feature map of the original image of each modality, and the feature extraction network uses a multi-scale feature extraction module to perform multi-scale feature extraction. The multi-feature fusion pyramid is obtained by combining the improvement of the FPN framework with the dilated convolution, and can extract a larger range of image features while ensuring the image resolution, thereby effectively extracting the context information of the image. The additional multi-feature association module can further enhance the multi-scale feature association. This feature extraction network can better mine the image information within the modality and obtain a single-modal feature map with stronger representation ability. After performing multi-modal information fusion on two single-modal feature maps to obtain a fused feature map, in addition to using the fused feature map, the original single-modal features are also retained. Figure 1 The two methods participate in decision-making fusion classification. This method can fully mine the image information within and between modalities, thereby effectively improving the classification accuracy and robustness.

[0041] In the feature extraction network used in this image classification method, the multi-feature fusion pyramid redistributes the output feature channels of the intermediate feature map after fusion of the void convolution layer through the frequency channel attention mechanism layer FCA before each upsampling process, thereby strengthening the feature information extraction of each layer of the network. Therefore, the network's ability to extract target features can be enhanced through the discrete cosine transform method, and the influence of other irrelevant areas can be weakened to improve the feature extraction ability. This solves the problem of feature loss in the pooling process of the traditional channel attention method and further mines the image information within the modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the processing flow of an image classification method in one embodiment of the present application.

[0043] Figure 2 It is a network structure diagram of a feature extraction network in one embodiment of the present application.

[0044] Figure 3 It is a network structure diagram of a multi-feature association module in a feature extraction network in one embodiment of the present application.

[0045] Figure 4 It is a schematic diagram of calculating the mixed loss function during the training phase in one embodiment of the present application. DETAILED DESCRIPTION

[0046] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0047] This application discloses an image classification method based on multimodal information fusion, which includes the following steps: Figure 1 The schematic diagram shown:

[0048] Step 1: Obtain an original image of a first modality and an original image of a second modality of an object to be classified.

[0049] The first modality and the second modality in this application are used for two different modalities and have no specific meanings. Different modalities refer to different angles for acquiring images or different technical means for acquiring images. Acquiring original images of the object to be classified in two different modalities includes: photographing the object to be classified from two different shooting angles to obtain original images of the two modalities respectively, or photographing the object to be classified using two different image acquisition means to obtain original images of the two modalities respectively. In different application fields and application scenarios, the two modalities for realizing information fusion have different meanings, such as the common use of images acquired from different angles of plants as images of different modalities, and different types of medical images as images of different modalities, and RGB images and depth images as images of different modalities respectively. The specific meaning of the original embodiment of each modality can be determined according to actual needs, and this application does not limit this and does not elaborate on it.

[0050] Step 2: Use the feature extraction network to extract features from the original image of each modality respectively, and obtain the unimodal feature map of the original image of each modality. For the convenience of distinction, the unimodal feature map of the original image of the first modality is defined as F1, and the unimodal feature map of the original image of the second modality is defined as F2.

[0051] Please refer to Figure 2 As shown in the network structure diagram, the feature extraction network in this application includes a multi-scale feature extraction module, a multi-feature fusion pyramid and a multi-feature association module, wherein:

[0052] (1) The multi-scale feature extraction module is used to extract multi-scale features from the input original image and output K initial feature maps of different scales. Please refer to Figure 1 The multi-scale feature extraction module uses convolutions of different sizes to perform convolution processing on the original image in sequence to output K initial feature maps with successively reduced scales. In this application, the first initial feature map to the Kth initial feature map are respectively denoted as ~ And the first initial feature map To the Kth initial feature map The size of increases successively. The integer parameter K≥2. The multi-scale feature extraction module is a module that is available in existing multi-modal information fusion, and can be implemented based on ResNet50, which is not described in detail in this application.

[0053] (2) Multi-feature fusion pyramid for K initial feature maps of different scales ~ Perform feature fusion and output K intermediate feature maps ~ .

[0054] The multi-feature fusion pyramid is improved based on the Feature Pyramid Networks framework. The multi-feature fusion pyramid of this application is designed based on the Feature Pyramid Networks (FPN) framework combined with the hole convolution layer. Please refer to Figure 2 The network structure diagram shown in the figure shows that the multi-feature fusion pyramid includes K hole convolution layers connected in sequence. The first hole convolution layer along the connection direction To the Kth dilated convolutional layer The sizes of the convolution kernels increase successively, and the output ends of the first K-1 hole convolution layers along the connection direction are connected to the input end of an upsampling layer respectively.

[0055] The first initial feature map output by the multi-scale feature extraction module After passing through the 1*1 convolution layer, the output is sent to the first hole convolution layer For any integer parameter k∈[2, K], the kth initial feature map output by the multi-scale feature extraction module is After passing through the 1*1 convolutional layer, the corresponding elements are summed with the output of the k-1th upsampling layer along the connection direction and then output to the kth dilated convolutional layer. The k-1th upsampling layer along the connection direction is connected between the k-1th dilated convolutional layer and the kth dilated convolutional layer.

[0056] The output of each hole convolution layer in the multi-feature fusion pyramid passes through a transition layer and outputs an intermediate feature map to the multi-feature association module. The first hole convolution layer To the Kth dilated convolutional layer Output intermediate feature maps separately ~ For multi-feature fusion pyramid, each transition layer consists of 3*3 convolution units and average pooling units connected in sequence.

[0057] Compared with the traditional feature pyramid network FPN, the multi-feature fusion pyramid of the present application adds multiple hole convolution layers of different sizes. By adjusting the expansion rate, the hole convolution layer can obtain a larger receptive field without increasing the number of parameters. This method can extract a larger range of image features while ensuring the image resolution, thereby effectively extracting the contextual information of the image, and thus more effectively mining the image information within the modality.

[0058] Furthermore, in another embodiment, in the multi-feature fusion pyramid, a frequency channel attention mechanism layer FCA is also connected between the input end of each upsampling layer and the connected hole convolution layer. The frequency channel attention mechanism layer FCA reallocates the output feature channels of the intermediate feature map after fusion of the hole convolution layer before each upsampling process using the upsampling layer, thereby strengthening the feature information extraction of each layer of the network, thereby enhancing the network's ability to extract target features through the discrete cosine transform method, reducing the influence of other irrelevant areas to improve the feature extraction capability, and solving the problem of feature loss in the pooling process of the traditional channel attention method, thereby more effectively mining the image information within the modality.

[0059] (3) Traditionally, a multi-scale feature extraction module is used in combination with a feature pyramid network (FPN) to directly extract features of the original image of a single modality. However, this application also adds a multi-feature association module to the feature extraction network. The multi-feature association module performs K intermediate feature maps. ~ Perform multi-scale feature enhancement and output a single-modal feature map. The addition of a multi-feature association module can further enhance the multi-scale feature association, thereby obtaining a single-modal feature map with stronger representation capabilities.

[0060] Please refer to Figure 3 As shown in the network structure diagram, the multi-feature association module is improved based on the channel attention mechanism network SENet. The multi-feature association module includes a fusion layer, a segmentation layer and an enhancement layer. The K intermediate feature maps input into the multi-feature association module are input into the fusion layer and the enhancement layer respectively.

[0061] After the fusion layer adds the corresponding elements of the K intermediate feature maps of the input, it passes through the global pooling layer, the first fully connected layer, the ReLU activation function and the second fully connected layer in sequence to obtain the intermediate fusion features and output them to the segmentation layer.

[0062] The segmentation layer performs an average segmentation operation on the intermediate fusion features on the channel to obtain K segmentation feature maps and output them to the enhancement layer.

[0063] The enhancement layer performs multi-scale feature enhancement on the input K intermediate feature maps and K segmentation feature maps and outputs a single-modal feature map, including: each segmentation feature map obtained by the average segmentation operation is subjected to a sigmoid activation function to obtain a weight parameter, and the K weight parameters are used to perform weighted addition of the corresponding elements of the input K intermediate feature maps and output a single-modal feature map. That is, for any integer parameter i∈[1,K], the weight parameter obtained by the sigmoid activation function of the i-th segmentation feature map obtained by segmentation is equal to the corresponding i-th intermediate feature map. Perform element-by-element multiplication to obtain the i-th weighted result, and add the corresponding elements of all K weighted results to obtain a unimodal feature map.

[0064] The multi-feature association module adds the intermediate feature maps of different scales to obtain multi-scale fusion information, and then uses compression, excitation, and reweighting operations to obtain the weight parameters of different channels of features of different scales. Finally, channel fusion is performed through weighted sum, which further enhances the multi-scale feature association and improves the representation ability of the output single-modal feature map.

[0065] Step 3, multimodal information fusion is performed on the unimodal feature maps of the original images of the two modalities to obtain a fused feature map F. In one embodiment, a low-rank multimodal fusion network LMF is used to perform multimodal information fusion on the two unimodal feature maps to obtain a fused feature map F.

[0066] Step 4: Use the classification network to obtain the classification result of the object to be classified based on the fused feature map F and the two unimodal feature maps F1 and F2. In this step, in addition to using the fused feature map F, the original unimodal feature map F1 and unimodal feature map F2 are also retained for decision fusion, which can further improve the classification accuracy and robustness.

[0067] In one embodiment, the classification network includes three base classifiers and one meta classifier. In the classification network, the fusion feature map F, the unimodal feature map F1 and the unimodal feature map F2 are respectively input into the three base classifiers. Figure 1 They are represented by base classifier 1, base classifier 2, and base classifier 3. Then the classification results of the three base classifiers are input into the meta classifier to obtain the classification results of the object to be classified. Any base classifier can use any one of XGBoost, LightGBM, and Random, and the three base classifiers can use the same or different classifiers. The meta classifier uses Nativebaye.

[0068] Before using the image classification method of the present application, it is necessary to train the feature extraction network and the classification network, so it also includes a model training method, including the following contents:

[0069] 1. Obtain a sample data set. The sample data set includes multiple training sample groups, each training sample group includes sample images of the first modality and sample images of the second modality of the same sample object, and each training sample group includes classification annotation results for the sample object. The sample object belongs to the same object as the object to be classified, and the meanings of the first modality and the second modality are the same as in step 1. The method for obtaining the training sample group is the same as the method for obtaining the original images of the object to be classified in two modalities in step 1, and will not be repeated here.

[0070] 2. Build the network structure of the feature extraction network, and use the sample data set to train the model based on the network structure of the feature extraction network to obtain the trained feature extraction network, including inputting the two sample images in each training sample group into the feature extraction network and combining the classification and annotation results to obtain the mixed loss function , and then according to the mixed loss function Adjust the network parameters of the feature extraction network to train the model. For each training sample group, combine Figure 4 :

[0071] (1) Input the two sample images in the training sample group into the feature extraction network for feature extraction, and obtain the feature maps of the two sample images respectively. and feature map .

[0072] (2) Feature map and feature map Subtract the corresponding pixels to get the feature map .

[0073] (3) Feature map Input the fully connected layer to get the vector , the feature map Input the fully connected layer to get the vector , the feature map Input the fully connected layer to get the vector .

[0074] (4) According to the vector ,vector and vector Calculate the hybrid loss function by combining the classification and annotation results of the training sample group And according to the mixed loss function Adjust the network parameters of the feature extraction network to train the model. This includes: using the binary cross entropy loss function to calculate the vector The loss function is calculated by the classification and labeling results of the training sample group. . Using the binary cross entropy loss function according to the vector The loss function is calculated by the classification and labeling results of the training sample group. . Using the binary cross entropy loss function according to the vector The loss function is calculated by the classification and labeling results of the training sample group. Then the mixed loss function is calculated ,in, and are coefficients respectively.

[0075] 3. After the feature extraction network is trained, the sample feature maps of the two sample images in each training sample group are extracted using the trained feature extraction network and multimodal information fusion is performed to obtain the sample fusion feature map of the training sample group. This step is the same as the above step 3, that is, the sample feature maps of the two sample images in the same training sample group are fused using the low-rank multimodal fusion network LMF to obtain the corresponding sample fusion feature map.

[0076] 4. Use the sample feature maps of the two sample images of each training sample group and the obtained sample fusion feature map combined with the classification annotation results to train the classification network.

[0077] After training the feature extraction network and classification network, Figure 1 The process can then be combined and further tested using the test set to optimize model performance.

[0078] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. An image classification method based on multimodal information fusion, characterized in that: The image classification method comprises: Acquire an original image of a first modality and an original image of a second modality of an object to be classified; The feature extraction network is used to extract features from the original image of each modality respectively, and a single-modal feature map of the original image of each modality is obtained; the feature extraction network includes a multi-scale feature extraction module, a multi-feature fusion pyramid and a multi-feature association module, the multi-scale feature extraction module is used to perform multi-scale feature extraction on the input original image and output K initial feature maps of different scales; the multi-feature fusion pyramid is designed based on the feature pyramid network framework combined with the hole convolution layer, the multi-feature fusion pyramid performs feature fusion on the K initial feature maps of different scales and outputs K intermediate feature maps; the multi-feature association module performs multi-scale feature enhancement on the K intermediate feature maps and outputs the single-modal feature map, and the integer parameter K≥2; Perform multimodal information fusion on the single-modal feature maps of the original images of the two modalities to obtain a fused feature map; Obtaining a classification result of the object to be classified based on the fused feature map and the two single-modal feature maps using a classification network; The multi-feature fusion pyramid includes K hole convolution layers connected in sequence, and the convolution kernel sizes from the first hole convolution layer to the Kth hole convolution layer along the connection direction increase in sequence, and the output ends of the first K-1 hole convolution layers along the connection direction are respectively connected to the input end of an upsampling layer; the first initial feature map output by the multi-scale feature extraction module is output to the first hole convolution layer after passing through a 1*1 convolution layer; for any integer parameter k∈[2, K], the kth initial feature map output by the multi-scale feature extraction module passes through a 1*1 convolution layer, and is output to the kth hole convolution layer after the corresponding elements are summed with the output of the k-1th upsampling layer along the connection direction; the sizes of the first initial feature map to the Kth initial feature map output by the multi-scale feature extraction module increase in sequence; the output ends of each hole convolution layer in the multi-feature fusion pyramid respectively pass through a transition layer and output an intermediate feature map to the multi-feature association module; The multi-feature association module includes a fusion layer, a segmentation layer and an enhancement layer. The K intermediate feature maps input into the multi-feature association module are respectively input into the fusion layer and the enhancement layer; the fusion layer adds the corresponding elements of the K intermediate feature maps input, and then passes through the global pooling layer, the first fully connected layer, the ReLU activation function and the second fully connected layer in sequence to obtain the intermediate fusion features and output them to the segmentation layer; the segmentation layer performs an average segmentation operation on the intermediate fusion features on the channel to obtain K segmentation feature maps and outputs them to the enhancement layer; the enhancement layer performs multi-scale feature enhancement on the input K intermediate feature maps and K segmentation feature maps and then outputs the single modal feature map.

2. The image classification method according to claim 1, characterized in that: In the multi-feature fusion pyramid, a frequency channel attention mechanism layer is also connected between the input end of each upsampling layer and the connected dilated convolution layer.

3. The image classification method according to claim 1, characterized in that: In the multi-feature association module, the enhancement layer obtains weight parameters by respectively applying a sigmoid activation function to each segmentation feature map obtained by the average segmentation operation, and uses the K weight parameters to perform weighted addition of corresponding elements of the input K intermediate feature maps to output the single-modal feature map.

4. The image classification method according to claim 1, characterized in that: The step of fusing the single-modal feature maps of the original images of the two modalities by multi-modal information to obtain a fused feature map comprises: The low-rank multimodal fusion network LMF is used to perform multimodal information fusion on two single-modal feature maps to obtain the fused feature map.

5. The image classification method according to claim 1, characterized in that: The method of obtaining the classification result of the object to be classified based on the fusion feature map and the two single-modal feature maps by using the classification network includes: Input the fused feature map and the two single-modal feature maps into three base classifiers respectively; The classification results of the three base classifiers are input into the meta classifier to obtain the classification result of the object to be classified.

6. The image classification method according to claim 1, characterized in that: The image classification method further comprises: Acquire a sample data set, wherein the sample data set includes a plurality of training sample groups, each training sample group includes a sample image of a first modality and a sample image of a second modality of a same sample object, and each training sample group includes a classification annotation result for the sample object; Building a network structure of a feature extraction network, and using the sample data set to perform model training based on the network structure of the feature extraction network to obtain the trained feature extraction network; Extracting sample feature maps of two sample images in each training sample group using the trained feature extraction network and performing multimodal information fusion to obtain a sample fusion feature map of the training sample group; The classification network is trained using the sample feature maps of the two sample images of each training sample group and the obtained sample fusion feature map combined with the classification annotation results.

7. The image classification method according to claim 6, characterized in that: The performing model training based on the network structure of the feature extraction network using the sample data set comprises for each training sample group: The two sample images in the training sample group are respectively input into the feature extraction network for feature extraction to obtain feature maps of the two sample images. and feature map ; For feature maps and feature map Subtract the corresponding pixels to get the feature map ; The feature map Input the fully connected layer to get the vector , the feature map Input the fully connected layer to get the vector , the feature map Input the fully connected layer to get the vector ; According to the vector ,vector and vector Calculate the mixed loss function based on the classification and annotation results of the training sample group And according to the mixed loss function The network parameters of the feature extraction network are adjusted to perform model training.

8. The image classification method according to claim 7, characterized in that: The vector ,vector and vector Calculate the mixed loss function based on the classification and annotation results of the training sample group include: Using the binary cross entropy loss function according to the vector The loss function is calculated by the classification and labeling results of the training sample group. ; Using the binary cross entropy loss function according to the vector The loss function is calculated by the classification and labeling results of the training sample group. ; Using the binary cross entropy loss function according to the vector The loss function is calculated by the classification and labeling results of the training sample group. ; Calculate the mixed loss function ,in, and are coefficients respectively.

Images