Image classification method based on dual-branch mutual fusion
Patent Information
- Application Number
- CN202210511831.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-11
- Publication Date
- 2025-04-29
- Estimated Expiration
- 2042-05-11
Smart Images
Figure CN114898149B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image classification and processing, and in particular relates to an image classification method that integrates two branches. Background Art
[0002] Satellite remote sensing technology can obtain multispectral (MS) remote sensing images with rich spectral information and panchromatic (PAN) remote sensing images with high spatial resolution. There is very strong information complementarity between multispectral (MS) remote sensing images and panchromatic (PAN) remote sensing images. How to effectively extract the common and unique information of multispectral remote sensing images and panchromatic remote sensing images has become a topic of great concern.
[0003] In the related technologies, there are traditional image fusion methods, multi-scale image fusion methods, and model-based image fusion methods for the fusion of multispectral remote sensing images and panchromatic remote sensing images; however, they have disadvantages such as single fusion effect, spectral distortion rate, large amount of calculation, and information redundancy.
[0004] Therefore, there is an urgent need for a classification method for multispectral remote sensing images and panchromatic remote sensing images to overcome the above shortcomings. Summary of the invention
[0005] In order to solve the above problems existing in the prior art, the present invention provides a dual-branch mutual fusion image classification method. The technical problem to be solved by the present invention is achieved by the following technical solutions:
[0006] In a first aspect, the present application provides a dual-branch mutual fusion image classification method, comprising:
[0007] Acquire an original data set, and read the first image and the second image, as well as the corresponding class map annotated with partial regions from the original data set;
[0008] Registering the first image and the second image, and fusing the registered first image and the second image to obtain a first image tensor and a second image tensor;
[0009] Preprocessing the first image tensor and the second image tensor, and selecting a training set and a test set based on the preprocessed first image tensor and the second image tensor;
[0010] Train the preset classification model based on the training set and construct the loss function of the classification model;
[0011] Classify the training set based on the classification model, compare the classification results with the class map annotated in some areas to solve the loss, and backpropagate the loss to optimize the classification model;
[0012] The pixels in the test set are classified based on the classification model to obtain the categories of the pixels in the test set.
[0013] Beneficial effects of the present invention:
[0014] The present invention provides a dual-branch mutual fusion image classification method, which first obtains an original data set, and reads a first image and a second image, as well as a corresponding partial area labeled class map from the original data set; aligns the first image and the second image, and fuses the aligned first image and the second image to obtain a first image tensor and a second image tensor; preprocesses the first image tensor and the second image tensor, and selects a training set and a test set based on the preprocessed first image tensor and the second image tensor; trains a preset classification model based on the training set, and constructs a loss function of the classification model; it can be understood that the preset classification model is trained using the training set, and during the training process, the loss function is used to optimize the robustness of the classification model; classifies the training set based on the classification model, compares the classification result with the partial area labeled class map to solve the loss, and back-propagates the loss to optimize the classification model; classifies the pixels in the test set based on the classification model, and obtains the category of the pixels in the test set; through the above method, the classification quality of the classification model can be improved.
[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is a flow chart of the image classification method with branch mutual fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0017] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.
[0018] See also Figure 1 , Figure 1 : is a flow chart of a dual-branch mutual fusion image classification method provided by an embodiment of the present invention. The dual-branch mutual fusion image classification method provided by the present application includes:
[0019] S101, obtaining an original data set, and reading a first image and a second image, as well as a corresponding class map with partial region annotations, from the original data set;
[0020] S102, registering the first image and the second image, and fusing the registered first image and the second image to obtain a first image tensor and a second image tensor;
[0021] S103, preprocessing the first image tensor and the second image tensor, and selecting a training set and a test set based on the preprocessed first image tensor and the second image tensor;
[0022] S104, training a preset classification model based on the training set, and constructing a loss function of the classification model;
[0023] S105, classifying the training set based on the classification model, comparing the classification result with the class map annotated in some areas to solve the loss, and back-propagating the loss to optimize the classification model;
[0024] S106: Classify the pixels in the test set based on the classification model to obtain categories of the pixels in the classified test set.
[0025] For more details, please refer to Figure 1 As shown, an image classification method provided in this embodiment is used for multispectral remote sensing images and panchromatic remote sensing images acquired by remote sensing satellite technology. Since there is very strong information complementarity between multispectral remote sensing images and panchromatic remote sensing images, it is not easy to obtain the common information and unique information of multispectral remote sensing images and panchromatic remote sensing images; and the present application adopts a dual-branch mutual fusion method of an adaptive channel to classify images, which can effectively improve the image classification quality.
[0026] In the related technologies, the fusion of multispectral remote sensing images and panchromatic remote sensing images based on the IHS transformation method has problems such as single fusion results and spectral distortion; the fusion of multispectral remote sensing images and panchromatic remote sensing images based on deep neural networks has problems such as large amount of calculation and information redundancy. Since multispectral remote sensing images and panchromatic remote sensing images have large modal differences and redundant information, if neural networks are used directly for fusion, the pressure of fusion will increase and unnecessary amount of calculation will also increase.
[0027] In view of this, in this embodiment, the multispectral remote sensing image and the panchromatic remote sensing image are classified by the following steps, specifically:
[0028] S101. Obtain an original data set, and read a first image and a second image, as well as a ground truth map annotated with corresponding partial areas, from the original data set, wherein the first image may be a multispectral remote sensing image, and the second image may be a panchromatic remote sensing image; the ground truth map may be annotated with multiple classes; optionally, the classes may be rockery, lakes, green plants or buildings, etc., which is not limited in this embodiment, and may be classified according to the environment in different images.
[0029] S102. Align the first image and the second image, and fuse the aligned first image and the second image to obtain a first image tensor and a second image tensor; optionally, the first image tensor includes multiple processed first images, and the second image tensor includes multiple processed second images.
[0030] S103, preprocessing the first image tensor and the second image tensor, and selecting a training set and a test set based on the preprocessed first image tensor and the second image tensor; optionally, the preprocessing includes but is not limited to normalization processing.
[0031] S104. Train a preset classification model based on the training set, and construct a loss function of the classification model. It can be understood that the training set is used to train the preset classification model, and during the training process, the loss function is used to optimize the robustness of the classification model.
[0032] S105. Classify the training set based on the classification model, compare the classification result with the class map annotated in some areas to solve the loss, and back-propagate the loss to optimize the classification model; it can be understood that in order to judge the quality of the trained classification model, the classification result output by the classification model is compared with the class map obtained by other means, and the loss is solved. For the existing loss, the back-propagation method is used to further optimize the model so that the classification effect of the classification model can be best.
[0033] S106. Classify the pixels in the test set based on the classification model to obtain the categories of the pixels in the test set. Optionally, the pixels in the test set are classified using the classification model to effectively obtain the categories of the pixels.
[0034] Through the above method, the classification quality of the classification model can be improved.
[0035] In an optional embodiment of the present application, the first image and the second image are registered, and the registered first image and the second image are fused to obtain the first image tensor and the second image tensor. The specific process is:
[0036] Performing an upsampling operation on the first image to increase the size of the first image; performing a segmentation operation on the second image to reduce the size of the second image; so that the number of channels of the first image and the number of channels of the second image are both extended to 4, and the size of the first image after the operation is the same as the size of the second image;
[0037] Obtain all the registration schemes, and select the registration scheme with the largest sum as the result scheme of the registration, and swap the channels of the first image and the second image to complete the registration;
[0038] Decomposing the registered first image to obtain an intensity component of the first image; decomposing the registered second image to obtain an intensity component of the second image;
[0039] deriving a weighted intensity component based on an intensity component of the first image and an intensity component of the second image;
[0040] An edge detection operator of the first image and an edge detection operator of the second image are introduced, and a weighted intensity component is combined to obtain a weighted edge detection operator;
[0041] The first image and the second image after registration are fused based on the weighted edge detection operator to obtain the first image tensor and the second image tensor.
[0042] Specifically, in this embodiment, the size and number of channels of the first image and the second image are mainly aligned. Since the size of the second image is four times that of the first image, and the second image has one channel and the first image has four channels, the second image needs to be segmented so that the size of the second image is halved and the number of channels becomes four; the first image needs to be upsampled twice so that the size of the first image is doubled; at this time, the size of the first image is the same as the size of the second image, and the number of channels is also the same, so as to facilitate subsequent feature fusion.
[0043] Furthermore, the first image and the second image are registered. Since the order of the channels of the first image and the second image after segmentation is not necessarily aligned, in order to facilitate subsequent feature fusion, the first image and the second image need to be registered; first, the Gram tensor of the first image and the second image is calculated:
[0044] Where u is the u-th channel of the first image, v is the v-th channel of the second image, W is the width of the first image, H is the height of the second image, and x and y refer to the element at the x-th row and y-th column in the first image or the second image respectively;
[0045] By solving the four queens problem, all the registration schemes are obtained, and the scheme with the largest sum is selected as the registration scheme, and the channels of the first image and the second image are registered;
[0046] Secondly, perform IHS decomposition on the first image and the second image to obtain the intensity component of the first image And the intensity component of the second image is obtained Among them, α i and β i is an adaptive parameter and can be obtained by solving the following optimization problem:
[0047]
[0048] in, Based on this, a trade-off intensity component I is proposed C =γI M +(1-γ)I P ,
[0049] Furthermore, an edge detection operator is introduced into the first image. and an edge detection operator based on the second image in, The recommended values of λ and ∈ are: λ = 10 -9 ,∈=10 -10 ;
[0050] Based on this, a trade-off edge detection operator is proposed and They are:
[0051]
[0052] The first image and the second image are mutually fused to obtain the first image tensor and the second image tensor:
[0053]
[0054] The first tensor and the second tensor can be effectively obtained in the above manner.
[0055] In an optional embodiment of the present application, the specific process of preprocessing the first image tensor and the second image tensor, and selecting the training set and the test set based on the preprocessed first image tensor and the second image tensor is:
[0056] Normalizing the values of all pixels in the first image tensor, and also normalizing the values of all pixels in the second image tensor;
[0057] Find the pixel corresponding to the class label in the normalized first image tensor and the second image tensor, take the 32*32 area of the first image and the second image neighborhood centered on the pixel, take 1% of the samples of each class as the training set, and the rest of the samples as the test set.
[0058] Specifically, in this embodiment, first, the data normalization method is used to normalize the values of the pixels in the first tensor image and the second tensor image to between [0, 1] respectively; secondly, the pixels with corresponding class lists are found in the normalized images, and a 32*32 area of the first image and the second image is taken with each pixel as the center, and the neighborhood of each block serves the classification of the center point; then 1% of the samples of each class are taken as the training set, and the rest are taken as the test set.
[0059] In an optional embodiment of the present application, the preset classification model includes a first module, a second module and a third module;
[0060] The first module includes a 3*3 convolutional layer; wherein the step size of the convolutional layer is 1, the number of input channels is n, and the number of output channels is 2n; the first module is used to obtain shallow information;
[0061] The second module includes a self-learning branch and a mutual learning branch, and both the self-learning branch and the mutual learning branch include two 3*3 convolutional layers; wherein, the step size of one convolutional layer is 1, the number of input channels is 2n, and the number of output channels is 2n; the step size of the other convolutional layer is 1, the number of input channels is 2n, and the number of output channels is 2n; the self-learning branch is used to obtain deep information, and the mutual learning branch is used to learn the information of the self-learning branch under the supervision of the ternary loss function;
[0062] The third module includes a residual layer, that is, a 1*1 convolution layer; wherein the step size of the convolution layer is 1, the number of input channels is n, and the number of output channels is 3n; wherein the third module is used for fusion.
[0063] Specifically, the specific parameters of the classification model in this embodiment are as follows:
[0064] First image branch:
[0065] First layer: 3*3 convolutional layer, step size is 1, input channel number is n, output channel number is 2n;
[0066] Residual block 1:
[0067] Self-learning branch: 3*3 convolutional layer, step size is 1, number of input channels is 2n, number of output channels is 2n;
[0068] Mutual learning branch: 3*3 convolutional layer, step size 1, input channel number 2n, output channel number 2n;
[0069] Residual branch: 1*1 convolution layer, step size 1, input channel number n, output channel number 3n;
[0070] Residual block 2:
[0071] Self-learning branch: 3*3 convolutional layer, step size is 1, number of input channels is 2n, number of output channels is 2n;
[0072] Mutual learning branch: 3*3 convolutional layer, step size 1, input channel number 2n, output channel number 2n;
[0073] Residual branch: 1*1 convolution layer, step size 1, input channel number n, output channel number 3n;
[0074] Second image branch:
[0075] First layer: 3*3 convolutional layer, step size is 1, input channel number is n, output channel number is 2n;
[0076] Residual block 1:
[0077] Self-learning branch: 3*3 convolutional layer, step size is 1, number of input channels is 2n, number of output channels is 2n;
[0078] Mutual learning branch: 3*3 convolutional layer, step size 1, input channel number 2n, output channel number 2n;
[0079] Residual branch: 1*1 convolution layer, step size 1, input channel number n, output channel number 3n;
[0080] Residual block 2:
[0081] Self-learning branch: 3*3 convolutional layer, step size is 1, number of input channels is 2n, number of output channels is 2n;
[0082] Mutual learning branch: 3*3 convolutional layer, step size 1, input channel number 2n, output channel number 2n;
[0083] Residual branch: 1*1 convolution layer, step size 1, input channel number n, output channel number 3n;
[0084] First image branch:
[0085] First layer: 3*3 convolutional layer, step size is 1, input channel number is n, output channel number is 2n;
[0086] Residual block 1:
[0087] Self-learning branch: 3*3 convolutional layer, step size is 1, number of input channels is 2n, number of output channels is 2n;
[0088] Mutual learning branch: 3*3 convolutional layer, step size 1, input channel number 2n, output channel number 2n;
[0089] Residual branch: 1*1 convolution layer, step size 1, input channel number n, output channel number 3n;
[0090] Residual block 2:
[0091] Self-learning branch: 3*3 convolutional layer, step size is 1, number of input channels is 2n, number of output channels is 2n;
[0092] Mutual learning branch: 3*3 convolutional layer, step size 1, input channel number 2n, output channel number 2n;
[0093] Residual branch: 1*1 convolutional layer, stride 1, input channel number n, output channel number 3n.
[0094] In an optional embodiment of the present application, the specific process of training a preset classification model based on the training set includes:
[0095] Obtain shallow information of the first image tensor and shallow information of the second image tensor through a first module of a preset classification model;
[0096] Obtaining the self-learning result and the mutual learning result of the first image tensor through the second module of the preset classification model, and obtaining the self-learning result and the mutual learning result of the second image tensor;
[0097] Through the preset classification model, the self-learning result of the first image tensor and the mutual learning result of the second image tensor are fused through the adaptive channel module, and the mutual learning result of the first image tensor and the self-learning result of the second image tensor are fused through the adaptive channel module; and the fusion result is added to the input information of the residual branch to obtain the output result of the first image tensor and the output result of the second image tensor respectively.
[0098] In an optional embodiment of the present application, the method for constructing the adaptive channel module is:
[0099] Input the anchor example A, positive example P and negative example N to the preset attention module respectively, and set the comprehensive distance tensor Z = |AN|-|AP|, and globally pool the comprehensive distance tensor to obtain the distance vector ζ; where |AN| is the absolute value of the difference between the anchor example A and the negative example N, and |AP| is the absolute value of the difference between the anchor example A and the positive example P; it can be understood that the distance vector ζ i The larger it is, the smaller the Euclidean distance between the anchor example A and the positive example P of the corresponding channel i is, and the higher the similarity is;
[0100] Select the n channels with the highest distance vector similarity, separate their anchor examples A and positive examples P, and set them as the first anchor example A s and the first positive example P s ; The remaining anchor examples A and positive examples P are set as the second anchor example A d and the second positive example P d ;
[0101] The first anchor example A s And the first positive example P s Weighted to get the weight value F AP ;
[0102] The weight value F AP 、Second anchor example A d and the second positive example P d Cascade to obtain the output results of the adaptive channel module.
[0103] In an optional embodiment of the present application, after adding the fusion result to the input information of the residual branch to obtain the output result of the first image tensor and the output result of the second image tensor respectively, the method further includes:
[0104] The output result of the first image tensor and the output result of the second image tensor are sequentially cascaded, globally pooled, and fully connected, and then classified in the classifier to obtain the category of each pixel.
[0105] In an optional embodiment of the present application, the fully connected process is performed in a fully connected layer, and the fully connected layer includes a first fully connected layer and a second fully connected layer;
[0106] The first fully connected layer includes 864 input nodes and 64 output nodes; the second fully connected layer includes 64 input nodes and output nodes of the number of sample categories.
[0107] In an optional embodiment of the present application, the ternary loss function used in the process of fusing the first fused image and the second fused image is:
[0108] L t =max(d(A,P)-d(A,N)+margin,0); wherein max() is the maximum value function, and margin is the interval in the ternary loss function;
[0109] The cross entropy loss function used in the classifier is:
[0110] Among them, p(x) is the true probability distribution of the classification, q(x) is the predicted probability distribution of the classification, and x is the pixel.
[0111] In an optional embodiment of the present application, the loss function is:
[0112] Among them, L c is the cross entropy loss function, L t is the ternary loss function, is the i-th ternary loss, λ i is the weight of the i-th ternary loss.
[0113] In an optional embodiment of the present application, the number of training samples is 2% of the total data set, the initial learning rate is 0.0003, the total number of iterations is 1000, and the number of samples input each time is 64.
[0114] In an optional embodiment of the present application, a detailed simulation experiment is used for illustration.
[0115] Simulation conditions:
[0116] The hardware platform is: HP-Z840 workstation, TITAN-X-12GB-GPU, 64GB-RAM.
[0117] The software platform is: Python, PyTorch deep learning framework.
[0118] Simulation content and results:
[0119] The images used in the simulation experiment of the present invention are remote sensing images of Hohhot, where PAN images consist of 8404*8004 pixels and MS images consist of 2101*2001*4 pixels; the labeled classification scenes include water, roads, wasteland, vegetation, farmland, and 6 types of buildings, a total of 11 categories. After training, the OA of our method is 0.9904, AA is 0.9896, and the Kappa coefficient is 0.9889. The specific classification accuracy of each category is as follows:
[0120]
[0121]
[0122] Through simulation experiments, it can be concluded that the method of the present application can be beneficial to the improvement of classification performance, and the proposed adaptive channel module can also further improve the classification performance of remote sensing images.
[0123] Beneficial effects of the present invention:
[0124] The present invention provides a dual-branch mutual fusion image classification method, which first obtains an original data set, and reads a first image and a second image, as well as a corresponding partial area labeled class map from the original data set; aligns the first image and the second image, and fuses the aligned first image and the second image to obtain a first image tensor and a second image tensor; preprocesses the first image tensor and the second image tensor, and selects a training set and a test set based on the preprocessed first image tensor and the second image tensor; trains a preset classification model based on the training set, and constructs a loss function of the classification model; it can be understood that the preset classification model is trained using the training set, and during the training process, the loss function is used to optimize the robustness of the classification model; classifies the training set based on the classification model, compares the classification result with the partial area labeled class map to solve the loss, and back-propagates the loss to optimize the classification model; classifies the pixels in the test set based on the classification model, and obtains the category of the pixels in the test set; through the above method, the classification quality of the classification model can be improved.
[0125] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A dual-branch mutual fusion image classification method, characterized in that: include: Acquire an original data set, and read the first image and the second image, as well as the corresponding class map of partial area annotations from the original data set; The first image is multispectral remote sensing data, and the second image is panchromatic remote sensing data; Registering the first image after the upsampling operation and the second image after the segmentation operation, and fusing the registered first image and the second image to obtain a first image tensor and a second image tensor; comprising: Perform IHS decomposition on the first image and the second image to obtain the intensity component I of the first image M and the intensity component I of the second image P , respectively: Among them, α i and β i is an adaptive parameter; By solving the optimization problem, we get: according to Introducing the trade-off intensity component I C , respectively: I C =γI M +(1-γ)I P ; Introduce the edge detection operator of the first image and the edge detection operator of the second image in, Introducing the trade-off edge detection operator and They are: The first image after registration and the second image after registration are mutually fused to obtain the first image tensor and the second image tensor Preprocessing the first image tensor and the second image tensor, and selecting a training set and a test set based on the preprocessed first image tensor and the second image tensor; Training a preset classification model based on the training set, and constructing a loss function of the classification model; Classifying the training set based on the classification model, comparing the classification result with the class map annotated with a partial area to solve the loss, and back-propagating the loss to optimize the classification model; The pixels in the test set are classified based on the classification model to obtain categories of the pixels in the test set.
2. The dual-branch mutual fusion image classification method according to claim 1 is characterized in that: The specific process of preprocessing the first image tensor and the second image tensor and selecting a training set and a test set based on the preprocessed first image tensor and the second image tensor is: Normalizing the values of all pixels in the first image tensor, and also normalizing the values of all pixels in the second image tensor; Find the pixel corresponding to the class label in the normalized first image tensor and the second image tensor, take a 32*32 area of the neighborhood of the first image and the second image with the pixel as the center, take 1% of the samples of each class as the training set, and the remaining samples as the test set.
3. The dual-branch mutual fusion image classification method according to claim 1 is characterized in that: The preset classification model includes a first module, a second module and a third module; The first module includes a 3*3 convolution layer; wherein the step size of the convolution layer is 1, the number of input channels is n, and the number of output channels is 2n; the first module is used to obtain shallow information; The second module includes a self-learning branch and a mutual learning branch, and both the self-learning branch and the mutual learning branch include two 3*3 convolutional layers; wherein the step size of one convolutional layer is 1, the number of input channels is 2n, and the number of output channels is 2n; the step size of the other convolutional layer is 1, the number of input channels is 2n, and the number of output channels is 2n; the self-learning branch is used to obtain deep information, and the mutual learning branch is used to learn the information of the self-learning branch under the supervision of a ternary loss function; The third module includes a residual layer, that is, a 1*1 convolution layer; wherein the step size of the convolution layer is 1, the number of input channels is n, and the number of output channels is 3n; wherein the third module is used for fusion.
4. The dual-branch mutual fusion image classification method according to claim 3 is characterized in that: The specific process of training the preset classification model based on the training set includes: Obtaining shallow information of the first image tensor and shallow information of the second image tensor through the first module of the preset classification model; Obtaining a self-learning result and a mutual learning result of the first image tensor through the second module of the preset classification model, and obtaining a self-learning result and a mutual learning result of the second image tensor; The self-learning result of the first image tensor and the mutual learning result of the second image tensor are fused through the preset classification model through an adaptive channel module, and the mutual learning result of the first image tensor and the self-learning result of the second image tensor are fused through an adaptive channel module; and the fusion result is added to the input information of the residual branch to obtain the output result of the first image tensor and the output result of the second image tensor respectively.
5. The dual-branch mutual fusion image classification method according to claim 4 is characterized in that: The construction method of the adaptive channel module is: Input the anchor example A, the positive example P and the negative example N to the preset attention module respectively, and set the comprehensive distance tensor Z = |AN|-|AP|, and globally pool the comprehensive distance tensor to obtain the distance vector ζ, where |AN| is the absolute value of the difference between the anchor example A and the negative example N, and |AP| is the absolute value of the difference between the anchor example A and the positive example P; Select the n channels with the highest distance vector similarity, and separate their anchor examples A and positive examples P, and set them as the first anchor example A s and the first positive example P s ; The remaining anchor examples A and positive examples P are set as the second anchor example A d and the second positive example P d ; The first anchor example A s And the first positive example P s Weighted fusion tensor F AP ; The fused tensor F AP 、Second anchor example A d and the second positive example P d Cascade to obtain the output results of the adaptive channel module.
6. The dual-branch mutual fusion image classification method according to claim 4 is characterized in that: After adding the fusion result to the input information of the residual branch to obtain the output result of the first image tensor and the output result of the second image tensor respectively, the method further includes: The output result of the first image tensor and the output result of the second image tensor are sequentially cascaded, globally pooled, and fully connected, and then classified in a classifier to obtain the category of each pixel.
7. The dual-branch mutual fusion image classification method according to claim 6 is characterized in that: The fully connected process is performed in a fully connected layer, and the fully connected layer includes a first fully connected layer and a second fully connected layer; The first fully connected layer includes 864 input nodes and 64 output nodes; the second fully connected layer includes 64 input nodes and output sample category number nodes.
8. The dual-branch mutual fusion image classification method according to claim 6, characterized in that: The mutual learning result of the first image tensor and the self-learning result of the second image tensor are fused through an adaptive channel module, and the ternary loss function used in the fusion process is: L t =max(d(A,P)-d(A,N)+margin,0); wherein max() is the maximum value function, and margin is the interval in the ternary loss function; The cross entropy loss function used in the classifier is: Among them, p(x) is the true probability distribution of the classification, q(x) is the predicted probability distribution of the classification, and x is the pixel.
9. The dual-branch mutual fusion image classification method according to claim 8, characterized in that: The loss function is: Among them, L c is the cross entropy loss function, L t is the ternary loss function, is the i-th ternary loss, λ i is the weight of the i-th ternary loss.