Training method of classification network, image classification method and device

Abstract

The present disclosure provides a training method of a classification network, an image classification method and device, comprising: obtaining a sample image and label information of the sample image; inputting the sample image into a binary initial feature extraction layer of a classification network to be trained to determine a target feature map corresponding to the sample image; wherein the initial feature extraction layer comprises a channel value adjustment parameter to be trained; dividing the target feature map into a plurality of feature units based on a preset size, and inputting the target feature map into a feature fusion layer; the feature fusion layer comprises a plurality of binary multi-layer perception modules, each of which is used for feature fusion and deep feature extraction of the feature units; and training the classification network based on a semantic feature map output by the feature fusion layer and the label information of the sample image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of neural network, in particular to a training method of a classification network, an image classification method and device. BACKGROUND

[0002] With the development of neural network technology, people have higher and higher requirements for the network computing speed and computing accuracy of neural network, and therefore the network scale of neural network is also getting larger. When a large-scale neural network is operated, it needs to use large-scale computing resources, and therefore the hardware requirements for the deployment device are also relatively high. Therefore, under the premise of ensuring network accuracy, compression of the neural network becomes a problem to be solved. SUMMARY

[0003] The present disclosure provides at least a training method of a classification network, an image classification method and device.

[0004] In a first aspect, the present disclosure provides a training method of a classification network, comprising:

[0005] obtaining a sample image and label information of the sample image;

[0006] inputting the sample image into a binary initial feature extraction layer of a classification network to be trained to determine a target feature map corresponding to the sample image, wherein the initial feature extraction layer comprises a channel value adjustment parameter to be trained;

[0007] dividing the target feature map into a plurality of feature units based on a preset size, and inputting the target feature map into a feature fusion layer, wherein the feature fusion layer comprises a plurality of binary multi-layer perception modules, and each multi-layer perception module is configured to perform feature fusion and deep feature extraction on the feature units;

[0008] training the classification network based on a semantic feature map output by the feature fusion layer and the label information of the sample image.

[0009] In the method, the initial feature extraction layer of the classification network to be trained includes a channel value adjustment parameter to be trained, and the channel value adjustment parameter can adjust the channel values of the first feature map of the sample image extracted by the initial feature extraction layer. Thus, the binarization of the first feature map after the channel value adjustment can be understood as dynamically determining a binarization threshold, thereby narrowing the accuracy gap between the binarization feature extraction and the full-precision feature extraction. Furthermore, the feature fusion layer can perform feature fusion and deep feature extraction on multiple feature units, thereby making the output semantic feature map take into account both local features and global features. Thus, the classification network trained based on the semantic feature map and the labeling information has high accuracy, and both the initial feature extraction layer and the feature fusion layer are binarized, thereby making the network size of the classification network small, i.e., both the network size and the network accuracy can be taken into account.

[0010] In a possible implementation, the inputting of the sample image into the binarized initial feature extraction layer of the classification network to be trained to determine a target feature map corresponding to the sample image includes:

[0011] performing global average pooling processing on the sample image to determine a first feature map corresponding to the sample image;

[0012] performing first adjustment on the channel values of the first feature map based on the channel value adjustment parameter to determine an adjusted second feature map;

[0013] performing binarization activation processing on the sample image and the second feature map to determine a third feature map;

[0014] performing feature extraction on the third feature map to determine a target feature map corresponding to the sample image.

[0015] Here, since the channel value adjustment parameter is dynamic (i.e., trainable), the embedding expression of the adjusted sample image is also dynamic, i.e., the sample image is represented by dynamic embedding. Thus, the accuracy gap between the binarization feature extraction and the full-precision feature extraction can be reduced.

[0016] In a possible implementation, the performing of the binarization activation processing on the sample image and the second feature map to determine a third feature map includes:

[0017] determining the third feature map based on a first threshold of a preset activation function and a difference between the channel values of the sample image and the second feature map.

[0018] In a possible implementation, the performing of the feature extraction on the third feature map to determine a target feature map corresponding to the sample image includes:

[0019] perform feature extraction on the third feature map to determine a fourth feature map;

[0020] perform second adjustment on feature values of the first feature map based on the channel value adjustment parameter to determine a fifth feature map after adjustment;

[0021] perform feature fusion on the fourth feature map and the fifth feature map to determine the target feature map.

[0022] Here, in order to avoid the influence of binarization on the accuracy of the extracted features, the features of the original sample image can be added to the fourth feature map, that is, the fifth feature map after the second adjustment on the first feature map is fused with the fourth feature map.

[0023] In a possible implementation, the output of an Nth multi-layer perception module in the feature fusion layer is the input of an (N+1)th multi-layer perception module, the input of the first multi-layer perception module is the target feature map, the output of the last multi-layer perception module is the semantic feature map, and N is a positive integer.

[0024] Deep feature extraction through the multi-layer perception module can ensure that the depth of the finally extracted semantic feature is deep enough, thereby improving the classification accuracy.

[0025] In a possible implementation, for any multi-layer perception module, the multi-layer perception module is configured to perform feature fusion and deep feature extraction on feature cells of an input feature map input to the multi-layer perception module by the following method:

[0026] perform binarization activation processing on the input feature map to determine a sixth feature map;

[0027] perform feature exchange on the feature cells of the sixth feature map according to at least one exchange distance to obtain an exchanged feature map;

[0028] perform feature extraction on the exchanged feature map and the sixth feature map respectively, and perform feature fusion on the input feature map to obtain a fused feature map;

[0029] perform activation processing on the fused feature map to obtain an output feature map of the multi-layer perception module.

[0030] Long-distance exchange can make the current feature cells fuse global features, and short-distance exchange can make the current feature cells fuse local features. Therefore, the feature map obtained through this implementation can combine local features and global features, and feature extraction based on convolution can only combine local features, so the features fused by this method are more comprehensive.

[0031] In a possible implementation, the feature units of the sixth feature map are exchanged to obtain an exchanged feature map, including:

[0032] For any feature unit, based on the interaction distance, a to-be-exchanged feature unit corresponding to the feature unit in the sixth feature map is determined;

[0033] Based on the value of the to-be-exchanged feature unit on the corresponding channel, the value of the feature unit on each channel after feature exchange is determined.

[0034] In a possible implementation, the exchanged feature map and the sixth feature map are respectively extracted to obtain a fused feature map, including:

[0035] The exchanged feature map and the sixth feature map are respectively extracted based on a binary multi-layer perception to determine a plurality of deep feature maps;

[0036] After the plurality of deep feature maps are normalized, the input feature map is fused to obtain the fused feature map.

[0037] In a possible implementation, the classification network is trained based on the semantic feature map output by the feature fusion layer and the label information of the sample image, including:

[0038] An all-precision teacher network corresponding to the classification network to be trained is obtained;

[0039] The classification network is trained based on the semantic feature map output by the feature fusion layer, the label information of the sample image, and the all-precision teacher network.

[0040] The obtained all-precision teacher network can be trained, and the inference target of the all-precision teacher network can be the same as the inference target of the classification network. Because the network parameters of the all-precision teacher network are all-precision, the network precision of the all-precision teacher network is higher than that of the classification network. The network precision of the classification network can be improved by distillation training of the classification network by the all-precision teacher network.

[0041] In a possible implementation, the classification network is trained based on the semantic feature map output by the feature fusion layer, the label information of the sample image, and the all-precision teacher network, including:

[0042] determine a first loss value based on the semantic feature map and a first prediction result of the sample image by the full-precision teacher network; and determine a second loss value based on the semantic feature map and label information of the sample image;

[0043] train the classification network based on the first loss value and the second loss value.

[0044] In a possible implementation, the determining of the first loss value based on the semantic feature map and the first prediction result of the sample image by the full-precision teacher network comprises:

[0045] determine a second prediction result of the classification network based on a distillation head module and the semantic feature map, wherein the distillation head module has the same number of parameter bits as the full-precision teacher network.

[0046] determine the first loss value based on the first prediction result and the second prediction result.

[0047] The purpose of keeping the number of parameter bits of the distillation head module the same as that of the full-precision teacher network is to enable the distillation head module to receive high-precision features from the full-precision teacher network, and on the other hand, the premise of distillation training is that the head module has the same number of parameter bits as the teacher network.

[0048] In a possible implementation, the determining of the second loss value based on the semantic feature map and the label information of the sample image comprises:

[0049] determine a third prediction result of the classification network based on a supervision head module and the semantic feature map.

[0050] determine the second loss value based on the third prediction result and the label information of the sample image.

[0051] In a possible implementation, after the training of the classification network is completed, the method further comprises:

[0052] construct an inference head module based on the trained distillation head module and the supervision head module, the inference head module being configured to determine an inference result based on an output of the feature fusion layer of the trained classification network when performing network inference.

[0053] This method can directly construct the inference head module without additional calculation, and the constructed inference head module has high precision.

[0054] In a second aspect, the embodiments of the present disclosure provide an image classification method, comprising:

[0055] obtain an image to be classified;

[0056] inputting the image to be classified into the classification network trained based on the training method of the classification network according to the first aspect or any possible implementation of the first aspect, to determine a classification result of the image to be classified.

[0057] In a third aspect, an apparatus for training a classification network is provided, including:

[0058] a first obtaining unit configured to obtain a sample image and label information of the sample image;

[0059] a feature extraction unit configured to input the sample image into a binarized initial feature extraction layer of a classification network to be trained, to determine a target feature map corresponding to the sample image, wherein the initial feature extraction layer includes a channel value adjustment parameter to be trained;

[0060] a feature fusion unit configured to divide the target feature map into a plurality of feature units based on a preset size, and input the target feature map into a feature fusion layer, wherein the feature fusion layer includes a plurality of binarized multi-layer perception modules, and each multi-layer perception module is configured to perform feature fusion and deep feature extraction on the feature units;

[0061] a training unit configured to train the classification network based on a semantic feature map output by the feature fusion layer and the label information of the sample image.

[0062] In a possible implementation, the feature extraction unit, when inputting the sample image into the binarized initial feature extraction layer of the classification network to be trained to determine the target feature map corresponding to the sample image, is configured to:

[0063] perform global average pooling processing on the sample image to determine a first feature map corresponding to the sample image;

[0064] perform first adjustment on channel values of the first feature map based on the channel value adjustment parameter to determine a second feature map after adjustment;

[0065] perform binarized activation processing based on the sample image and the second feature map to determine a third feature map;

[0066] perform feature extraction on the third feature map to determine the target feature map corresponding to the sample image.

[0067] In a possible implementation, the feature extraction unit, when performing binarized activation processing based on the sample image and the second feature map to determine a third feature map, is configured to:

[0068] determine the third feature map based on a first threshold of a preset activation function and a difference between the channel values of the sample image and the second feature map.

[0069] In a possible implementation, the feature extraction unit, when performing feature extraction on the third feature map and determining the target feature map corresponding to the sample image, is configured to:

[0070] perform feature extraction on the third feature map to determine a fourth feature map;

[0071] perform second adjustment on the feature values of the first feature map based on the channel value adjustment parameter to determine a fifth feature map after adjustment;

[0072] perform feature fusion on the fourth feature map and the fifth feature map to determine the target feature map.

[0073] In a possible implementation, an output of an Nth multi-layer perception module in the feature fusion layer is an input of an (N+1)th multi-layer perception module, an input of a first multi-layer perception module is the target feature map, an output of a last multi-layer perception module is the semantic feature map, and N is a positive integer.

[0074] In a possible implementation, for any multi-layer perception module, the multi-layer perception module is configured to perform feature fusion and deep feature extraction on feature cells of an input feature map input into the multi-layer perception module by performing the following steps:

[0075] perform binary activation processing on the input feature map to determine a sixth feature map;

[0076] perform feature exchange on the feature cells of the sixth feature map according to at least one exchange distance to obtain an exchanged feature map;

[0077] perform feature extraction on the exchanged feature map and the sixth feature map respectively, and perform feature fusion on the exchanged feature map and the sixth feature map and the input feature map to obtain a fused feature map;

[0078] perform activation processing on the fused feature map to obtain an output feature map of the multi-layer perception module.

[0079] In a possible implementation, for any exchange distance, the feature fusion unit, when performing feature exchange on the feature cells of the sixth feature map to obtain an exchanged feature map, is configured to:

[0080] for any feature cell, determine a to-be-exchanged feature cell corresponding to the feature cell in the sixth feature map based on the exchange distance;

[0081] Determine values of the feature unit on each channel after the feature exchange based on values of the feature unit on the corresponding channel.

[0082] In a possible implementation, when performing feature fusion on the input feature map after performing feature extraction on the exchanged feature map and the sixth feature map respectively, the feature fusion unit is configured to:

[0083] Perform feature extraction on the exchanged feature map and the sixth feature map based on a binary multi-layer perception to determine a plurality of deep feature maps.

[0084] Perform normalization processing on the plurality of deep feature maps, and perform feature fusion on the input feature map to obtain the fused feature map.

[0085] In a possible implementation, when training the classification network based on the semantic feature map output by the feature fusion layer and the annotation information of the sample image, the training unit is configured to:

[0086] Obtain a full-precision teacher network corresponding to the classification network to be trained;

[0087] Train the classification network based on the semantic feature map output by the feature fusion layer, the annotation information of the sample image, and the full-precision teacher network.

[0088] In a possible implementation, when training the classification network based on the semantic feature map output by the feature fusion layer, the annotation information of the sample image, and the full-precision teacher network, the training unit is configured to:

[0089] Determine a first loss value based on a first prediction result of the sample image based on the semantic feature map and the full-precision teacher network, and determine a second loss value based on the semantic feature map and the annotation information of the sample image;

[0090] Train the classification network based on the first loss value and the second loss value.

[0091] In a possible implementation, when determining the first loss value based on the first prediction result of the sample image based on the semantic feature map and the full-precision teacher network, the training unit is configured to:

[0092] Determine a second prediction result of the classification network based on a distillation head module and the semantic feature map, wherein a number of parameter bits of the distillation head module is the same as a number of parameter bits of the full-precision teacher network;

[0093] Determine the first loss value based on the first prediction result and the second prediction result.

[0094] In a possible implementation, the training unit is configured to, when determining the second loss value based on the semantic feature map and the label information of the sample image:

[0095] determine a third prediction result of the classification network based on the supervised head module and the semantic feature map;

[0096] determine the second loss value based on the third prediction result and the label information of the sample image.

[0097] In a possible implementation, after the training of the classification network is completed, the apparatus further includes a construction unit configured to:

[0098] construct an inference head module based on the trained distillation head module and the supervised head module, the inference head module being configured to determine an inference result based on the output of the feature fusion layer of the trained classification network when performing network inference.

[0099] In a fourth aspect, an image classification apparatus is provided, and the apparatus includes:

[0100] a second acquisition unit configured to acquire a to-be-classified image;

[0101] a classification unit configured to input the to-be-classified image into the classification network trained by the training method of the classification network according to the first aspect or any possible implementation of the first aspect, and determine a classification result of the to-be-classified image.

[0102] In a fifth aspect, an apparatus is provided, and the apparatus includes a processor, a memory, and a bus, the memory stores machine readable instructions executable by the processor, when the apparatus is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the first aspect or any possible implementation of the first aspect, or perform the steps of the second aspect.

[0103] In a sixth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, when the computer program is run by a processor, the steps of the first aspect or any possible implementation of the first aspect, or the steps of the second aspect are performed.

[0104] For the effects of the apparatus for training the classification network, the computer device, and the computer readable storage medium, refer to the description of the method for training the classification network, which will not be repeated here.

[0105] In order to make the above objectives, characteristics and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0106] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. The drawings herein are incorporated into the description and form a part of the description, which show the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0107] Figure 1 A schematic diagram of a method for binarizing and compressing a convolutional neural network provided by an embodiment of the present disclosure is shown;

[0108] Figure 2 A flowchart of a training method of a classification network provided by an embodiment of the present disclosure is shown;

[0109] Figure 3 A flowchart of a method for determining a target feature map corresponding to a sample image provided by an embodiment of the present disclosure is shown;

[0110] Figure 4 An internal structure schematic diagram of an initial feature extraction layer provided by an embodiment of the present disclosure is shown;

[0111] Figure 5 A structure schematic diagram of a feature fusion layer provided by an embodiment of the present disclosure is shown;

[0112] Figure 6 A flowchart of an execution method of a multi-layer perception module provided by an embodiment of the present disclosure is shown;

[0113] Figure 7a An execution schematic diagram of a short-distance exchange operation provided by an embodiment of the present disclosure is shown;

[0114] Figure 7b An execution schematic diagram of a long-distance exchange operation provided by an embodiment of the present disclosure is shown;

[0115] Figure 8 A schematic diagram of an internal structure of a multi-layer perception module provided by an embodiment of the present disclosure is shown;

[0116] Figure 9 A structure schematic diagram of a training method of a classification network provided by an embodiment of the present disclosure is shown;

[0117] Figure 10 A flowchart of an image classification method provided by an embodiment of the present disclosure is shown.

[0118] Figure 11 An architectural schematic diagram of a training device of a classification network provided by an embodiment of the present disclosure is shown.

[0119] Figure 12 An architectural schematic diagram of an image classification device provided by an embodiment of the present disclosure is shown.

[0120] Figure 13 A structural schematic diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0121] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings for the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure and not all the embodiments. The components of the embodiments of the present disclosure described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0122] In related technologies, when compressing a classification network, a full-precision convolutional neural network (CNN) is generally binarized and compressed, and the network result is as shown in Figure 1 In the initial feature extraction module, a binarization threshold is generally set to perform binarization processing on the extracted feature map, and then deep feature extraction is performed based on the binarized convolution module, and result prediction and loss value calculation are performed based on the head module, and the loss value training module. However, in this method, on the one hand, the binarization threshold is fixed, and the accuracy is severely damaged when the feature map is binarized; on the other hand, the deep feature extraction based on the convolution module can only guarantee the feature depth, and ignores the correlation between the features, so this compression method will result in low model accuracy.

[0123] Based on the above research, the present disclosure provides a training method of a classification network, in the initial feature extraction layer of the classification network to be trained, including a channel value adjustment parameter to be trained, which can adjust the channel value of the first feature map of the sample image extracted by the initial feature extraction layer. Thus, the binarization processing of the first feature map after the channel value adjustment can be understood as dynamically determining the binarization threshold, thereby narrowing the accuracy gap between the binarization feature extraction and the full-precision feature extraction. Further, the feature fusion layer can perform feature fusion and deep feature extraction on multiple feature units, so that the output semantic feature map takes into account both local features and global features. Therefore, the classification network trained based on the semantic feature map and the labeling information has high accuracy, and the initial feature extraction layer and the feature fusion layer are both binarized, so that the network size of the classification network is small, that is, both the network size and the network accuracy can be taken into account.

[0124] The defects of the above solutions are the results of the inventors after practice and careful research, therefore, the discovery process of the above problems and the solutions proposed by the present disclosure to solve the above problems should be the contributions of the inventors to the present disclosure in the process of the present disclosure.

[0125] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0126] In order to facilitate the understanding of the present embodiment, first, a training method of a classification network disclosed by the present embodiment is introduced in detail, the execution subject of the training method of the classification network provided by the present embodiment is generally a computer device with certain computing power, which for example includes: a terminal device or a server or other processing device, the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the training method of the classification network can be realized by the way that the processor calls the computer readable instructions stored in the memory.

[0127] Referring to Figure 2 The flowchart of the training method of the classification network provided by the present embodiment is shown, the method includes steps 201-204, wherein:

[0128] Step 201, acquiring a sample image and labeling information of the sample image.

[0129] Step 202, inputting the sample image into a binarized initial feature extraction layer of a classification network to be trained to determine a target feature map corresponding to the sample image; wherein the initial feature extraction layer comprises a channel value adjustment parameter to be trained.

[0130] Step 203, dividing the target feature map into a plurality of feature units based on a preset size, and inputting the target feature map into a feature fusion layer, the feature fusion layer comprising a plurality of binarized multi-layer perception modules, each multi-layer perception module being configured to perform feature fusion and deep feature extraction on the feature units.

[0131] Step 204, training the classification network based on a semantic feature map output by the feature fusion layer and annotation information of the sample image.

[0132] The following is a detailed introduction to the above method.

[0133] For step 201,

[0134] In a possible implementation, the sample image can be an image to be classified, and the annotation information of the sample image can be a pre-labeled classification result of the sample image.

[0135] For step 202,

[0136] In a possible implementation, when the sample image is input into the binarized initial feature extraction layer of the classification network to be trained to determine the target feature map corresponding to the sample image, the following steps can be performed, as shown in the following steps: Figure 3

[0137] Step 301, performing global average pooling processing on the sample image to determine a first feature map corresponding to the sample image.

[0138] Step 302, performing first adjustment on channel values of the first feature map based on the channel value adjustment parameter to determine an adjusted second feature map.

[0139] Step 303, performing binarized activation processing based on the sample image and the second feature map to determine a third feature map.

[0140] Step 304, performing feature extraction on the third feature map to determine a target feature map corresponding to the sample image.

[0141] ​In step 301, after global average pooling (GAP) processing of the sample image, a preliminary embedding representation corresponding to the sample image can be obtained, that is, the first feature map is a preliminary embedding of the sample image.

[0142] Then in step 302, the channel values of the first feature map are adjusted by the channel value adjustment parameter, which can be understood as adjusting the preliminary embedding.

[0143] Specifically, the channel value adjustment parameter can include a first adjustment parameter, a second adjustment parameter, and a third adjustment parameter. When the channel values of the first feature map are adjusted based on the channel value adjustment parameter, the first adjustment parameter and the second adjustment parameter can be used to adjust the channel values of the first feature map. For example, the values on each channel of the first feature map can be linearly operated with the first adjustment parameter and the second adjustment parameter. The third adjustment parameter is used to perform a second adjustment on the channel values of the first feature map, which will be described below.

[0144] Here, since the channel value adjustment parameter is dynamic (i.e., trainable), the adjusted embedding is also dynamic, that is, the sample image is represented by a dynamic embedding.

[0145] In step 303, the second feature map is subjected to binary activation processing based on the sample image and the second feature map to determine a third feature map. This can mean that based on a first threshold of a preset activation function and a difference between the channel values of the sample image and the second feature map, the third feature map is determined.

[0146] Specifically, the difference between the channel values of the sample image and the second feature map can be calculated. If the channel value of the sample image is greater than the channel value of the second image, the channel value is set to +1. If the channel value of the sample image is less than or equal to the channel value of the second image, the channel value is set to -1.

[0147] Here, although the first threshold of the activation function is fixed, since the channel values of the first feature map are adjusted in step 302 before the binary processing based on the activation function, it is actually equivalent to dynamically adjusting the first threshold of the activation function.

[0148] In step 304, when the third feature map is extracted and the target feature map corresponding to the sample image is determined, the third feature map can be extracted first to determine a fourth feature map; then the feature values of the first feature map are adjusted based on the channel value adjustment parameter to determine an adjusted fifth feature map; and the fourth feature map and the fifth feature map are fused to determine the target feature map.

[0149] Specifically, when the feature values of the first feature map are adjusted based on the channel value adjustment parameter, the feature values of the first feature map can be adjusted based on the first adjustment parameter and the third adjustment parameter, and the specific adjustment method can be similar to the first adjustment method.

[0150] The fourth feature map and the fifth feature map can be fused, for example, by summing the values on the corresponding channels of the fourth feature map and the fifth feature map to determine the target feature map.

[0151] For example, the internal structure of the initial feature extraction layer can refer to Figure 4 As shown, W1 represents the first adjustment parameter, W2 represents the second adjustment parameter, and W3 represents the third adjustment parameter.

[0152] After the sample image is input, a part of it is processed by GAP to obtain a first feature map, and then the first feature map is adjusted by the first adjustment parameter and the second adjustment parameter to obtain a second feature map. For example, the first feature map can be adjusted based on the first adjustment parameter by the following formula:

[0153] α(X)=GAP(X)W1+bias α (1)

[0154] Where X represents each feature point (i.e. pixel point) of the second sample image, bias α represents a bias parameter (hyperparameter), and α(X) represents the adjusted feature map. GAP(X) represents the first feature map.

[0155] For example, the second adjustment parameter can be adjusted by the following formula:

[0156] β(X)=α(X)W2+bias β (2)

[0157] Where bias β represents a bias parameter (hyperparameter), and β(X) represents the second feature map.

[0158] Then, based on the difference between the channel values of the sample image and the second feature map and the first threshold, a binary activation process is performed to determine a third feature map. The third feature map can be calculated by the following formula:

[0159]

[0160] wherein x represents the channel value in the sample image, β(X) represents the first threshold, Q b (x) represents the third feature map, and sign represents an activation function.

[0161] After the above processing, the sample image can be converted into a binary embedding representation, and then the features of the third feature map can be extracted by binary convolution to obtain a fourth feature map.

[0162] Further, in order to avoid the influence of binarization on the accuracy of the extracted features, the features of the original sample image can be added to the fourth feature map, that is, the fifth feature map after the second adjustment of the first feature map is fused with the fourth feature map.

[0163] For example, when the first feature map is adjusted for the second time, the following formula can be used:

[0164] γ(X)=α(X)W3+bias γ (4)

[0165] wherein bias γ represents a bias parameter (hyperparameter), and γ(X) represents the fifth feature map.

[0166] When the fifth feature map and the fourth feature map are fused, the following formula can be used for fusion:

[0167] x2'=x1'+γ(X) (5)

[0168] x1' represents the fourth feature map after feature extraction, and x2' represents the target feature map after fusion.

[0169] In the above method, the channel values of the first feature map are adjusted, and then the first feature map after the adjustment of the channel values is binarized. In essence, this can be understood as dynamically determining the binarization threshold, thereby narrowing the accuracy gap between binarized feature extraction and full-precision feature extraction.

[0170] Here, although the third feature map is a binary feature map, the fourth feature map obtained after feature extraction (binary convolution) on the third feature map is also a binary feature map, but since the fifth feature map is not a binary feature map, the target feature map obtained after fusion of the fourth feature map and the fifth feature map is also not a binary feature map, so when subsequent processing is performed, a binary operation still needs to be performed through an activation function.

[0171] For step 203,

[0172] Here, when the target feature map is divided into multiple feature units, it can be divided according to a preset size, and each feature unit can be regarded as a symbol token. Each token can be understood as a c x n x m matrix, where c represents the number of channels, and n and m represent the preset size.

[0173] The structure of the feature fusion layer can be exemplarily as shown in Figure 5 The output of the Nth multi-layer perception module is the input of the (N+1)th multi-layer perception module, the input of the first multi-layer perception module is the target feature map, the output of the last multi-layer perception module is the semantic feature map, and N is a positive integer.

[0174] For any multi-layer perception module, in one possible implementation, feature fusion and deep feature extraction can be performed by a method as shown in Figure 6 The method includes the following steps:

[0175] Step 601, performing binary activation processing on the input feature map to determine a sixth feature map.

[0176] Step 602, performing feature exchange on the feature units of the sixth feature map according to at least one exchange distance to obtain an exchanged feature map.

[0177] Step 603, performing feature extraction on the exchanged feature map and the sixth feature map respectively, and performing feature fusion on the input feature map to obtain a fused feature map.

[0178] Step 604, performing activation processing on the fused feature map to obtain the output feature map of the multi-layer perception module.

[0179] In step 601, the binary activation processing of the input feature map can be understood as binary processing of the input feature map based on an activation function. Since the input of the first multi-layer perception module is the target feature map, the target feature map is not a binary feature map, and therefore the network parameters can be further compressed through activation processing. The input of other multi-layer perception modules is the output of the previous multi-layer perception module, and the output of the previous multi-layer perception module has been processed in the above steps. Therefore, binary activation processing is also needed.

[0180] For example, when performing binary activation processing, the following formula can be used for calculation:

[0181]

[0182] Here, a represents the second threshold of the activation function, sign represents the activation function, x represents the feature value of the input feature map, Q b (x) represents the sixth feature map.

[0183] In step 602, the exchange distance can be a preset distance, for example, it can refer to the distance adjacent to the feature unit, or half of the size of the feature map.

[0184] For any interaction distance, the feature exchange of the feature unit of the sixth feature map can refer to, for any feature unit, determining the to-be-exchanged feature unit corresponding to the feature unit in the sixth feature map based on the interaction distance; determining the value of each channel of the feature unit after feature exchange based on the value of the to-be-exchanged feature unit on the corresponding channel.

[0185] In one possible implementation, the exchange distance can be divided into long-distance exchange operation and short-distance exchange operation based on different exchange distances. The long-distance exchange operation can refer to the feature exchange of the to-be-exchanged unit with a distance of or from the current feature unit, and the short-distance exchange operation can refer to the feature exchange of the to-be-exchanged unit adjacent to the current feature unit.

[0186] For example, as shown in Figure 7a , if the current feature unit is feature unit A (i.e., the feature unit pointed by the arrow), the feature units adjacent to the current feature unit A are feature units B, C, D, and E; for feature unit a, when performing short-distance exchange operation, the values of feature units B, C, D, and E on the corresponding channels can be used to determine the values of feature unit a on each channel after feature exchange.

[0187] Specifically, when determining the values ​​of a feature unit in each channel after feature exchange based on the values ​​of the feature units to be exchanged in the corresponding channels, the corresponding channels of feature units to be exchanged at different positions may be different. For example, if a feature unit has c channels, the values ​​of the 0th to c / 4th channels of the feature units to be exchanged to the left of the current feature unit can be taken as the values ​​of the 0th to c / 4th channels of the feature unit a after feature exchange; the values ​​of the c / 4th to c / 4th channels of the feature units to be exchanged to the right of the current feature unit can be taken as the values ​​of the feature unit a after feature exchange. The values ​​on channels c / 2 are used as the values ​​on channels c / 4 to c / 2 of the feature unit a after feature exchange; the values ​​on channels c / 2 to 3c / 4 of the feature unit to be exchanged above the current feature unit are used as the values ​​on channels c / 2 to 3c / 4 of the feature unit a after feature exchange; the values ​​on channels 3c / 4 to c of the feature unit to be exchanged below the current feature unit are used as the values ​​on channels 3c / 4 to c of the feature unit a after feature exchange.

[0188] For example, such as Figure 7b As shown, if the current feature unit is feature unit A (i.e., the feature unit pointed to by the arrow), and the size of the feature map is 7*7 during long-distance exchange operations, then the distance to feature unit a is... or The units to be exchanged are feature units B, C, D, and E. The remaining exchange processes are similar to those described above and will not be repeated here.

[0189] In special cases, for feature units at the edge, if the feature units to be exchanged that are a certain exchange distance from the feature unit exceed the boundary, then it can be done as follows: Figure 7b As shown, feature maps are re-added at the edge locations, and the feature units to be exchanged are determined according to the exchange distance.

[0190] In practical implementation, the position coordinates of each feature unit to be exchanged can be determined by the following formula:

[0191] S(r1,r2)={y:y=(x1+r1,x2+r2)} (7)

[0192] Where S(r1,r2) represents the set of coordinates of the feature units to be exchanged, x1 and x2 represent the position coordinates of the current feature unit, and r1 and r2 represent the horizontal and vertical exchange distances. In the above formula, the coordinates of the feature units to be exchanged should be (x1,x2+r2), (x1+r1,x2), (x1-r1,x2), and (x1,x2-r2), respectively.

[0193] For example, when performing short-distance swap operations, the calculation can be performed using the following formula:

[0194]

[0195] wherein, denotes the value of the 0th channel to the c / 4th channel of the feature unit with the coordinate difference (-1, 0) from the current feature unit, A b [0:c / 4] S(-1,0) denotes the value of the 0th channel to the c / 4th channel of the feature unit with the coordinate difference (-1, 0) from the current feature unit, A b [c / 4:c / 2] S(1,0) denotes the value of the c / 4th channel to the c / 2th channel of the feature unit with the coordinate difference (1, 0) from the current feature unit, A b [c / 2:3c / 4] S(0,-1) denotes the value of the c / 2th channel to the 3c / 4th channel of the feature unit with the coordinate difference (0, -1) from the current feature unit, A b [3c / 4:c] S(0,1) denotes the value of the 3c / 4th channel to the cth channel of the feature unit with the coordinate difference (0, 1) from the current feature unit, Cat denotes the concat operation, and c denotes the number of channels.

[0196] For example, when performing the long-distance exchange operation, the calculation can be performed by the following formula:

[0197]

[0198] wherein, h denotes the length of the feature map, and w denotes the width of the feature map, denotes the current feature unit after the long-distance exchange, and the rest of the explanation is similar to that in formula (8), which will not be repeated here.

[0199] Through the above method, the current feature unit can be fused with global features through long-distance exchange, and the current feature unit can be fused with local features through short-distance exchange. Therefore, the feature map obtained by this method can combine local features and global features, and feature extraction based on convolution can only combine local features, so the features fused by this method are more comprehensive.

[0200] In step 603, the number of exchanged feature maps can be multiple, and after feature extraction is performed on the exchanged feature maps and the sixth feature map respectively, feature fusion is performed with the input feature map to obtain a fused feature map. The feature extraction can be performed on the exchanged feature maps and the sixth feature map based on a binary multi-layer perceptron to determine multiple deep feature maps, and then the multiple deep feature maps are normalized and fused with the input feature map to obtain the fused feature map.

[0201] Here, the purpose of the feature fusion with the input feature map is to prevent the phenomenon of overfitting or gradient problems of the features after the previous feature exchange and feature extraction.

[0202] Alternatively, in another possible implementation, the values on each channel of the to-be-exchanged feature unit can be averaged, and the average value can be taken as the value on the channel of the current feature unit.

[0203] The activation processing in step 604 can be different from the binary activation processing in steps 303 and 601, for example, it can be RPRelu processing, etc. The threshold values in the binary activation processing in steps 303 and 601 can be different, that is, the first threshold value and the second threshold value can be different.

[0204] For example, the internal structure of the multi-layer perception module can be as shown in Figure 8 The Binary MLP is a binary multi-layer perception machine, which is used for deep feature extraction, and the RPReLU is an activation function.

[0205] The normalization processing on the plurality of deep feature maps can be performed after the plurality of deep feature maps are fused, for example, the step of fusing the plurality of deep feature maps can be calculated by the following formula:

[0206]

[0207] wherein A i represents the fused feature map, represents the sixth feature map, represents the deep feature map after the long-distance exchange operation, represents the deep feature map after the short-distance exchange, represents the feature map after feature extraction corresponding to the sixth feature map (i.e. the output of the third MLP in Figure 8 represents the deep feature map after the long-distance exchange operation corresponding to the feature map after feature extraction (i.e. the output of the first MLP in represents the deep feature map after the short-distance exchange corresponding to the feature map after feature extraction (i.e. the output of the second MLP in Figure 8 represents the deep feature map after the short-distance exchange corresponding to the feature map after feature extraction (i.e. the output of the second MLP in Figure 8

[0208] After the normalization processing on the plurality of deep feature maps, the feature fusion with the input feature map is performed to obtain the fused feature map, which can be calculated by the following formula:

[0209] A i ​​= RPReLU(BN(A i ')+ A i-1 ) (11)

[0210] wherein, A i represents a fusion feature map, RPReLU represents an activation function, BN represents a normalization processing, and A i-1 represents an input feature map.

[0211] Here, the fusion feature map fuses the local features and the global features with higher accuracy, and thus, when determining the classification result based on the fusion feature map, the accuracy is higher.

[0212] For step 104,

[0213] In one possible implementation, when the classification network is trained based on the semantic feature map output by the feature fusion layer and the label information of the sample image, the classification result of the classification network can be determined based on the semantic feature map and a supervision head, then a loss value (for example, a cross-entropy loss) can be determined based on the classification result and the label information, and the classification network can be trained based on the loss value.

[0214] In another possible implementation, in order to improve the network accuracy of the classification network, the classification network can be distilled.

[0215] Specifically, when the classification network is trained based on the semantic feature map output by the feature fusion layer and the label information of the sample image, the following steps can be included:

[0216] Step A, obtaining a full-precision teacher network corresponding to the classification network to be trained.

[0217] The obtained full-precision teacher network can be trained, and the inference target of the full-precision teacher network can be the same as the inference target of the classification network. Since the network parameters of the full-precision teacher network are full-precision, the network accuracy of the full-precision teacher network is higher than that of the classification network. By distilling the classification network based on the full-precision teacher network, the network accuracy of the classification network can be improved.

[0218] Step B, training the classification network based on the semantic feature map output by the feature fusion layer, the label information of the sample image, and the full-precision teacher network.

[0219] In a possible implementation, when the classification network is trained based on the semantic feature map output by the feature fusion layer, the label information of the sample image, and the full-precision teacher network, a first loss value can be determined based on the semantic feature map and a first prediction result of the sample image by the full-precision teacher network; and a second loss value can be determined based on the semantic feature map and the label information of the sample image, and then the classification network is trained based on the first loss value and the second loss value.

[0220] The first loss value is used to represent a distillation loss of the full-precision teacher network during distillation training, and the second loss value is used to represent a classification loss of the classification network. Training the classification network in combination with the first loss value and the second loss value can improve the network precision of the classification network.

[0221] Specifically, when the first loss value is determined based on the semantic feature map and the first prediction result of the sample image by the full-precision teacher network, a second prediction result of the classification network can be determined based on a distillation head module and the semantic feature map; the parameter bit number of the distillation head module is the same as that of the full-precision teacher network; and then the first loss value is determined based on the first prediction result and the second prediction result.

[0222] Here, the purpose of keeping the parameter bit number of the distillation head module the same as that of the full-precision teacher network is to enable the distillation head module to receive high-precision features of the full-precision teacher network, and on the other hand, the premise of distillation training is that the parameter bit number of the head module is the same as that of the teacher network.

[0223] Specifically, when the second loss value is determined based on the semantic feature map and the label information of the sample image, a third prediction result of the classification network can be determined based on a supervision head module and the semantic feature map; and then the second loss value is determined based on the third prediction result and the label information of the sample image.

[0224] For example, the first loss value can be a relative entropy, a divergence (Kullback-Leibler divergence, KL) loss; and the second loss value can be a cross-entropy loss, etc.

[0225] When the classification network is trained based on the first loss value and the second loss value, the first loss value and the second loss value can be weighted and summed to determine a total loss value, and then the classification network is trained based on the total loss value.

[0226] In the calculation of the total loss value, for example, the following formula can be used:

[0227]

[0228] wherein, represents the second loss value, represents the first loss value, represents the output value of the supervision head module, represents the weight of the supervision head module, b s represents the bias parameter of the supervision head module, y represents the label information of the sample image, y t represents the output of the full-precision teacher network, that is, the first prediction result, represents the weight of the distillation head module, b d represents the bias parameter of the distillation head module, Z represents the semantic feature map output by the feature fusion layer, and L represents the total loss value.

[0229] In a possible implementation, after the training of the classification network is completed, the inference head module can be constructed based on the trained distillation head module and the supervision head module, that is, the parameters of the head inference module are constructed to improve the accuracy of the classification network when network recommendation is performed.

[0230] wherein the inference head module is configured to determine an inference result based on the output of the feature fusion layer of the trained classification network when network inference is performed.

[0231] For example, the internal calculation process of the inference head module can be as shown in the following formula:

[0232]

[0233] wherein BiMLPs(X) represents the inference result of the inference head module, W s represents the weight of the supervision head module, represents the weight of the distillation head module, b s represents the bias parameter of the supervision head module, b d represents the bias parameter of the distillation head module.

[0234] In this way, the inference head module can be directly constructed without additional and excessive calculation, and the accuracy of the constructed inference head module is relatively high.

[0235] The training method of the classification network will be described below with reference to specific structural drawings. As shown in FIG. 1, a structural schematic diagram of a training method of a classification network provided by an embodiment of the present disclosure is specifically provided, which specifically includes: Figure 9

[0236] ​After the sample image is acquired, part of the sample image is input into the full-precision teacher network for distillation training, and part of the sample image is input into the classification network to be trained for feature extraction.

[0237] Specifically, after being input into the classification network, the following steps are performed:

[0238] Step 1: inputting the sample image into an initial feature extraction layer to extract initial features (for details, refer to the introduction in Figure 4 ), to obtain a target feature map.

[0239] Step 2: dividing a plurality of tokens based on the target feature map.

[0240] Step 3: inputting the target feature map into a feature fusion layer to extract and fuse features between the tokens.

[0241] The feature fusion layer includes a plurality of multi-layer perception modules.

[0242] Step 4: inputting part of the semantic feature output by the feature fusion layer into a distillation head module to calculate a KL divergence loss together with the output of the full-precision teacher network, and inputting part of the semantic feature into a supervision head module to calculate a cross-entropy loss together with the label information of the sample image.

[0243] Step 5: training the classification network based on the KL loss and the cross-entropy loss.

[0244] Step 6: after the neural network training is completed, constructing an inference head module based on the distillation head module and the supervision head module for inference.

[0245] In the above method, the initial feature extraction layer of the classification network to be trained includes a channel value adjustment parameter to be trained, which can adjust the channel values of the first feature map of the sample image extracted by the initial feature extraction layer. Thus, the binarization processing of the first feature map after the channel value adjustment can be understood as dynamically determining the binarization threshold, thereby narrowing the accuracy gap between the binarization feature extraction and the full-precision feature extraction. Further, the feature fusion layer can perform feature fusion and deep feature extraction on a plurality of feature units, so that the output semantic feature map takes into account both local features and global features, thereby improving the accuracy of the classification network trained based on the semantic feature map and the label information. Moreover, the initial feature extraction layer and the feature fusion layer are both binarized, so the network size of the classification network is small, which can balance the network size and the network accuracy.

[0246] Based on the same concept, the disclosure also provides an image classification method, as shown in Figure 10 The flowchart of the image classification method provided by the disclosure includes the following steps:

[0247] Step 1001, obtaining an image to be classified.

[0248] Step 1002, inputting the image to be classified into the classification network trained by the training method of the classification network according to the above embodiments, and determining the classification result of the image to be classified.

[0249] Since the network size of the classification network trained by the training method of the classification network according to the above embodiments is small and the accuracy is high, the image to be classified can be quickly classified, and the classification result has high accuracy.

[0250] Those skilled in the art can understand that in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0251] Based on the same inventive concept, the disclosure embodiments also provide a classification network training device corresponding to the classification network training method. Since the principle of solving problems in the device in the disclosure embodiments is similar to the above classification network training method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described.

[0252] Referring to Figure 11 Fig. 1 shows an architecture schematic diagram of a classification network training device provided by the disclosure embodiments. The device comprises a first acquisition unit 1101, a feature extraction unit 1102, a feature fusion unit 1103, a training unit 1104 and a construction unit 1105; wherein,

[0253] The first acquisition unit 1101 is configured to acquire a sample image and annotation information of the sample image.

[0254] The feature extraction unit 1102 is configured to input the sample image into a binary initial feature extraction layer of a classification network to be trained, and determine a target feature map corresponding to the sample image; wherein the initial feature extraction layer comprises a channel value adjustment parameter to be trained.

[0255] The feature fusion unit 1103 is configured to divide the target feature map into a plurality of feature units based on a preset size, and input the target feature map into a feature fusion layer, wherein the feature fusion layer comprises a plurality of binary multi-layer perception modules, and each multi-layer perception module is configured to perform feature fusion and deep feature extraction on the feature units.

[0256] The training unit 1104 is configured to train the classification network based on a semantic feature map output by the feature fusion layer and the annotation information of the sample image.

[0257] In one possible implementation, the feature extraction unit 1102, when inputting the sample image into the binarized initial feature extraction layer of the classification network to be trained, and determining the target feature map corresponding to the sample image, is used to:

[0258] The sample image is subjected to global average pooling to determine the first feature map corresponding to the sample image;

[0259] Based on the channel value adjustment parameters, the channel values ​​of the first feature map are adjusted to determine the adjusted second feature map;

[0260] Binary activation processing is performed on the sample image and the second feature map to determine the third feature map;

[0261] Feature extraction is performed on the third feature map to determine the target feature map corresponding to the sample image.

[0262] In one possible implementation, the feature extraction unit 1102, when performing binary activation processing based on the sample image and the second feature map to determine the third feature map, is used to:

[0263] The third feature map is determined based on the first threshold of the preset activation function and the difference between the corresponding channel values ​​of the sample image and the second feature map.

[0264] In one possible implementation, the feature extraction unit 1102, when performing feature extraction on the third feature map to determine the target feature map corresponding to the sample image, is used to:

[0265] Feature extraction is performed on the third feature map to determine the fourth feature map;

[0266] Based on the channel value adjustment parameters, the feature values ​​of the first feature map are adjusted a second time to determine the adjusted fifth feature map;

[0267] The fourth feature map and the fifth feature map are fused to determine the target feature map.

[0268] In one possible implementation, the output of the Nth multilayer perception module in the feature fusion layer is the input of the (N+1)th multilayer perception module, the input of the first multilayer perception module is the target feature map, and the output of the last multilayer perception module is the semantic feature map, where N is a positive integer.

[0269] In one possible implementation, for any multilayer sensing module, the multilayer sensing module is used to perform feature fusion and deep feature extraction on the feature units of the input feature map input to the multilayer sensing module through the following method:

[0270] perform binary activation processing on the input feature map to determine a sixth feature map;

[0271] perform feature exchange on feature cells of the sixth feature map according to at least one exchange distance to obtain an exchanged feature map;

[0272] perform feature extraction on the exchanged feature map and the sixth feature map respectively, and perform feature fusion on the input feature map to obtain a fused feature map;

[0273] perform activation processing on the fused feature map to obtain an output feature map of the multi-layer perception module.

[0274] In a possible implementation, for any exchange distance, the feature fusion unit 1103, when performing feature exchange on feature cells of the sixth feature map to obtain an exchanged feature map, is configured to:

[0275] for any feature cell, determine a to-be-exchanged feature cell corresponding to the feature cell in the sixth feature map based on the interaction distance;

[0276] determine values of the feature cell on channels after feature exchange based on values of the to-be-exchanged feature cell on the channels.

[0277] In a possible implementation, the feature fusion unit 1103, when performing feature extraction on the exchanged feature map and the sixth feature map respectively, and performing feature fusion on the input feature map to obtain a fused feature map, is configured to:

[0278] perform feature extraction on the exchanged feature map and the sixth feature map respectively based on a binary multi-layer perception machine to determine a plurality of deep feature maps;

[0279] perform normalization processing on the plurality of deep feature maps, and perform feature fusion on the input feature map to obtain the fused feature map.

[0280] In a possible implementation, the training unit 1104, when training the classification network based on the semantic feature map output by the feature fusion layer and the annotation information of the sample image, is configured to:

[0281] obtain a full-precision teacher network corresponding to the to-be-trained classification network;

[0282] train the classification network based on the semantic feature map output by the feature fusion layer, the annotation information of the sample image, and the full-precision teacher network.

[0283] In a possible implementation, the training unit 1104 is configured to, when training the classification network based on the semantic feature map output by the feature fusion layer, the annotation information of the sample image, and the full-precision teacher network, perform the following operations:

[0284] determine a first loss value based on the semantic feature map and a first prediction result of the sample image by the full-precision teacher network, and determine a second loss value based on the semantic feature map and the annotation information of the sample image;

[0285] train the classification network based on the first loss value and the second loss value.

[0286] In a possible implementation, the training unit 1104 is configured to, when determining the first loss value based on the semantic feature map and a first prediction result of the sample image by the full-precision teacher network, perform the following operations:

[0287] determine a second prediction result of the classification network based on a distillation head module and the semantic feature map, where the distillation head module has the same number of parameters as the full-precision teacher network.

[0288] determine the first loss value based on the first prediction result and the second prediction result.

[0289] In a possible implementation, the training unit 1104 is configured to, when determining the second loss value based on the semantic feature map and the annotation information of the sample image, perform the following operations:

[0290] determine a third prediction result of the classification network based on a supervision head module and the semantic feature map.

[0291] determine the second loss value based on the third prediction result and the annotation information of the sample image.

[0292] In a possible implementation, after the training of the classification network is completed, the apparatus further includes a construction unit 1105 configured to:

[0293] construct an inference head module based on the trained distillation head module and the supervision head module, where the inference head module is configured to determine an inference result based on an output of the feature fusion layer of the trained classification network when performing network inference.

[0294] The description of the processing procedure of each module in the apparatus and the interaction procedure between the modules can refer to the related description in the method embodiments, which will not be repeated here.

[0295] Based on the same inventive concept, the embodiment of the present disclosure also provides an image classification device corresponding to the image classification method. Since the principle of the device solving the problem in the embodiment of the present disclosure is similar to the above-mentioned image classification method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0296] Referring to Figure 12 Fig. 13 shows an architecture schematic diagram of an image classification device provided by the embodiment of the present disclosure. The device comprises a second acquisition unit 1201 and a classification unit 1202; wherein,

[0297] The second acquisition unit 1201 is configured to acquire an image to be classified.

[0298] The classification unit 1202 is configured to input the image to be classified into a classification network trained by the training method of the classification network based on the first aspect or any possible implementation manner of the first aspect, and determine the classification result of the image to be classified.

[0299] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Referring to Figure 13 Fig. 13 shows a structure schematic diagram of a computer device 1300 provided by the embodiment of the present disclosure, which comprises a processor 1301, a memory 1302, and a bus 1303. The memory 1302 is configured to store execution instructions, including an internal memory 13021 and an external memory 13022; the internal memory 13021 is also called an internal memory, which is used to temporarily store operation data in the processor 1301 and exchange data with the external memory 13022 such as a hard disk, the processor 1301 exchanges data with the external memory 13022 through the internal memory 13021, and when the computer device 1300 is running, the processor 1301 and the memory 1302 communicate through the bus 1303, so that the processor 1301 executes the following instructions:

[0300] Acquire a sample image and label information of the sample image;

[0301] Input the sample image into a binary initial feature extraction layer of a classification network to be trained, and determine a target feature map corresponding to the sample image; wherein the initial feature extraction layer comprises a channel value adjustment parameter to be trained;

[0302] Divide the target feature map into a plurality of feature units based on a preset size, and input the target feature map into a feature fusion layer, the feature fusion layer comprising a plurality of binary multi-layer perception modules, each multi-layer perception module being configured to perform feature fusion and deep feature extraction on the feature units;

[0303] The classification network is trained based on the semantic feature map output by the feature fusion layer and the label information of the sample image.

[0304] Alternatively, the processor 1301 can execute the following instructions:

[0305] An image to be classified is obtained.

[0306] The image to be classified is input into the classification network trained by the training method of the classification network described in the above embodiments, and a classification result of the image to be classified is determined.

[0307] The disclosure embodiments also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the training method of the classification network and the image classification method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0308] The disclosure embodiments also provide a computer program product carrying a program code, the instructions included in the program code being used to perform the steps of the training method of the classification network and the image classification method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0309] The computer program product can be specifically implemented by hardware, software or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0310] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. In several embodiments provided by the disclosure, it should be understood that the disclosed system, device and method can be implemented by other ways. The above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0311] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0312] In addition, each functional unit in various embodiments of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0313] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part of the prior art or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various program code storage media.

[0314] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and not to limit them, the protection scope of the present disclosure is not limited thereto, although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: any person skilled in the art in the technical range disclosed by the present disclosure, it can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and all should be covered in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method for training a classification network, characterized in that, include: Obtain the sample image and its annotation information; The sample image is input into the binarized initial feature extraction layer of the classification network to be trained to determine the target feature map corresponding to the sample image; wherein, the initial feature extraction layer includes channel value adjustment parameters to be trained; The target feature map is divided into multiple feature units based on a preset size, and the target feature map is input to a feature fusion layer. The feature fusion layer includes multiple binarized multilayer perception modules, each of which is used to perform feature fusion and deep feature extraction on the feature units. The classification network is trained based on the semantic feature map output by the feature fusion layer and the annotation information of the sample images; The step of inputting the sample image into the binarized initial feature extraction layer of the classification network to be trained, and determining the target feature map corresponding to the sample image, includes: The sample image is subjected to global average pooling to determine the first feature map corresponding to the sample image; Based on the channel value adjustment parameters, the channel values ​​of the first feature map are adjusted to determine the adjusted second feature map; A third feature map is determined based on a first threshold of a preset activation function and the difference between the corresponding channel values ​​of the sample image and the second feature map. Feature extraction is performed on the third feature map to determine the target feature map corresponding to the sample image.

2. The method according to claim 1, characterized in that, The step of extracting features from the third feature map to determine the target feature map corresponding to the sample image includes: Feature extraction is performed on the third feature map to determine the fourth feature map; Based on the channel value adjustment parameters, the feature values ​​of the first feature map are adjusted a second time to determine the adjusted fifth feature map; The fourth feature map and the fifth feature map are fused to determine the target feature map.

3. The method according to claim 1, characterized in that, The output of the Nth multilayer perceptron in the feature fusion layer is the input of the (N+1)th multilayer perceptron, the input of the first multilayer perceptron is the target feature map, and the output of the last multilayer perceptron is the semantic feature map, where N is a positive integer.

4. The method according to any one of claims 1 to 3, characterized in that, For any multilayer perceptron, the multilayer perceptron is used to perform feature fusion and deep feature extraction on the feature units of the input feature map input to the multilayer perceptron using the following method: The input feature map is subjected to binary activation processing to determine the sixth feature map; The feature units of the sixth feature map are swapped according to at least one swapping distance to obtain a swapped feature map. After feature extraction is performed on the exchanged feature map and the sixth feature map respectively, they are fused with the input feature map to obtain a fused feature map; The fused feature map is activated to obtain the output feature map of the multilayer sensing module.

5. The method according to claim 4, characterized in that, For any exchange distance, the feature exchange of the feature units of the sixth feature map to obtain an exchanged feature map includes: For any feature unit, based on the any exchange distance, determine the feature unit to be exchanged in the sixth feature map corresponding to that feature unit; Based on the value of the feature unit to be exchanged in the corresponding channel, determine the value of the feature unit in each channel after feature exchange.

6. The method according to claim 4, characterized in that, After extracting features from the exchanged feature map and the sixth feature map respectively, feature fusion is performed with the input feature map to obtain a fused feature map, including: Based on a binarized multilayer perceptron, feature extraction is performed on the exchanged feature map and the sixth feature map respectively to determine multiple deep feature maps; After normalizing the multiple deep feature maps, they are fused with the input feature map to obtain the fused feature map.

7. The method according to claim 1, characterized in that, The training of the classification network based on the semantic feature map output by the feature fusion layer and the annotation information of the sample images includes: Obtain the full-precision teacher network corresponding to the classification network to be trained; The classification network is trained based on the semantic feature map output by the feature fusion layer, the annotation information of the sample images, and the full-precision teacher network.

8. The method according to claim 7, characterized in that, The training of the classification network based on the semantic feature map output by the feature fusion layer, the annotation information of the sample images, and the full-precision teacher network includes: Based on the semantic feature map and the first prediction result of the full-precision teacher network on the sample image, a first loss value is determined; and based on the semantic feature map and the annotation information of the sample image, a second loss value is determined. The classification network is trained based on the first loss value and the second loss value.

9. The method according to claim 8, characterized in that, The determination of the first loss value based on the first prediction result of the sample image using the semantic feature map and the full-precision teacher network includes: Based on the distillation head module and the semantic feature map, a second prediction result of the classification network is determined; wherein the number of parameters in the distillation head module is the same as the number of parameters in the full-precision teacher network. The first loss value is determined based on the first prediction result and the second prediction result.

10. The method according to claim 9, characterized in that, Determining the second loss value based on the semantic feature map and the annotation information of the sample image includes: Based on the supervised head module and the semantic feature map, the third prediction result of the classification network is determined; Based on the third prediction result and the annotation information of the sample image, the second loss value is determined.

11. The method according to claim 10, characterized in that, After the classification network has been trained, the method further includes: Based on the trained distillation head module and the supervised head module, an inference head module is constructed. The inference head module is used to determine the inference result based on the output of the feature fusion layer of the trained classification network when performing network inference.

12. An image classification method, characterized in that, include: Obtain the image to be classified; The image to be classified is input into a classification network trained using the training method of any one of claims 1 to 11, and the classification result of the image to be classified is determined.

13. A training device for a classification network, characterized in that, include: The first acquisition unit is used to acquire the sample image and the annotation information of the sample image; The feature extraction unit is used to input the sample image into the binarized initial feature extraction layer of the classification network to be trained, and determine the target feature map corresponding to the sample image; wherein, the initial feature extraction layer includes channel value adjustment parameters to be trained; The feature fusion unit is used to divide the target feature map into multiple feature units based on a preset size, and input the target feature map into the feature fusion layer. The feature fusion layer includes multiple binarized multilayer perception modules, each of which is used to perform feature fusion and deep feature extraction on the feature units. The training unit is used to train the classification network based on the semantic feature map output by the feature fusion layer and the annotation information of the sample image; Specifically, the feature extraction unit is used for: The sample image is subjected to global average pooling to determine the first feature map corresponding to the sample image; Based on the channel value adjustment parameters, the channel values ​​of the first feature map are adjusted to determine the adjusted second feature map; A third feature map is determined based on a first threshold of a preset activation function and the difference between the corresponding channel values ​​of the sample image and the second feature map. Feature extraction is performed on the third feature map to determine the target feature map corresponding to the sample image.

14. An image classification device, characterized in that, include: The second acquisition unit is used to acquire the image to be classified. A classification unit is used to input the image to be classified into a classification network trained by the training method of any one of claims 1 to 11, and to determine the classification result of the image to be classified.

15. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the training method for the classification network as described in any one of claims 1 to 11, or the steps of the image classification method as described in claim 12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the training method for the classification network as described in any one of claims 1 to 11, or the steps of the image classification method as described in claim 12.

Images