Training method of classification network, image classification method and related equipment

Abstract

The application discloses a training method of a classification network, an image classification method, an electronic device and a computer readable storage medium. The training method comprises the following steps: obtaining a training set, each sample image sequence comprising a plurality of sample images of a same object to be classified; extracting a first feature of the sample image sequence by using a classification network, and obtaining a first processing result by processing the sample image sequence by using a segmentation network; classifying the sample image sequence based on the first feature and the first processing result by using the classification network, and obtaining a first classification result; and adjusting parameters of the classification network based on the first classification result. In the foregoing manner, the training effect of the classification network can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method for training a classification network, an image classification method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Image classification technology is frequently used in fields such as medicine, finance, and transportation. Currently, deep learning-based image classification technology is gaining increasing popularity. It involves inputting an image / image sequence of the object to be classified into a classification network, which then classifies the image / image sequence to determine the category of the object. Generally, to ensure classification accuracy, the classification network needs to be trained to the desired level before being used in classification applications.

[0003] However, existing training methods are not effective for training classification networks, which affects the accuracy of subsequent classification of objects. Summary of the Invention

[0004] This application provides a training method for classification networks, an image classification method, an electronic device, and a computer-readable storage medium, which can solve the problem of poor training results for existing classification networks.

[0005] To address the aforementioned technical problems, this application provides a technical solution: a training method for a classification network, an image classification method, an electronic device, and a storage medium. The training method includes: acquiring a training set comprising multiple sample image sequences, each sample image sequence including multiple sample images of the same object to be classified; extracting a first feature from the sample image sequences using a classification network; processing the sample image sequences using a segmentation network to obtain a first processing result, wherein the segmentation network is trained on the training set and, during the training process, is used to perform semantic segmentation on category-related locations corresponding to the sample image sequences; classifying the sample image sequences using the classification network based on the first feature and the first processing result to obtain a first classification result, the first classification result representing the category of the object to be classified; and adjusting the parameters of the classification network based on the first classification result.

[0006] To address the aforementioned technical problems, another technical solution adopted in this application is: providing an image classification method, the method comprising: acquiring a target image sequence, the target image sequence including multiple target images; inputting the target image sequence into a classification network to obtain a classification result of the target image sequence, the classification result of the target image sequence being used to represent the category of the target; wherein, the classification network is trained based on the aforementioned training method.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, which includes a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above-mentioned method.

[0008] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium storing program instructions that, when executed, can implement the above-mentioned method.

[0009] Through the above method, in the training process of the classification network, when the classification network classifies the sample image sequence based on the first feature, this application additionally considers the first processing result of the segmentation network on the sample image sequence. Since the segmentation network is trained using a training set, and during its training, it is used to perform semantic segmentation of the sample image sequence at category-related locations, the first processing result can assist the classification network in classifying the sample image sequence, improving the accuracy of the first classification result. Therefore, this application can improve the training effect of the classification network. Furthermore, the trained classification network can achieve end-to-end classification of the object to be classified based on the corresponding image sequence, without manual intervention. This not only improves the accuracy of the classification result but also increases the efficiency of the classification result and reduces the required resources. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the training method for the classification network in this application;

[0011] Figure 2 This is a flowchart illustrating Embodiment 2 of the training method for the classification network of this application;

[0012] Figure 3 This is a structural diagram of the classification network and segmentation network of this application;

[0013] Figure 4 This is a structural diagram of a coding block in the classification network / segmentation network of this application;

[0014] Figure 5 This is a schematic diagram illustrating the implementation principle of the recalibration layer in this application;

[0015] Figure 6 This is a schematic diagram illustrating the implementation principle of the AFF module in this application;

[0016] Figure 7 This is a schematic diagram illustrating the implementation principle of MS-CAM in this application;

[0017] Figure 8 This is a flowchart illustrating Embodiment 3 of the training method for the classification network in this application;

[0018] Figure 9 This is a schematic diagram illustrating the implementation principle of S31-S33;

[0019] Figure 10 This is another structural diagram of the classification network and segmentation network in this application;

[0020] Figure 11 This is another structural diagram of the coding block in the classification network / segmentation network of this application;

[0021] Figure 12 This is a flowchart illustrating the fourth embodiment of the training method for the classification network in this application;

[0022] Figure 13 This is a flowchart illustrating the fifth embodiment of the training method for the classification network in this application;

[0023] Figure 14 This is a flowchart illustrating an embodiment of the image classification method of this application;

[0024] Figure 15 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0025] Figure 16 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] The terms "first," "second," and "third" used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments without conflict.

[0029] Before introducing the method provided in this application, the training modes of the classification network and the segmentation network will be explained. There are several training modes; two implementation methods are listed below:

[0030] The first approach involves training the segmentation network on a training set (discussed later, not elaborated here). After the segmentation network has reached the desired training level, a joint model of the trained segmentation and classification networks is trained. The joint training process essentially trains the classification network, with the segmentation network playing a supporting role.

[0031] The second approach involves training a joint model of the segmentation and classification networks directly, without first training the segmentation network.

[0032] Furthermore, after the above training, the trained joint model is validated and tested using validation and test sets, respectively. The training, validation, and test sets are obtained by splitting the original dataset. For example, the split ratio could be 6:2:2.

[0033] Figure 1 This is a flowchart illustrating a first embodiment of the training method for the classification network in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that outcome. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment may include:

[0034] S11: Obtain the training set.

[0035] The training set consists of multiple sample image sequences, each of which includes multiple sample images of the same object to be classified.

[0036] The objects to be classified in this application include, but are not limited to, objects in the fields of medicine, transportation, and finance. For example, the objects to be classified can be biological tissues, license plates, etc. It should be noted that this application uses a classification network to classify objects based on image sequences of the objects to be classified. In other implementations, the classification network can also classify objects based on multiple images of the objects to be classified.

[0037] Each sample image sequence includes multiple sample images used to represent the object to be classified from different perspectives. For example, if the object to be classified is a license plate, multiple sample images can represent the color, character length, number of character lines, etc. Similarly, if the object to be classified is the airway of a living organism (such as a human or other living being), multiple sample images can represent the environment at different locations within the airway.

[0038] Sample image sequences can carry category labels for the objects to be classified, which serve as the criteria for classification networks to classify the sample image sequences. Sample image sequences can also carry semantic category labels, which serve as the criteria for segmentation networks to segment the sample image sequences.

[0039] To align with the application scenario, the following explanation uses the airway of a living organism as an example. The sample image sequence comprises multiple sample images corresponding to different locations of the airway in the same living organism. Each sample image includes at least a portion of the airway and its related tissues. Airway-related tissues can include tissues such as the base of the tongue and fat associated with the airway in the sample image. The sample image sequence carries an airway category label to identify the true category of the airway. Additionally, the sample image sequence may also carry a semantic category label to identify the semantic category of each pixel (e.g., airway, base of the tongue, fat, and others). Methods for obtaining the sample image sequence include, but are not limited to, CT scans. In the CT scan acquisition method, the sample image sequence is obtained by sequentially performing CT scans from the head to the lungs of a living organism, and the sample image sequence is a CT image sequence. Each image includes a cross-section of the airway and the airway-related tissues located on that cross-section.

[0040] S12: Use a classification network to extract the first feature of the sample image sequence, and use a segmentation network to process the sample image sequence to obtain the first processing result.

[0041] The segmentation network is trained on the training set, and during the training process, the segmentation network is used to perform semantic segmentation on the category-related locations corresponding to the sample image sequence.

[0042] Semantic segmentation refers to pixel-level classification of sample images to group pixels belonging to the same semantic category in the sample image sequence into one class.

[0043] When the object to be classified is the airway of a living organism, the category-related location of the airway is the airway and its related tissues. During the training of the segmentation network, the segmentation network is used to perform semantic segmentation of the airway and its related tissues corresponding to the sample image sequence. For example, if the semantic categories include airway, tongue root, fat, and others, then during the training of the segmentation network, the segmentation result of the segmentation network on the sample image sequence is the probability that each pixel in the sample image sequence belongs to the airway, tongue root, fat, and others.

[0044] It is understandable that, since the segmentation network is used to perform semantic segmentation on the category-related positions corresponding to the sample image sequence during the training process, the first processing result of the trained segmentation network on the sample image sequence covers the semantic category information of the pixels in the sample image sequence. In other words, it can indirectly reflect which pixels in the sample image sequence correspond to category-related positions and which pixels correspond to category-independent positions.

[0045] Optionally, the first processing result is the second feature of the sample image sequence. A segmentation network can be used to extract features from the sample image sequence to obtain the second feature.

[0046] Optionally, the first processing result is the first segmentation result of the sample image sequence. A segmentation network can be used to perform first semantic segmentation of the sample image sequence at category-related locations to obtain the first segmentation result.

[0047] S13: Use a classification network to classify the sample image sequence based on the first feature and the first processing result to obtain the first classification result.

[0048] The first classification result is used to represent the category of the object to be classified.

[0049] The first classification result can be the category probability of the object to be classified. The category of the object can be determined by the relationship between this category probability and a threshold. The value of this threshold can be set according to the required precision.

[0050] S14: Adjust the parameters of the classification network based on the first classification result.

[0051] The loss function (e.g., BCEloss) of the classification network can be obtained based on the difference between the first classification result and the category label of the object to be classified, and the parameters of the classification network can be adjusted based on the loss function.

[0052] S15: Repeat the above process until the preset conditions are met.

[0053] Preset conditions can include loss function convergence, training times / time reaching a threshold, training effect meeting expectations, etc.

[0054] Through the implementation of this embodiment, during the training of the classification network, when the classification network classifies the sample image sequence based on the first feature, the application additionally considers the first processing result of the segmentation network on the sample image sequence. Since the segmentation network is trained using a training set, and during its training, the segmentation network is used to perform semantic segmentation of the sample image sequence at category-related locations, the first processing result can assist the classification network in classifying the sample image sequence, improving the accuracy of the first classification result. Therefore, this application can improve the training effect of the classification network. Furthermore, the trained classification network can achieve end-to-end classification of the object to be classified based on the corresponding image sequence, without manual intervention. This not only improves the accuracy of the classification result but also increases the efficiency of the classification result and reduces the required resources.

[0055] Figure 2This is a flowchart illustrating a second embodiment of the training method for the classification network in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that outcome. Figure 2 The illustrated process sequence is limited. This embodiment is a further extension of S13, such as... Figure 2 As shown, this embodiment may include:

[0056] S21: Use a classification network to adjust the first feature based on the first processing result, so as to adjust the attention to the first feature.

[0057] The first processing result can provide attention instructions to the classification network, so that the classification network increases its attention to the classification-related features (the part corresponding to the category-related position) in the first feature and decreases its attention to the classification-irrelevant features (the part corresponding to the category-irrelevant position) in the first feature.

[0058] The adjustment method for the first feature based on the first processing result can be either performing context-adaptive convolution (CAC) on the first processing result and the first feature, or fusing the first processing result and the first feature.

[0059] S22: Use a classification network to classify the sample image sequence based on the adjusted first feature to obtain the classification result.

[0060] By adjusting the attention mentioned above, we can reduce the influence of irrelevant features on classification and increase the influence of relevant features on classification, thereby obtaining more accurate classification results.

[0061] The following description further illustrates the second embodiment based on the structure of classification networks and segmentation networks.

[0062] In one specific implementation (first network structure), the segmentation network may include at least one encoding block and at least one decoding block. The encoding / decoding blocks are used to extract features from the sample image sequence, and different encoding / decoding blocks extract features at different scales. The encoding and decoding blocks can be symmetrically arranged, so that the feature scales output by corresponding encoding and decoding blocks are the same. The semantic segmentation block is used to perform semantic segmentation on the features of the sample image sequence at category-related locations, obtaining a semantic segmentation result.

[0063] The segmentation network can process the sample image sequence sequentially using at least one encoding block and at least one decoding block to obtain the second feature.

[0064] In addition, the segmentation network may also include a semantic segmentation module, which can sequentially process the sample image sequence using at least one encoding block, at least one decoding block, and a semantic segmentation block to obtain the first segmentation result.

[0065] Furthermore, to improve the accuracy of semantic segmentation results, the segmentation network can also include an AFF module. The AFF module is used to perform weighted processing on the output of the current decoded block and the output of the corresponding encoded block, and then concatenates the weighted processing result with the output of the current decoded block to obtain the concatenated result, which serves as the input to the next block of the current decoded block.

[0066] A classification network may include at least one encoding block, an attention block, and a classification block. The encoding block of a classification network has the same structure as the encoding block of a segmentation network. The classification network can use at least one encoding block to extract a first feature from a sample image sequence, use an attention module to adjust the first feature based on the first processing result to change the attention to the first feature, and use the classification block to classify the adjusted first feature to obtain a first classification result.

[0067] In the training architecture of the above-mentioned segmentation network to assist the above-mentioned classification network, the first processing result of the segmentation network is the output of the first decoding block (second feature) or the output of the semantic segmentation block (first segmentation result). That is, the output of the first decoding block or the output of the semantic segmentation block of the segmentation network can be used to assist the training of the classification network.

[0068] Combination Figure 3 For example, Figure 3 This is a schematic diagram of the structure of the classification network and the segmentation network under this method.

[0069] like Figure 3 As shown, the segmentation network includes four sequentially connected encoding blocks (Encoderblock1-4), four decoding blocks (Decoderblock1-4), a semantic segmentation block, and AFF modules 1-4. The input of AFF module 1 is connected to Encoderblock4 and Decoderblock1, and its output is connected to Decoderblock2. The input of AFF module 2 is connected to Encoderblock3 and Decoderblock2, and its output is connected to Decoderblock3. The input of AFF module 3 is connected to Encoderblock2 and Decoderblock3, and its output is connected to Decoderblock4. The input of AFF module 4 is connected to Encoderblock1 and Decoderblock4, and its output is connected to Segmentation. The classification network includes four sequentially connected encoding blocks (Encoderblock1-4), an attention block, and a classification block. In the segmentation network, Decoderblock1 is connected to the attention block.

[0070] Thus, the classification network can obtain the first feature through Encoderblock1-4, the segmentation network can obtain the second feature through Encoderblock1-4 and Decoderblock1, and the Attention of the classification network processes the first and second features and then sends them to Classification for classification to obtain the first classification result.

[0071] The following explains the coded blocks and AFF modules:

[0072] Each encoder block may include at least one encoder sub-block, a perturbation layer, a recalibration layer, and a downsampling layer. Each encoder sub-block includes cascaded 3D convolutional layers, instance normalization layers, and activation layers. Figure 4 For example, each encoding block includes two encoding sub-blocks, a disout layer, a recalibration layer (Feature Realibration), and downsampling (maxpooling, not shown in the figure). One encoding sub-block includes a 3D convolutional layer (3D conv), an instance normalization layer (InstanceNorml3d), and an activation layer (PRelu). The channel data of the convolutional kernels of the two 3D convs gradually increases with the feature extraction stage (n).

[0073] The Decoderblock has a similar structure to the Encoderblock, but it differs from the Encoderblock in that it includes upsampling layers instead of downsampling layers, and the number of channels in its convolution kernels is equal to the number of convolutions and channels in the Encoderblock.

[0074] Perturbation layers are used to perturb features to enhance the model's generalization ability. Recalibration layers are used for information filtering. Combined... Figure 5 The implementation of the recalibration layer is explained. For feature A obtained from the perturbation layer... m The spatial mapping is integrated along the three dimensions of depth, height, and width using Z(.).

[0075]

[0076]

[0077]

[0078]

[0079] Secondly, channel reorganization is performed on the spatial mapping to obtain the channel descriptor U. m :

[0080]

[0081] Finally, the feature A obtained by the channel description operator and the perturbation layer m Multiplication yields the result of the recalibration layer:

[0082]

[0083] Where B represents the broadcast mechanism, and h j w k d i Here, f1 and f2 are the learnable parameters, f1 and f2 are the activation functions, K1 and K2 are 1*1*1 3D convolutional layers, and ⊙ represents the pointwise multiplication operation.

[0084] Combination Figure 6-7 The AFF module is illustrated by taking the weighted processing of the output X of Encoderblock4 and the output Y of Decoderblock1 by AFF module 1 as an example.

[0085] like Figure 6 As shown, AFF module 2 first adds X and Y point by point and then inputs the result into the MS-CAM block for processing. For example... Figure 7 As shown, MS-CAM consists of two parts: a local attention part and a global attention part. In the local attention part, the sum of X and Y is sequentially processed by point-wise convolution (Conv) for dimensionality reduction, ReLU transformation, and point-wise convolution (Conv) for dimensionality increase to suppress irrelevant information in the sum of X and Y and enhance classification-related information. In the global attention part, unlike the local attention part, global average pooling is performed on the sum of X and Y before point-wise convolution (Conv) for dimensionality reduction. The MS-CAM block adds the results of the local attention part and the global attention part, performs a sigmoid function, and obtains the result, i.e., the weight MS_CAM(X+Y). AFF module 2 weights X and Y based on MS_CAM(X+Y) to obtain the weighted result Z:

[0086]

[0087] The classification block consists of a 3DConv layer with a 1*1*1 convolutional kernel and a Sigmoid layer.

[0088] The semantic segmentation consists of a 3D Conv layer with a 1*1*1 convolutional kernel and a Softmax layer.

[0089] If the adjustment method mentioned in S21 above involves performing context-adaptive convolution on the first processing result and the first feature, then based on the first network structure, the above embodiment two can be further extended as follows:

[0090] Figure 8 This is a flowchart illustrating a third embodiment of the training method for the classification network in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that outcome. Figure 8 The illustrated process sequence is limited. S21 can be extended to S31-S32, and S22 can be extended to S33. For example... Figure 8 As shown, this embodiment may include:

[0091] S31: Use a classification network to obtain the attention weights corresponding to the first feature based on the first processing result.

[0092] A classification network can be used to obtain convolutional kernels that are compatible with the first processing result. The classification network can be used to convolve the first processing result based on the convolutional kernels to obtain attention weights.

[0093] S32: Use a classification network to multiply the first feature with the attention weight to obtain the multiplied feature.

[0094] S33: Use a classification network to classify the sample image sequence based on multiplication features to obtain the first classification result.

[0095] Combination Figure 9 The implementation process of S31 to S33 is illustrated with examples. For example... Figure 9 As shown, a classification network can be used to obtain convolutional kernels (CaC Kernels) that are appropriate for the first processing result (the output of the DecoderBlock, the first feature). The classification network can then convolve the first processing result based on the CaC Kernels to obtain attention weights. Finally, the classification network can multiply the first processing result and the weighting factors to obtain multiplied features (Out feature maps).

[0096] Specifically, the second feature can be convolved twice to generate features Q(B, C, H, W) and K(B, S^2, H, W); the corresponding points of Q and K are multiplied and summed, and then batch normalized to obtain CaCKernels. CaCKernels are convolved with the second feature with multiple dilation coefficients (e.g., 1, 2, 3) to obtain multiple different results; finally, the multiple different results are added to obtain weighting factors; the weighting factors are multiplied with the first feature to obtain the multiplied features.

[0097] In another specific implementation (the second network structure), the segmentation network includes at least one encoder block, a head block, at least one decoder block, and a semantic segmentation block. The encoder and decoder blocks can be configured correspondingly, with the decoder blocks of corresponding encoder blocks obtaining the same feature scale. The input to the at least one decoder block includes the input of the previous block and the input of the corresponding encoder block. The classification network includes at least one cascaded encoder block, a fusion block, and a classification block. Furthermore, a Transformer block may be included before the classification block.

[0098] The head block is used to integrate semantic information from the features output by the previous block. The structure of the head module is the same as that of the EncoderBlock, but it does not have a downsampling layer.

[0099] A fusion block is used to fuse the features output by at least one decoder block of a segmentation network and at least one coded block of a classification network. When the fusion block only needs to fuse the features output by one decoder block and one coded block, the features output by the decoder and coded blocks have the same scale, and the features output by the decoder and coded blocks can be fused directly. Alternatively, when the fusion block only needs to fuse the features output by multiple decoder blocks and multiple coded blocks, the features output by multiple decoder blocks / coded blocks form a feature pyramid, and the Feature Pyramid Theory (FPT) can be used to fuse the features output by multiple decoder blocks and multiple coded blocks.

[0100] A Transformer block can consist of two Layer Normalization layers, a multi-head self-attention layer (MSA), and a multi-layer perceptron layer (MLP).

[0101] For example, Figure 10 This is a schematic diagram of the structure of the classification network and the segmentation network under this method.

[0102] like Figure 10 As shown, the segmentation network comprises four sequentially connected encoding blocks (Encoderblock1-4), a head block, four decoding blocks (Decoderblock1-4), and a semantic segmentation block. Specifically, the output of Encoderblock4 is connected to the input of Decoderblock1; the output of Encoderblock3 is connected to the input of Decoderblock2; the output of Encoderblock2 is connected to the input of Decoderblock3; and the output of Encoderblock1 is connected to the input of Decoderblock4. The features output by Decoderblock1-4 are the second features, forming the second feature pyramid.

[0103] The classification network consists of four sequentially connected encoding blocks (Encoderblock1-4), a Fuse block, a Transformer block, and a Classification block. The features output by Encoderblock1-4 form the first feature pyramid. The input to the Fuse block is the first feature pyramid and the second feature pyramid.

[0104] Thus, the classification network can obtain the first feature pyramid through Encoderblock1-4, the segmentation network can obtain the second feature pyramid through Encoderblock1-4, the head block, and Decoderblock1-4, and the classification network's Fuse block processes the first and second feature pyramids before sending them to Classification for classification, thus obtaining the first classification result.

[0105] In this approach, the structures of the network's encoding blocks, decoding blocks, semantic segmentation blocks, and classification blocks can be the same as those mentioned in the previous approaches, or they can be different.

[0106] For example, the encoding and decoding blocks in this method differ from those mentioned earlier. (See also...) Figure 11 The encoding block consists of cascaded 3D convolutional layers (8*n), Batch Normlization layers, ReLU layers, 3D convolutional layers (16*n), Batch Normlization layers, ReLU layers, and downsampling layers (not shown in the figure). The first 3D convolutional layer has a 1×1×1 kernel, and the second 3D convolutional layer also has a 1×1×1 kernel. The input to each encoding block is added to the output of the second Batch Normlization layer to prevent gradient vanishing. The decoding block has a similar structure to the encoding block.

[0107] If the adjustment method mentioned in S21 above involves performing context-adaptive convolution on the first processing result and the first feature, then based on the second network structure, the above embodiment two can be further extended as follows:

[0108] Figure 12 This is a flowchart illustrating Embodiment 4 of the training method for the classification network of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that outcome. Figure 12 The illustrated process sequence is limited. S21 can be extended to S41, and S22 can be extended to S42. For example... Figure 12 As shown, this embodiment may include:

[0109] S41: Use a classification network to fuse the first feature and the second feature to obtain the fused feature.

[0110] Understandably, fusing the first and second features can increase the classification network's attention to classification-related features and reduce its attention to classification-irrelevant features. This allows the fused features to be applied to subsequent classifications, resulting in a more accurate first classification result.

[0111] If there are multiple first features, and these multiple first features constitute a first feature pyramid, and there are multiple second features, and these multiple second features constitute a second feature pyramid, then the FPT module of the classification network can be used to fuse the first feature pyramid and the second feature pyramid to obtain a fused feature.

[0112] Among them, FPT is a feature interaction method that is fully active in space and scale. It uses three specially designed Transformers to fuse semantic and spatial information between features in a top-down and bottom-up interaction manner.

[0113] S42: Use a classification network to classify the fused features and obtain the first classification result.

[0114] The following describes the training process of the segmentation network on the training set before obtaining the first processing result by processing the sample image sequence using the segmentation network in S12 above.

[0115] Figure 13 This is a flowchart illustrating Embodiment 5 of the training method for the classification network in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that outcome. Figure 13 The illustrated process sequence is limited. For example... Figure 13 As shown, training a segmentation network on the training set can include the following sub-steps:

[0116] S51: Use a segmentation network to perform second semantic segmentation on the sample image sequence at category-related locations to obtain the second segmentation result.

[0117] Taking the first network structure as an example, the sample image sequence can be processed sequentially using at least one encoding block, at least one decoding block, and a semantic segmentation block of the segmentation network to obtain the second segmentation result.

[0118] S52: Adjust the parameters of the segmentation network based on the second segmentation result.

[0119] The loss function of the segmentation network (e.g., OHEM+Boundary Loss) can be obtained based on the difference between the second segmentation result and the semantic label, so as to adjust the parameters of the segmentation network.

[0120] The classification network obtained through the above training can be used for classification applications.

[0121] Figure 14 This is a flowchart illustrating an embodiment of the image classification method of this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 14 The sequence of processes shown is limited.

[0122] like Figure 14 As shown, this embodiment may include:

[0123] S61: Obtain the target image sequence.

[0124] The target image sequence includes multiple target images.

[0125] Multiple target images are used to represent the target from different angles. Therefore, compared to using only a single image of the target for subsequent classification, more accurate classification results can be obtained.

[0126] S62: Input the target image sequence into the classification network to obtain the classification result of the target image sequence.

[0127] The classification results of the target image sequence are used to represent the category of the target.

[0128] The classification network can be trained using the aforementioned training methods.

[0129] For a detailed description of this embodiment, please refer to the preceding embodiments; it will not be repeated here.

[0130] Through the implementation of this embodiment, this application is able to achieve end-to-end classification of targets based on target-related image sequences through a classification network, so as to determine the category of the target.

[0131] Figure 15 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Figure 15 As shown, the electronic device may include a processor 71 and a memory 72 coupled to the processor 71.

[0132] The memory 72 stores program instructions for implementing the methods of any of the above embodiments; the processor 71 executes the program instructions stored in the memory 72 to implement the steps of the above method embodiments. The processor 71 may also be referred to as a CPU (Central Processing Unit). The processor 71 may be an integrated circuit chip with signal processing capabilities. The processor 71 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor, or the processor 71 may be any conventional processor.

[0133] Figure 16 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Figure 16 As shown, the computer-readable storage medium 80 of this application embodiment stores program instructions 81, which, when executed, implement the methods provided in the above embodiments of this application. The program instructions 81 can form a program file and be stored in the computer-readable storage medium 80 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned computer-readable storage medium 80 includes various media capable of storing program code, such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for training a classification network, characterized in that, include: A training set is obtained, which includes multiple sample image sequences, and each sample image sequence includes multiple sample images of the same object to be classified. The classification network is used to extract the first feature of the sample image sequence, and the segmentation network is used to extract the first processing result of the sample image sequence. The segmentation network is trained on the training set, and during the training of the segmentation network, the segmentation network is used to perform semantic segmentation on the category-related positions corresponding to the sample image sequence. The first processing result is the second feature of the sample image sequence, or the first processing result is the first segmentation result of the sample image sequence. The sample image sequence is classified using a classification network based on the first feature and the first processing result to obtain a first classification result, which is used to represent the category of the object to be classified. The parameters of the classification network are adjusted based on the first classification result; The segmentation network and the classification network are independent. The segmentation network sequentially includes several encoding blocks, several decoding blocks, and a semantic segmentation block. The encoding blocks and decoding blocks in the segmentation network are used for feature extraction, and the semantic segmentation block is used for semantic segmentation. The encoding blocks and decoding blocks in the segmentation network correspond one-to-one, and the feature scales output by the corresponding encoding blocks and decoding blocks are the same. The output of at least one decoding block in the segmentation network is the second feature. The classification network sequentially includes several encoding blocks, intermediate blocks, and classification blocks. The intermediate blocks are fusion blocks or attention blocks. The output of at least one encoding block in the classification network is the first feature, and the input of the intermediate block is the first feature and the second feature, or the input of the intermediate block is the first feature and the first segmentation result.

2. The training method according to claim 1, characterized in that, The step of classifying the sample image sequence using a classification network based on the first feature and the second feature to obtain a first classification result includes: The classification network is used to adjust the first feature based on the second feature, so as to adjust the attention to the first feature; The sample image sequence is classified using the adjusted first feature to obtain the first classification result.

3. The training method according to claim 2, characterized in that, The step of adjusting the first feature based on the second feature using the classification network to adjust the attention to the first feature includes: The classification network is used to obtain the attention weights corresponding to the first feature based on the second feature; The first feature is multiplied by the attention weight using the classification network to obtain the multiplied feature. The step of classifying the sample image sequence using the classification network based on the adjusted first feature to obtain the first classification result includes: The classification network is used to classify the sample image sequence based on the multiplication features to obtain the first classification result.

4. The training method according to claim 3, characterized in that, The step of using the classification network to obtain the attention weights corresponding to the first feature based on the second feature includes: The classification network is used to obtain convolutional kernels that are adapted to the second feature; The attention weights are obtained by convolving the second feature with the convolution kernel using the classification network.

5. The training method according to claim 2, characterized in that, The step of adjusting the first feature based on the second feature using the classification network to adjust the attention to the first feature includes: The first feature and the second feature are fused using the classification network to obtain a fused feature; The step of classifying the sample image sequence using the adjusted first feature to obtain the first classification result includes: The fused features are classified using the classification network to obtain the first classification result.

6. The training method according to claim 5, characterized in that, There are multiple first features, and these multiple first features constitute a first feature pyramid. There are multiple second features, and these multiple second features constitute a second feature pyramid. The step of fusing the first features and the second features using the segmentation network to obtain fused features includes: The first feature pyramid and the second feature pyramid are fused using the FPT module of the classification network to obtain the fused features.

7. The training method according to claim 1, characterized in that, The step of extracting the first feature of the sample image sequence using the classification network includes: The first feature of the sample image sequence is extracted using at least one of the coding blocks in the classification network.

8. The training method according to claim 7, characterized in that, The coding block includes at least one coding sub-block, a perturbation layer, a recalibration layer, and a downsampling layer. The coding sub-block includes a 3D convolutional layer, an instance normalization layer, and an activation layer. Extracting the first feature of the sample image sequence using at least one coding block in the classification network includes: The sample image sequence is processed sequentially using the at least one coded sub-block, the perturbation layer, the recalibration layer, and the downsampling layer to obtain the first feature.

9. The training method according to claim 1, characterized in that, Before extracting the second feature of the sample image sequence using the segmentation network, the following training steps for the segmentation network are also included: The segmentation network is used to perform second semantic segmentation on the category-related locations of the sample image sequence to obtain a second segmentation result; The parameters of the segmentation network are adjusted based on the second segmentation result.

10. An image classification method, characterized in that, include: Obtain a target image sequence, wherein the target image sequence includes multiple target images; The target image sequence is input into a classification network to obtain a classification result for the target image sequence, and the classification result of the target image sequence is used to represent the category of the target; The classification network is trained based on the training method described in any one of claims 1-9.

11. An electronic device, characterized in that, Includes a processor and a memory connected to the processor, wherein, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method of any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The storage medium stores program instructions, which, when executed, implement the method as described in any one of claims 1-10.

Images