Image multi-classification system based on feature remapping and training method

Abstract

This invention discloses an image multi-classification system and training method based on feature remapping. During the training of the multi-classification system, samples are simultaneously input into the multi-classification network to be trained and the pre-trained binary classification network. The classification results output by the binary classification network are used to correct the classification results output by the multi-classification network, so that the normal class probability output by the multi-classification network during training approaches the normal class probability output by the binary classification network, thus achieving feature remapping. By correcting the normal sample class results of the multi-classification network through the pre-trained binary classification network, the recognition accuracy of the multi-classification network for normal class samples can be improved, especially when there are few normal samples, a multi-classification system with high recognition accuracy for normal class samples can still be obtained.

Description

Technical Field

[0001] This invention belongs to the field of computer vision deep learning technology, and more specifically, relates to an image multi-classification system and training method based on feature remapping. Background Technology

[0002] In recent years, with continuous urban construction, the task of inspecting underground drainage pipes has become increasingly demanding. Traditional underground drainage pipe inspection requires a large amount of manpower, which faces challenges not only in terms of time but also efficiency. Deep learning-based methods for inspecting drainage pipes improve efficiency while saving labor costs.

[0003] In typical drainage pipe inspections, the accuracy of defect category samples is often prioritized over that of normal category samples, leading to a large number of normal samples being misidentified as defective. This situation arises because industrial data collection for drainage pipes focuses excessively on defect categories, resulting in a large number of defect samples and insufficient normal category samples during training. Since data collection is not a short-term process, this issue can be addressed using deep learning networks.

[0004] Because binary classification networks are simple, requiring only the distinction between normal and defective categories, when using networks of equal complexity, the simpler the task, the stronger the network's information extraction capability, and the more accurate the results. Therefore, in traditional techniques, to improve accuracy, two networks are typically trained separately: first, a binary classification network is trained to extract defect samples; then, samples identified as defects by the binary classification network are input into a multi-class classification network for specific defect category prediction. While using a binary classification network can improve the accuracy of normal category samples, the inability to guarantee 100% accuracy means that passing samples through two separate classification networks results in significant cumulative error. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an image multi-classification system and training method based on feature remapping. The purpose is to improve the accuracy of the multi-classification network in recognizing normal samples by using a binary classification network to correct the multi-classification network during the training of the multi-classification network.

[0006] To achieve the above objectives, according to one aspect of the present invention, an image multi-classification system based on feature remapping is provided, comprising:

[0007] A multi-class classification network is established by sequentially passing the input samples through convolutional layers, fully connected layers, and softmax layers. Classify the data and output the classification results. ,in, One represents the probability of the normal category, and the others represent the probabilities of different defects, where n = m - 1. ≥3;

[0008] A binary classification network, in which samples are input to the network and sequentially pass through convolutional layers, fully connected layers, and softmax layers to achieve two classifications and output the classification results. ,in, These represent the probability of the defect category and the probability of the normal category, respectively.

[0009] A feature remapping network is used to correct the classification result output by the multi-classification network based on the classification result output by the binary classification network, so that the normal class probability output by the multi-classification network during training is correct. Approaching the normal class probability output by the binary classification network This enables feature remapping.

[0010] The loss calculation module is used to calculate the training loss and inversely adjust the parameters of the multi-class network to make the loss converge.

[0011] In one embodiment, the binary classification network and the multi-classification network share a common sample input terminal.

[0012] In one embodiment, the feature remapping network includes:

[0013] The weight parameter adjustment module is used to obtain the output results of the fully connected layer of the multi-classification network. The output results of the fully connected layer of the binary classification network After concatenation, the weight parameters are output through a fully connected layer. ;

[0014] The normal category probability correction module is used to utilize weight parameters. normal category probability Revised to ,in,

[0015] ;

[0016] The loss calculation module includes a feature difference loss module and a classification loss module. The feature difference loss module is used to calculate the feature difference loss between the input features of the fully connected layer of the multi-class network and the input features of the fully connected layer of the binary network. The classification loss module is used to calculate the classification loss of the multi-class network after feature remapping. The loss calculation module uses the sum of the classification loss and the feature difference loss as the training loss.

[0017] In one embodiment, the multi-classification network includes a backbone network and a feature propagation enhancement module connected to the backbone network. The backbone network includes a plurality of convolutional layers, fully connected layers, and softmax layers connected in sequence. The feature propagation enhancement module includes:

[0018] The mid-to-high frequency feature transfer enhancement module, used to extract edge and texture information from shallow layers, is connected across the front-end convolutional layer J of the backbone network. 前 Both ends include 1×1 convolutional layers, mid-to-high frequency domain channel attention modules, and a first cross-attention module. Features in the backbone network are split into three paths upon reaching the input of the mid-to-high frequency feature transfer enhancement module. The first path passes through the front-end convolutional layer J. 前 Continuing forward, the second path sequentially passes through a 1×1 convolutional layer, a mid-to-high frequency domain channel attention module, and a 1×1 convolutional layer to extract high-frequency information of the image before merging into the main path. The third path sequentially passes through a 1×1 convolutional layer, a first cross attention module, and a 1×1 convolutional layer to extract long-distance dependencies between pixels at different locations before merging into the main path.

[0019] The low-to-mid-frequency feature transfer enhancement module, used for extracting deep semantic information, is connected across the back-end convolutional layer J of the backbone network. 后 Both ends include 1×1 convolutional layers, mid-to-low frequency domain channel attention modules, and a second cross-attention module. Features in the backbone network are split into three paths upon reaching the input of the mid-to-low frequency feature propagation enhancement module. The first path passes through the back-end convolutional layer J. 后 Continuing forward, the second path sequentially passes through a 1×1 convolutional layer, a mid-to-low frequency domain channel attention module, and a 1×1 convolutional layer to extract low-frequency information from the image before merging into the main path. The third path sequentially passes through a 1×1 convolutional layer, a second cross-attention module, and a 1×1 convolutional layer to extract long-distance dependencies between pixels at different locations before merging into the main path.

[0020] In one embodiment, a feature remapping control module is also included for determining the probability of the normal category. Is it greater than the preset value, when the probability of the normal category is... When the value exceeds a preset value, the feature remapping network is used for feature remapping. When the normal class probability... If the value is not greater than a preset value, the feature remapping network will not perform feature remapping.

[0021] According to another aspect of the present invention, a method for training an image multi-classification network based on feature remapping is provided, comprising:

[0022] Samples are input into a multi-classification network to train the network, which then implements... The system classifies data into several categories, and outputs the classification results after passing through a softmax layer in a multi-classification network. ,in, One represents the probability of the normal category, and the others represent the probabilities of different defects, where n = m - 1. ≥3;

[0023] The same samples are input into a pre-trained binary classification network, and the classification result is output after passing through the softmax layer of the binary classification network. ,in, These represent the probability of the defect category and the probability of the normal category, respectively.

[0024] The classification results output by the binary classification network are used to correct the classification results output by the multi-class classification network, so that the normal class probability output by the multi-class classification network during training is corrected. Approaching the normal class probability output by the binary classification network This enables feature remapping.

[0025] Calculate the training loss and adjust the parameters of the multi-class network in reverse. Continue training until the loss converges to the expected range, then end the training.

[0026] In one embodiment, the classification result output by the multi-classification network is corrected using the classification result output by the binary classification network, including:

[0027] Obtain the output of the fully connected layer of a multi-classification network The output results of the fully connected layer of the binary classification network After concatenation, the weight parameters are output through a fully connected layer. ;

[0028] Using weight parameters normal category probability Revised to ,in,

[0029]

[0030] Calculate the training loss, including:

[0031] Calculate the feature difference loss between the input features of the fully connected layer of the multi-class network and the input features of the fully connected layer of the binary network, as well as the classification loss of the multi-class network after feature remapping. Training ends when the sum of the classification loss and the feature difference loss converges to the expected range.

[0032] In one embodiment, before correcting the classification result output by the multi-classification network using the classification result output by the binary classification network, the probability of the normal category is first determined. Is it greater than the preset value, when the probability of the normal category is... When the value is greater than the preset value, feature remapping is performed, and the probability of the normal class is... If the value is not greater than the preset value, feature remapping will not be performed.

[0033] In one embodiment, the preset value is 0.9.

[0034] According to another aspect of the present invention, an image multi-classification method is provided, comprising:

[0035] The image is input into the trained multi-classification network to obtain the classification result, wherein the multi-classification network is trained according to the above-described image multi-classification network training method based on feature remapping.

[0036] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0037] This invention utilizes a pre-trained binary classification network during the training of a multi-classification network. Through feature remapping, the features output by the normal category samples in the multi-classification network during training are made to approximate the features output by the binary classification network. Since the binary classification network only needs to distinguish between normal and defective categories, without further classifying defects, its task is relatively simple. The simpler the task, the stronger the network's information extraction capability and the more accurate the results. Under the same sample conditions, the binary classification network has higher accuracy in recognizing normal samples than the multi-classification network. Therefore, in this application, during the training of the multi-classification network, the normal sample category results of the multi-classification network are corrected using a pre-trained binary classification network. This improves the recognition accuracy of the multi-classification network for normal samples, especially when there are few normal samples, resulting in a multi-classification network with high accuracy in recognizing normal samples. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the structure of an image multi-classification system based on feature remapping according to an embodiment;

[0039] Figure 2 This is a schematic diagram of the structure of a mid-to-high frequency feature transfer enhancement module according to one embodiment;

[0040] Figure 3 This is a schematic diagram of the structure of a low-to-medium frequency feature transfer enhancement module according to one embodiment;

[0041] Figure 4 This is a flowchart illustrating the steps of a feature remapping-based image multi-classification network training method according to one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0043] like Figure 1 The diagram shows a schematic of an image multi-classification system based on feature remapping in one embodiment. The image multi-classification system includes four functional structures: a multi-classification network, a binary classification network, a feature remapping network, and a loss calculation module.

[0044] A multi-class classification network consists of convolutional layers, fully connected layers, and a softmax layer connected sequentially. The convolutional layers extract image features, the fully connected layers calculate the logit value (unnormalized probability value) for each class, and the softmax layer normalizes the logit values ​​for all classes to obtain the final classification result, i.e., the probability of each class. Specifically, the multi-class classification network is implemented... Classification, ≥3, and output the classification result. ,in, Indicates the first The probability of a category, specifically. Representing the probability of the normal category, the others represent the probabilities of different defects, for example... This represents the probability of a Type 1 defect. This represents the probability of a Type 2 defect, and so on. Indicates the first The probability of a defect, n = m - 1.

[0045] A binary classification network consists of sequentially connected convolutional layers, fully connected layers, and a softmax layer. The functions of the convolutional layers, fully connected layers, and softmax layer are the same as in a multi-class classification network. Specifically, a multi-class classification network performs two classifications and outputs the classification results. , ],in, , These represent the probability of the defect category and the probability of the normal category, respectively. In other words, the binary classification network only distinguishes between defects and normal categories.

[0046] A feature remapping network is used to correct the classification result output by the multi-classification network based on the classification result output by the binary classification network, so that the normal class probability output by the multi-classification network during training is correct. Approaching the normal class probability output by the binary classification network This achieves feature remapping. In essence, it involves adjusting the parameters of the multi-class classification network based on the classification results output by the binary classification network, so that the multi-class network outputs a higher probability of the normal class. Approaching the normal class probability output by the binary classification network It should be noted that this application does not limit the specific method of correcting the parameters of the multi-class network based on the classification results output by the binary classification network, as long as the normal class probability output by the multi-class network can be obtained during training. Approaching the normal class probability output by the binary classification network That's all.

[0047] The loss calculation module is used to calculate the training loss and adjust the parameters of the multi-classification network in reverse to make the loss converge. Training continues until the loss converges to the expected range and then training ends.

[0048] The aforementioned image multi-classification system based on feature remapping can first train the binary classification network to obtain a trained binary classification network. Then, during the training of the multi-classification network, samples are simultaneously input into both the binary classification network and the multi-classification network. Through the feature remapping method, the features output by the normal category samples in the multi-classification network during training are made to approach the features output by the normal category samples in the binary classification network. This improves the recognition accuracy of the multi-classification network for normal category samples, especially when there are few normal samples, it can still obtain a multi-classification network with high recognition accuracy for normal category samples.

[0049] In one embodiment, the binary classification network and the multi-class classification network use the same network model, specifically the MobileNet-V3 network model, or other network models such as ShuffleNet, SqueezeNet, and Xception. Using the same network model facilitates feature remapping, allowing the features output by the normal class samples in the multi-class network to better approximate the features output by the normal class samples in the binary classification network.

[0050] In one embodiment, the binary classification network and the multi-class classification network share the same sample input terminal. When training the multi-class classification network, the structural parameters of the binary classification network need to be frozen. Then, the image input to the multi-class classification network is simultaneously input to the binary classification network through the same input terminal to predict the normal category information.

[0051] In one embodiment, see further. Figure 1 As shown, the feature remapping network includes a weight parameter adjustment module and a normal class probability correction module.

[0052] The weight parameter adjustment module is used to obtain the output results of the fully connected layer of the multi-classification network. The output results of the fully connected layer of the binary classification network [ , After concatenation, the weight parameters are output through a fully connected layer. . and correspond, After softmax normalization, the result is obtained .

[0053] Specifically, the weight parameter adjustment module may include a feature concatenation layer and multiple fully connected layers connected sequentially. The feature concatenation layer is connected to the output of the fully connected layer of the multi-class network and the output of the fully connected layer of the binary network, respectively, to obtain the weight parameters. and[ , After splicing, the splicing feature is obtained. , , , ……, , , , Then, after passing through multiple fully connected layers, the output is a 1-dimensional feature value, which is the weight parameter. .exist Figure 1 Specifically, taking m=17 as an example, after feature concatenation, a 19-dimensional feature is obtained. This 19-dimensional feature is then passed through the first fully connected layer, outputting a 17-dimensional output. Finally, it passes through the second fully connected layer, outputting weight parameters with a 1-dimensional output. .

[0054] The normal category probability correction module is used to utilize weight parameters normal category probability Revised to Specifically, the correction can be made using the following formula:

[0055] .

[0056] In this embodiment, the weighting parameter It is used to control the mapping ratio, when Use when it is 0 The original value is not mapped.

[0057] The derivation of the above modified formula is as follows:

[0058] When m=3, let the output of the binary classification network after softmax be [ (In this article) Represents the probability of defect category. The logit value output by the 3-class classification network (representing the probability of the normal class) is [ ]( and The logit values ​​represent two different types of defects. The logit value representing the normal category is obtained after softmax. The most direct way to achieve feature convergence is to make the probabilities of the corresponding categories converge. Furthermore, since softmax(x) is a monotonically increasing function, the changes in probability can be further reflected in the logit value of the network output.

[0059] Let the probability of the 3-class classification network output after feature remapping be [ The logit value is [ ] ].

[0060] Step 1: Make the probability of the normal class in the 3-class classification network approach the probability of the normal class in the 2-class classification network:

[0061]

[0062] For ease of calculation, when When, map its value The range of values ​​is specified in ;when When, map its value The range of values ​​is specified in ;when When the value remains unchanged, its value is not altered. Based on the monotonically increasing characteristic of softmax(x), we can conclude that:

[0063]

[0064]

[0065] Combining formulas (1) and (3) yields the mapped result. :

[0066]

[0067]

[0068] To unify formulas (4) and (5) and prevent the loss of the 3-class network from abruptly changing due to excessive differences in the output features of normal samples between the 3-class and 2-class networks, momentum operations are finally added to formulas (4) and (5), thus yielding an additive approximation solution:

[0069]

[0070] by ,Right now For example, (1) if ,but hour > At this time, it can be made Towards Approaching; (2) If ,but hour > This can also be used at this time. Towards Approaching; at In addition to the two cases mentioned above, although other cases may cause the features output by the 3-class classification network to deviate significantly from the features output by the 2-class classification network, based on the network connectivity in this embodiment, the weight coefficients... The loss can be adjusted through training backpropagation; that is, the degree of convergence can be changed by training the weight coefficients, and finally, the magnitude of the weight coefficients can be controlled through backpropagation of the loss. At this point, the multi-class network not only has the ability to determine whether a sample belongs to the normal class based on its own network parameters, but also has the ability to learn to determine normal class samples from binary classification networks, ultimately enabling... Towards Approaching.

[0071] exist Figure 1 Specifically, taking m=17 as an example, the output of the softmax function in the 2-classification network is [ , The logit value output by the 17-class network is out_logit: [ , , ,……, , The result after softmax is outsoft_17cls: [ , , , ……, , The mapping method is as follows:

[0072]

[0073] Formula (7) is analogous to formulas (1)-(6) to obtain formula (8):

[0074]

[0075] in To use formula (8) for The updated logit value of the normal category sample (before it has been processed by softmax).

[0076] Similarly, for m-classification networks, the correction formula is:

[0077] ;

[0078] Simultaneously, the loss calculation module includes a feature difference loss module and a classification loss module. The feature difference loss module is used to calculate the feature difference loss between the input features of the fully connected layer of the multi-class network and the input features of the fully connected layer of the binary classification network. In one embodiment, KL divergence can be used as the loss function to calculate the feature difference loss; specifically, JS divergence, Wasserstein distance, etc., can be used. Principal component analysis (PCA) is used to reduce the dimensionality of the output features, and then KL divergence is used to calculate the similarity between the two, which is then used as the loss for backpropagation, making the feature information output by the normal class samples in the multi-class network converge to the feature information output by the normal class samples in the binary classification network. The classification loss module is used to calculate the classification loss of the multi-class network after feature remapping, i.e., the difference between the predicted value and the true value; specifically, Focal loss can be used. The loss calculation module uses the sum of the classification loss and the feature difference loss as the training loss. Training ends when the training loss converges to the expected range.

[0079] In one embodiment, the image multi-classification system further includes a feature remapping control module for determining the normal class probability of the binary classification network. Does it exceed a preset value when the probability of the normal class in the binary classification network is greater than the preset value? When the value exceeds a preset value, the feature remapping network is used for feature remapping. When the normal class probability... If the value is not greater than a preset value, the feature remapping network is cut off and no feature remapping is performed. This preset value can be set flexibly. The larger the preset value, the higher the probability value of normal samples in the binary classification network used when correcting the multi-class network, and the higher the correction accuracy, but the correction frequency will be reduced. In this embodiment, after comprehensive analysis, the overall correction effect is better when the preset value is 0.9.

[0080] In one embodiment, such as Figure 1 As shown, the multi-classification network includes a backbone network and a feature delivery enhancement module connected to the backbone network. The backbone network includes multiple convolutional layers, fully connected layers and softmax layers connected in sequence. The feature delivery enhancement module includes a mid-to-high frequency (shallow edge and texture information) feature delivery enhancement module (FTSMA) and a mid-to-low frequency (deep semantic information) feature delivery enhancement module (FTSMB).

[0081] like Figure 2 As shown, the high-frequency feature transfer enhancement module (FTSMA) is connected across the front-end convolutional layer J of the backbone network. 前Both ends include 1×1 convolutional layers, mid-to-high frequency domain channel attention modules, and a first cross-attention module. Features in the backbone network are split into three paths upon reaching the input of the mid-to-high frequency feature transfer enhancement module. The first path passes through the front-end convolutional layer J. 前 Continuing forward, the second path sequentially passes through a 1×1 convolutional layer, a mid-to-high frequency domain channel attention module, and another 1×1 convolutional layer to extract high-frequency information from the image before merging into the main path. The third path sequentially passes through a 1×1 convolutional layer, a first cross-attention module, and another 1×1 convolutional layer to extract long-distance dependencies between pixels at different locations before merging into the main path. Specifically, the mid-to-high frequency feature transfer enhancement module can be located at the 1 / 3 position of the multi-classification network.

[0082] like Figure 3 As shown, the low- and mid-frequency feature propagation enhancement module is connected across the back-end convolutional layer J of the backbone network. 后 Both ends include 1×1 convolutional layers, mid-to-low frequency domain channel attention modules, and a second cross-attention module. Features in the backbone network are split into three paths upon reaching the input of the mid-to-low frequency feature propagation enhancement module. The first path passes through the back-end convolutional layer J. 后 Continuing forward, the second path sequentially passes through a 1×1 convolutional layer, a mid-to-low frequency domain channel attention module, and another 1×1 convolutional layer to extract low-frequency information from the image before merging into the main path. The third path sequentially passes through a 1×1 convolutional layer, a second cross-attention module, and another 1×1 convolutional layer to extract long-distance dependencies between pixels at different locations before merging into the main path. Specifically, the mid-to-low frequency feature transfer enhancement module can be located at the 2 / 3 position of the multi-classification network.

[0083] In this embodiment, a feature propagation enhancement module is added to improve the problem of feature loss during network propagation. First, a mid-to-high frequency feature propagation enhancement module is used to extract high-frequency information of the image and long-distance dependencies between pixels at different locations to reduce the loss of high-frequency information (texture, edges, etc.) during network feature propagation. Then, a mid-to-low frequency feature propagation enhancement module is used to extract low-frequency information of the image and long-distance dependencies between pixels at different locations to prevent the network from focusing too much on high-frequency information and losing too much low-frequency information.

[0084] The above image multi-classification system based on feature remapping utilizes the feature remapping method to make complex classification networks approach simpler classification networks, enabling the complex classification network to learn the key information extracted by the simpler classification network. A dual-branch structure is employed to learn the features of the multi-classification network, and a Feature Transmission Enhancement Module (FTSM) is used to enable the network to focus on high-frequency information during forward propagation, while also paying attention to the attention relationships between defect locations in the image, thereby improving the network's feature transmission capability.

[0085] Accordingly, this application also relates to a training method for an image multi-classification system based on feature remapping. This training method can be completed by relying on the image multi-classification system based on feature remapping described above, or it can be completed without relying on the image multi-classification system based on feature remapping described above. Figure 4 As shown, the training method includes:

[0086] Step S100: Input the samples simultaneously into the multi-class network to be trained and the trained binary network to train the multi-class network.

[0087] Specifically, the samples are input into a trained binary classification network, and the classification result is output after passing through the softmax layer of the binary classification network. , ],in, , These represent the probability of the defect category and the probability of the normal category, respectively. Samples are input into a multi-classification network to train the network, which implements... The system classifies data into several categories, and outputs the classification results after passing through a softmax layer in a multi-classification network. , , , ……, , ],in, One represents the probability of the normal category, and the others represent the probabilities of different defects, where n = m - 1. ≥3.

[0088] In this approach, a binary classification network is pre-trained. In one embodiment, a distillation network with stronger feature extraction capabilities is first trained using roughly equal numbers of defect and normal category samples randomly sampled from the dataset. Then, the same defect-normal sample data is input into the binary classification network, and the pre-trained distillation network is used to train the binary classification network. Specifically, the distillation network can use the EfficientNet-B6 network model, or a deeper model such as ResNet-101, ResNet-152, or SE-Net.

[0089] When training a multi-classification network, all data is divided into a training set and a validation set in a 7:3 ratio. The training set is used for network training, and the validation set is used to select the better training results. After setting the training data, the images to be trained are input into the two branches of the model. Optionally, depending on the amount of data, the ratio of the training set to the validation set can be 6:4, 7:3, 8:2, 9:1, etc.

[0090] In one embodiment, before training the network using image data from the samples, the image data is preprocessed to perform data augmentation, thereby improving the network's anti-interference ability and generalization ability. Optionally, data augmentation methods include geometric transformations, such as flipping, rotating, cropping, scaling, translating, and dithering; and pixel transformations, such as adding salt-and-pepper noise, Gaussian noise, performing Gaussian blur, adjusting HSV contrast, adjusting brightness and saturation, histogram equalization, and adjusting white balance.

[0091] Step S200: Use the classification results output by the binary classification network to correct the classification results output by the multi-class classification network, so that the normal class probability output by the multi-class classification network during training approaches the normal class probability output by the binary classification network, thereby achieving feature remapping.

[0092] In one specific embodiment, step S200 includes:

[0093] Step S210: Obtain the output results of the fully connected layer of the multi-classification network and the fully connected layer of the binary classification network, concatenate them, and then output the weight parameters through the fully connected layer. .

[0094] The output of the fully connected layer in a multi-class network can be represented as [ , , ,……, , The output of the fully connected layer of a binary classification network can be represented as [ , ].

[0095] Step S220: Using weight parameters normal category probability Revised to .

[0096] The correction formula is as follows:

[0097] .

[0098] The derivation of the corrected formula is described above and will not be repeated here.

[0099] During the feature remapping process, when training a multi-class network, the logit value of the normal class sample output in the trained binary network is used to supervise the logit value of the normal class sample output in the multi-class network. In the supervision process, the logit value of the normal class sample output in the multi-class network is updated according to the trainable weight coefficients.

[0100] In one embodiment, before performing step S200, the method further includes:

[0101] Determine the probability of the normal category Is it greater than the preset value, when the probability of the normal category is... When the value is greater than the preset value, feature remapping is performed, and the probability of the normal class is... If the value is not greater than a preset value, feature remapping will not be performed. Specifically, the preset value can be 0.9.

[0102] Step S300: Calculate the training loss and adjust the parameters of the multi-class network in reverse. Continue training until the loss converges to the expected range, and then end the training.

[0103] Specifically, calculating the training loss includes:

[0104] Calculate the feature difference loss between the input features of the fully connected layer of the multi-class classification network and the input features of the fully connected layer of the binary classification network, as well as the classification loss of the multi-class network after feature remapping. Training ends when the sum of the classification loss and the feature difference loss converges to the expected range. The specific choice of loss function can be found in the above introduction and will not be repeated here.

[0105] The above-described image multi-classification system training method based on feature remapping corrects the normal sample category results of the multi-classification network by using the trained binary classification network during the training of the multi-classification system. This can improve the recognition accuracy of the multi-classification network for normal category samples, especially when there are few normal samples, and still obtain a multi-classification network with high recognition accuracy for normal category samples.

[0106] Accordingly, this application also relates to an image multi-classification method, which obtains classification results by inputting an image into a trained multi-classification system, wherein the multi-classification network is trained according to the above-described training method. Specifically, the input image can be an image of a drainage pipe, and the drainage pipe defects are classified using the aforementioned multi-classification network. It is understood that this method is not limited to classifying drainage pipe defects, but can also be used for defect classification in other contexts.

[0107] The following uses images of pipeline defects in 17 categories as samples. The original image size is 480×270 pixels, totaling 25939 images. These images are input into multi-classification network models trained using different methods, and the effectiveness of the proposed solution is demonstrated by comparing the following results. Here, Baseline represents the test results of the multi-classification network without feature remapping; Baseline +FL represents the test results using focal loss (FL) as the loss function; Baseline +FL+FRM represents the multi-classification network with feature remapping (FRM); and Baseline +FL+FRM+FTEM represents the test results of the multi-classification network with feature remapping and the addition of the FTEM module.

[0108] The comparison results based on MobileNet-V3 as the basic network model are shown in Table 1 below:

[0109] Table 1: Comparison Results with MobileNet-V3 as the Base Network Model

[0110]

[0111] The comparison results using ShuffleNet-V2 as the base network model are shown in Table 2 below:

[0112] Table 2: Comparison Results When Using ShuffleNet-V2 as the Base Network Model

[0113]

[0114] The comparison results using EfficientNet-B0 as the base network model are shown in Table 3 below:

[0115] Table 3: Comparison Results When Using EfficientNet-B0 as the Base Network Model

[0116]

[0117] The above results all indicate that training a multi-class network using feature remapping can improve the accuracy of the multi-class network for normal category samples. Furthermore, adding an FTEM module to the multi-class network further enhances its performance, thus verifying the effectiveness of the proposed solution.

[0118] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image multi-classification system based on feature remapping, characterized in that, include: A multi-class classification network is established by sequentially passing the input samples through convolutional layers, fully connected layers, and softmax layers. Classify the data and output the classification results. ,in, One represents the probability of the normal category, and the others represent the probabilities of different defects, where n = m - 1. ≥3; A binary classification network, in which samples are input to the network and sequentially pass through convolutional layers, fully connected layers, and softmax layers to achieve two classifications and output the classification results. ,in, These represent the probability of the defect category and the probability of the normal category, respectively. A feature remapping network is used to correct the classification result output by the multi-classification network based on the classification result output by the binary classification network, so that the normal class probability output by the multi-classification network during training is correct. Approaching the normal class probability output by the binary classification network This enables feature remapping. The loss calculation module is used to calculate the training loss and inversely adjust the parameters of the multi-classification network to make the loss converge. The feature remapping network includes: The weight parameter adjustment module is used to obtain the output results of the fully connected layer of the multi-classification network. The output results of the fully connected layer of the binary classification network After concatenation, the weight parameters are output through a fully connected layer. ; The normal category probability correction module is used to utilize weight parameters. normal category probability Revised to ,in, ； The loss calculation module includes a feature difference loss module and a classification loss module. The feature difference loss module is used to calculate the feature difference loss between the input features of the fully connected layer of the multi-class network and the input features of the fully connected layer of the binary network. The classification loss module is used to calculate the classification loss of the multi-class network after feature remapping. The loss calculation module uses the sum of the classification loss and the feature difference loss as the training loss.

2. The image multi-classification system based on feature remapping as described in claim 1, characterized in that, The binary classification network and the multi-class classification network share the same sample input terminal.

3. The image multi-classification system based on feature remapping as described in claim 1, characterized in that, The multi-classification network includes a backbone network and a feature propagation enhancement module connected to the backbone network. The backbone network includes multiple convolutional layers, fully connected layers, and softmax layers connected in sequence. The feature propagation enhancement module includes: The mid-to-high frequency feature transfer enhancement module, used to extract edge and texture information from shallow layers, is connected across the front-end convolutional layer J of the backbone network. 前 Both ends include 1×1 convolutional layers, mid-to-high frequency domain channel attention modules, and a first cross-attention module. Features in the backbone network are split into three paths upon reaching the input of the mid-to-high frequency feature transfer enhancement module. The first path passes through the front-end convolutional layer J. 前 Continuing forward, the second path sequentially passes through a 1×1 convolutional layer, a mid-to-high frequency domain channel attention module, and a 1×1 convolutional layer to extract high-frequency information of the image before merging into the main path. The third path sequentially passes through a 1×1 convolutional layer, a first cross attention module, and a 1×1 convolutional layer to extract long-distance dependencies between pixels at different locations before merging into the main path. The low-to-mid-frequency feature transfer enhancement module, used for extracting deep semantic information, is connected across the back-end convolutional layer J of the backbone network. 后 Both ends include 1×1 convolutional layers, mid-to-low frequency domain channel attention modules, and a second cross-attention module. Features in the backbone network are split into three paths upon reaching the input of the mid-to-low frequency feature propagation enhancement module. The first path passes through the back-end convolutional layer J. 后 Continuing forward, the second path sequentially passes through a 1×1 convolutional layer, a mid-to-low frequency domain channel attention module, and a 1×1 convolutional layer to extract low-frequency information from the image before merging into the main path. The third path sequentially passes through a 1×1 convolutional layer, a second cross-attention module, and a 1×1 convolutional layer to extract long-distance dependencies between pixels at different locations before merging into the main path.

4. The image multi-classification system based on feature remapping as described in claim 1, characterized in that, It also includes a feature remapping control module for determining the probability of the normal category. Is it greater than the preset value, when the probability of the normal category is... When the value exceeds a preset value, the feature remapping network is used for feature remapping. When the normal class probability... If the value is not greater than a preset value, the feature remapping network will not perform feature remapping.

5. A training method for an image multi-classification system based on feature remapping, characterized in that, include: Samples are input into a multi-classification network to train the network, which then implements... The system classifies data into several categories, and outputs the classification results after passing through a softmax layer in a multi-classification network. ,in, One represents the probability of the normal category, and the others represent the probabilities of different defects, where n = m - 1. ≥3; The same samples are input into a pre-trained binary classification network, and the classification result is output after passing through the softmax layer of the binary classification network. ,in, These represent the probability of the defect category and the probability of the normal category, respectively. The classification results output by the binary classification network are used to correct the classification results output by the multi-class classification network, so that the normal class probability output by the multi-class classification network during training is corrected. Approaching the normal class probability output by the binary classification network This enables feature remapping. Calculate the training loss and adjust the parameters of the multi-class network in reverse. Continue training until the loss converges to the expected range, and then end the training. Correcting the classification result output by the multi-classification network using the classification result output by the binary classification network includes: Obtain the output of the fully connected layer of a multi-classification network The output results of the fully connected layer of the binary classification network After concatenation, the weight parameters are output through a fully connected layer. ; Using weight parameters normal category probability Revised to ,in, ； Calculate the training loss, including: Calculate the feature difference loss between the input features of the fully connected layer of the multi-class network and the input features of the fully connected layer of the binary network, as well as the classification loss of the multi-class network after feature remapping. Training ends when the sum of the classification loss and the feature difference loss converges to the expected range.

6. The image multi-classification system training method based on feature remapping as described in claim 5, characterized in that, Before correcting the classification result output by the multi-classification network using the classification result output by the binary classification network, the probability of the normal category is first determined. Is it greater than the preset value, when the probability of the normal category is... When the value is greater than the preset value, feature remapping is performed, and the probability of the normal class is... If the value is not greater than the preset value, feature remapping will not be performed.

7. The image multi-classification system training method based on feature remapping as described in claim 6, characterized in that, The preset value is 0.

9.

8. An image multi-classification method, characterized in that, include: The image is input into a trained multi-classification system to obtain the classification result, wherein the multi-classification system is trained by the image multi-classification system training method based on feature remapping according to any one of claims 5 to 7.

Images