An unbalanced target classification method, product, medium and device

Abstract

The application discloses an unbalanced target classification method, product, medium and equipment, relates to the field of class unbalanced learning of image classification, and solves the problem of poor classification effect caused by class unbalanced learning in real domain data through mixed learning of real domain data and virtual domain data, that is, through virtual-real data mixed training, improves the classification accuracy of tail classes in the real domain, thereby improving the target classification performance in the real domain data. Meanwhile, the application solves the domain offset problem through hidden layer feature enhancement in the network, improves the target classification precision in the real domain, and finally achieves the effect of greatly improving the offshore target classification accuracy in the real domain.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of class imbalance learning of image classification, and particularly relates to an unbalanced target classification method, a product, a medium and equipment. BACKGROUND

[0002] In the problem of maritime target classification, the collected data is often unbalanced in proportion, and the imbalance of classes will affect the training of the classification model. Introducing virtual class balanced data in the robust learning of the model of real class imbalance data can increase the feature diversity of each target, but due to the domain shift problem between real data and virtual data (dividing the data into real domain and virtual domain, there is a domain shift problem between the domains), which will cause confusion in model learning.

[0003] With the popularity of generative adversarial networks, some studies based on the idea of generative adversarial networks realize data generation from the source domain to the target domain to solve the domain shift problem. Among them, the domain adaptive learning method MDLT based on the mixing of real domain and virtual domain data focuses on the influence of unbalanced distribution of data on the domain adaptive learning process of the model, and effectively shares the information between the same features of different domains in the hidden layer, which also provides convenience for the method of data enhancement based on hidden layer features.

[0004] MDLT aims to learn from unbalanced data from multiple domains, solve the label imbalance within each domain, different label distribution between different domains, and finally the model can generalize to all classes of all domains for the problem of target classification of multi-domain long-tail distribution (which can be understood as the domain shift problem between real data and virtual data). Specifically, first, a domain class transfer graph is proposed to characterize the transferability (similarity) between different <domain, class> pairs. Based on this defined transferability, the performance of the model on the MDLT task is directly determined. Then BoDA is proposed to improve the performance of the model on the MDLT problem.

[0005] Although MDLT provides convenience for the method of data enhancement based on hidden layer features when facing the task of target recognition of multi-domain long-tail distribution, it does not optimize the different distribution of classes in different domains in depth, and only alleviates the bias of the model by simply weighting the loss.

[0006] In summary, the existing methods such as the MDLT method mainly optimize from the perspective of the loss function. For the MDLT method which only uses simple weighting balance, using it for real domain maritime target classification cannot greatly improve the accuracy of real domain maritime target classification. SUMMARY

[0007] The application aims to provide an unbalanced target classification method, product, medium and equipment, which can greatly improve the real domain maritime target classification accuracy.

[0008] To achieve the above-mentioned purpose, the application provides the following solutions.

[0009] In one aspect, the application provides an unbalanced target classification method, comprising:

[0010] Step S1: obtaining a virtual domain training data set and a real domain training data set corresponding to the virtual domain training data set; the virtual domain training data set and the real domain training data set each comprise a plurality of maritime target image sample sets of different categories; each maritime target image sample set of a category comprises a plurality of maritime target image samples;

[0011] Step S2: obtaining a batch training sample of the real domain and a batch training sample of the virtual domain; the batch training sample of the real domain is obtained by sampling a batch of maritime target image samples from the real domain training data set; the batch training sample of the virtual domain is obtained by sampling a batch of maritime target image samples from the virtual domain training data set;

[0012] Step S3: using the batch training sample of the real domain and the batch training sample of the virtual domain, and passing through a feature extraction backbone network part of a classification model, respectively obtaining real domain hidden layer features and virtual domain hidden layer features;

[0013] Step S4: based on the real domain hidden layer features, calculating real domain class hidden layer feature mean and covariance at a current training time, and updating the real domain class hidden layer feature mean and covariance; based on the virtual domain hidden layer features, calculating virtual domain class hidden layer feature mean at the current training time, and updating the virtual domain class hidden layer feature mean;

[0014] Step S5: using all maritime target image samples in the real domain training data set to calculate a real domain class confusion matrix, and based on the class confusion matrix, calculating real domain class hidden layer feature correction mean and correction covariance; the real domain class hidden layer feature correction mean and correction covariance are respectively used to correct the real domain class hidden layer feature mean and covariance;

[0015] Step S6: based on the real domain class hidden layer feature correction mean and correction covariance, performing hidden layer feature enhancement on features from tail class samples in the batch training sample of the real domain, generating augmented features, and using the augmented features and the real domain hidden layer features to obtain balanced real domain hidden layer features;

[0016] Step S7: based on the balanced real domain hidden layer features, updating the real domain class hidden layer feature mean, and obtaining updated real domain class hidden layer feature mean;

[0017] Step S8: calculating a domain adaptive training loss based on the virtual domain hidden layer features, the balanced real domain hidden layer features, the virtual domain class hidden layer feature mean values and the updated real domain class hidden layer feature mean values, adding a supervised learning loss based on the domain adaptive training loss to obtain a total training loss, and optimizing the classification model in the training process by using the total training loss to obtain an optimized classification model;

[0018] Step S9: determining whether the total training loss is reduced to a set threshold value.

[0019] If not, returning to step S2.

[0020] If yes, performing step S10: inputting a to-be-classified marine target image into the optimized classification model, and outputting a class corresponding to the to-be-classified marine target image by using the optimized classification model; the to-be-classified marine target image is a real marine target image obtained in a real scene.

[0021] Optionally, the virtual domain training data set is a class-balanced distribution virtual domain marine target classification training data set; and the real domain training data set is a class-unbalanced distribution real domain marine target classification training data set.

[0022] The total number of classes and the number of class types of the marine target image sample set in the class-balanced distribution virtual domain marine target classification training data set are the same as the total number of classes and the number of class types of the marine target image sample set in the class-unbalanced distribution real domain marine target classification training data set.

[0023] The number of marine target image samples in the marine target image sample set of different classes in the class-balanced distribution virtual domain marine target classification training data set is balanced; and the number of marine target image samples in the marine target image sample set of different classes in the class-unbalanced distribution real domain marine target classification training data set is unbalanced.

[0024] Optionally, the batch training sample of the real domain is obtained by sampling a batch of marine target image samples from the real domain training data set using an instance-balanced sampling distribution; and the batch training sample of the virtual domain is obtained by sampling a batch of marine target image samples from the virtual domain training data set using an instance-balanced sampling distribution.

[0025] Optionally, the classification model is a domain adaptive classification model.

[0026] Optionally, the balanced real domain hidden layer features are obtained by using the augmented features and the real domain hidden layer features, and specifically include:

[0027] obtain a union of the augmented feature and the real domain hidden layer feature, and take the union of the augmented feature and the real domain hidden layer feature as the balanced real domain hidden layer feature.

[0028] Optionally, the real domain class hidden layer feature mean is updated based on the balanced real domain hidden layer feature to obtain an updated real domain class hidden layer feature mean.

[0029] The real domain class hidden layer feature mean is calculated based on the balanced real domain hidden layer feature, and the real domain class hidden layer feature mean is updated to obtain an updated real domain class hidden layer feature mean.

[0030] In another aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the unbalanced target classification method.

[0031] In another aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the unbalanced target classification method.

[0032] In still another aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the unbalanced target classification method.

[0033] According to the specific embodiments of the present application, the following technical effects are achieved:

[0034] The unbalanced target classification method, product, medium and device disclosed by the present application solve the problem of poor classification effect caused by class imbalance in real domain data through mixed learning of real domain data and virtual domain data, i.e., through mixed training of virtual and real data, improve the classification accuracy of tail classes in the real domain, thereby improving the target classification performance in the real domain data. At the same time, the present application solves the domain shift problem through hidden layer feature enhancement in the network and improves the target classification precision in the real domain, ultimately achieving the effect of greatly improving the classification accuracy of real domain marine targets. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0036] Figure 1 The flowchart of the unbalanced target classification method provided by Embodiment One of the present application;

[0037] Figure 2 A schematic diagram of the number of category samples of the real and virtual domain maritime target classification dataset;

[0038] Figure 3 A flow chart of an unbalanced target classification method based on virtual-real data hybrid learning provided by the present application;

[0039] Figure 4 A schematic diagram of a category unbalanced classification model based on virtual-real inter-class feature hybrid enhancement provided by the present application;

[0040] Figure 5 A schematic diagram of the category sample distribution of the self-constructed virtual-real hybrid maritime target classification dataset provided by the present application;

[0041] Figure 6 A schematic diagram of the MNIST-M data and dataset category distribution;

[0042] Figure 7 A schematic diagram of the SVHN data and balanced sampling dataset category distribution. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0044] The purpose of the present application is to provide an unbalanced target classification method, product, medium and equipment, which can greatly improve the accuracy of real domain maritime target classification.

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0046] Embodiment one

[0047] As shown in the figure, the unbalanced target classification method provided by the present embodiment one comprises: Figure 1

[0048] Step S1: obtaining a virtual domain training dataset and a real domain training dataset corresponding to the virtual domain training dataset; the virtual domain training dataset and the real domain training dataset each comprise a plurality of maritime target image sample sets of different categories; each category of maritime target image sample set comprises a plurality of maritime target image samples.

[0049] ​In the step S1, the virtual domain training data set is a virtual domain maritime target classification training data set with balanced distribution of categories, and the real domain training data set is a real domain maritime target classification training data set with unbalanced distribution of categories. The total number of categories and the number of category types of the maritime target image sample set in the virtual domain maritime target classification training data set with balanced distribution of categories are the same as those of the maritime target image sample set in the real domain maritime target classification training data set with unbalanced distribution of categories. The number of maritime target image samples in the maritime target image sample set of different categories in the virtual domain maritime target classification training data set with balanced distribution of categories is balanced, and the number of maritime target image samples in the maritime target image sample set of different categories in the real domain maritime target classification training data set with unbalanced distribution of categories is unbalanced.

[0050] In step S2, the batch training sample of the real domain is obtained by sampling a batch of maritime target image samples from the real domain training data set, and the batch training sample of the virtual domain is obtained by sampling a batch of maritime target image samples from the virtual domain training data set.

[0051] In the step S2, the batch training sample of the real domain is obtained by sampling a batch of maritime target image samples from the real domain training data set using the instance balanced sampling distribution, and the batch training sample of the virtual domain is obtained by sampling a batch of maritime target image samples from the virtual domain training data set using the instance balanced sampling distribution.

[0052] In step S3, the batch training sample of the real domain and the batch training sample of the virtual domain are used to obtain the real domain hidden layer feature and the virtual domain hidden layer feature through the feature extraction backbone network part of the classification model.

[0053] In the step S3, the classification model is a domain self-adaptive classification model.

[0054] In step S4, the real domain class hidden layer feature mean and covariance at the current training time are calculated based on the real domain hidden layer feature, and the real domain class hidden layer feature mean and covariance are updated; the virtual domain class hidden layer feature mean at the current training time is calculated based on the virtual domain hidden layer feature, and the virtual domain class hidden layer feature mean is updated.

[0055] In step S5, the real domain class confusion matrix is calculated using all the maritime target image samples in the real domain training data set, and the real domain class hidden layer feature correction mean and correction covariance are calculated based on the class confusion matrix; the real domain class hidden layer feature correction mean and correction covariance are used to correct the real domain class hidden layer feature mean and covariance, respectively.

[0056] Step S6: Based on the mean and the corrected covariance of each type of hidden layer feature in the real domain, the hidden layer feature of the feature from the tail class sample in the batch training sample in the real domain is enhanced to generate augmented features, and the balanced hidden layer features in the real domain are obtained by using the augmented features and the hidden layer features in the real domain.

[0057] In this step S6, the balanced hidden layer features in the real domain are obtained by using the augmented features and the hidden layer features in the real domain, which specifically includes:

[0058] The union of the augmented features and the hidden layer features in the real domain is obtained, and the union of the augmented features and the hidden layer features in the real domain is taken as the balanced hidden layer features in the real domain.

[0059] Step S7: Based on the balanced hidden layer features in the real domain, the mean of each type of hidden layer feature in the real domain is updated to obtain the updated mean of each type of hidden layer feature in the real domain.

[0060] In this step S7, based on the balanced hidden layer features in the real domain, the mean of each type of hidden layer feature in the real domain is updated to obtain the updated mean of each type of hidden layer feature in the real domain, which specifically includes:

[0061] Based on the balanced hidden layer features in the real domain, the mean of each type of hidden layer feature in the real domain is calculated, and the mean of each type of hidden layer feature in the real domain is updated to obtain the updated mean of each type of hidden layer feature in the real domain.

[0062] Step S8: Based on the virtual domain hidden layer feature, the balanced hidden layer feature in the real domain, the mean of each type of hidden layer feature in the virtual domain, and the updated mean of each type of hidden layer feature in the real domain, the domain adaptive training loss is calculated, the supervised learning loss is added based on the domain adaptive training loss, the total training loss is obtained, the classification model in the training process is optimized by using the total training loss, and the optimized classification model is obtained.

[0063] Step S9: Determine whether the total training loss is reduced to a set threshold.

[0064] If the output result of step S9 is no, return to step S2.

[0065] If the output result of step S9 is yes, execute step S10: input the to-be-classified marine target image into the optimized classification model, and output the corresponding class of the to-be-classified marine target image by using the optimized classification model; the to-be-classified marine target image is a real marine target image obtained in a real scene.

[0066] The technical scheme of the present application is described below with one specific embodiment:

[0067] The application provides an unbalanced target classification method, which is an unbalanced target classification method based on virtual-real data mixed learning, and is different from the most popular method based on a generative adversarial network. The method mainly improves the target classification performance in the real domain data through mixed learning of real domain data and virtual domain data. For a classification model, to improve the classification performance of the classification model for targets in different domains, there are various ways to improve, which can be through changing the training network structure or optimizing the loss function. The existing method (for example, MDLT) mainly optimizes from the perspective of the loss function. Using the existing method (for example, MDLT) to classify marine targets in the real domain cannot greatly improve the accuracy of marine target classification in the real domain. The application solves the domain shift problem through implicit feature enhancement (i.e., hidden layer feature enhancement) in the network, improves the target classification accuracy in the real domain, and finally achieves the effect of greatly improving the accuracy of marine target classification in the real domain.

[0068] In order to achieve the goal of further enhancing the robustness of model learning with balanced virtual data, the application needs to study effective cross-domain learning of the model under the condition of mixed multi-domain data and further improve its performance under the test condition of balanced distribution of real categories. The general idea is as follows:

[0069] First, in view of the domain shift problem existing between real data and virtual data, a domain adaptive classification model is introduced, a transferability matrix between classes in the real domain and the virtual domain is calculated by defining a distance function, so as to reduce the distance between the same classes in different domains and increase the distance between different classes in the same domain and different domains, and realize cross-domain learning of the model. Domain adaptive classification learning is an important content in transfer learning, which is mainly to map data with different feature distributions in the source domain and the target domain to a feature space, so that the distance in the space is as close as possible, so that the model can realize cross-domain classification after training.

[0070] Secondly, the implicit feature enhancement method is introduced to realize the balanced and uniform distribution of the number of class samples of the training data in the real domain and the virtual domain from the feature level. Data implicit feature enhancement is to enhance the feature vector obtained after feature extraction of sample data by the backbone network to generate a new feature vector, which can be regarded as resampling in the feature space, increasing the training samples of the model to enhance the generalization ability of the model.

[0071] Finally, before calculating the transferability matrix, the tail class features of the real data augmented based on the implicit feature enhancement method are introduced, so that the data class distribution of the real domain and the virtual domain is consistent, thereby realizing effective virtual-real data mixed learning and further improving the performance of the model in the test. For example Figure 2The object to be achieved by the present application is to solve the problem of poor classification effect caused by class imbalance in the real domain data by virtual-real data mixed training, and to improve the classification accuracy of the tail classes in the real domain. The present application solves the domain shift problem through implicit feature enhancement in the network, and improves the target classification precision in the real domain.

[0072] The object of the present application is to enable the model to effectively learn the data feature diversity in the balanced virtual domain and migrate to the unbalanced real domain, and to improve the application effect of the model in the way of virtual-real data mixed learning. Therefore, an unbalanced target classification method based on virtual-real data mixed learning is proposed, which uses a domain adaptive method to solve the domain shift problem caused by the introduction of a balanced virtual domain, and enhances the generalization performance of the model in the unbalanced real domain. At the same time, an implicit layer feature enhancement method is used to improve the feature diversity of the tail classes in the real domain, and to solve the problem of difficult domain adaptive learning caused by the inconsistency of class distribution between the real domain and the virtual domain. The process of the unbalanced target classification method based on virtual-real data mixed learning proposed by the present application is shown in Figure 3 . Referring to Figure 3 , the unbalanced target classification method based on virtual-real data mixed learning of the present application comprises the following steps:

[0073] S1: Construct a virtual marine simulation scene to collect virtual domain marine target classification training data set with balanced class distribution , the real domain marine target classification training data set with unbalanced class distribution is , and the total number of target classes is .

[0074] The background of the present application is the problem of marine target classification, so the simulation scene of the present application is mainly a marine scene. Of course, if there is a demand for target classification in other fields, the classification precision can also be improved by using simulation data and the method of the present application.

[0075] S2: Using instance balancing sampling to obtain a batch of training samples and from the real domain training set and the virtual domain training set .

[0076] This step S2 describes the single optimization process of the marine target classification model (domain adaptive classification model). First, is a batch of training samples sampled from , is sampled from , represents the size of the batch Batch-Size. Among them, and These refer to the real-domain and virtual-domain target image data selected in the same batch, respectively. These are image labels (i.e., the true category labels of the target). and All of them have This is because the selected pair of samples have the same target category, which makes it easier to perform feature enhancement on subsequent targets of the same type.

[0077] S3: Using the model backbone network right and After feature extraction, the hidden layer features of the real domain are obtained. and virtual domain hidden layer features .

[0078] Step S3 utilizes the training samples obtained in step S2. and The feature extraction backbone network part of the model (i.e., the model training network) The hidden features of the real domain were obtained respectively. With virtual domain hidden layer features .in, yes Extracted features, Refers to the real-domain hidden layer features extracted from real samples. yes Extracted features, Refers to the virtual domain hidden layer features extracted from virtual samples.

[0079] S4: Based on Real-Domain Hidden Layer Features and virtual domain hidden layer features Calculate the mean values ​​of various hidden features in the real and virtual domains and the covariance values ​​of various hidden features in the real domain at the current training time, and update the mean values ​​of various hidden features in the real and virtual domains and the covariance values ​​of various hidden features in the real domain.

[0080] Step S4 is based on real-domain hidden layer features. and virtual domain hidden layer features Calculate the current time Mean values ​​of various hidden features in virtual and real domains Covariance of various hidden layer features in the real domain and based on The value of each time point is updated globally for each category. and .in, include and , Indicates the current time Mean values ​​of various hidden layer features in the real domain. denotes the current time virtual domain class-specific hidden layer feature mean. denotes the input sample, denotes that the sample belongs to the real domain and the virtual domain sample set.

[0081] S5: Calculate the real domain class confusion matrix after classifying all training data in the real domain training set , and calculate the real domain class-specific hidden layer feature correction mean and correction covariance.

[0082] This step S5 uses all training data in the real domain maritime target classification training data set obtained in step S1 to calculate the real domain class confusion matrix , wherein the element in the matrix can be represented as:

[0083]

[0084] wherein, is an indicator function, and outputs 1 when the condition in the parentheses is true, and outputs 0 when it is false. denotes the class prediction of the input feature, denotes the number of all training samples of the class , denotes the real space with a confusion matrix size of CxC, that is, the total number of classes x the total number of classes, and both denote a specific target class.

[0085] Based on the class confusion matrix , the real domain class-specific hidden layer feature correction mean and the correction covariance are calculated:

[0086]

[0087]

[0088] wherein, denotes the feature mean of the class , denotes the number of samples in the training data that belong to the class , is an indicator function, which takes 1 when , and 0 otherwise. denotes the specific class correction covariance, that is, the covariance matrix of the hidden layer features of the class . denotes the set of class-specific correction covariances. This indicates the correction. Not used alone. and Each represents a different dimension. , , , The top right corner and Both represent dimensional transformations. Represents the modified covariance matrix The dimension is .

[0089] S6: Correcting the mean and covariance of training samples in the real domain based on various hidden layer features. Features from tail class samples Perform hidden layer feature enhancement to generate augmented features ,Will and Merging yields the equilibrium real-domain hidden layer features. .

[0090] Step S6 corrects the mean and covariance of the real-domain dataset based on various hidden features in the real-domain domain. Features from tail class samples Perform hidden layer feature enhancement to generate augmented features And define the hidden features of the real domain after equilibrium as .

[0091]

[0092] in, Indicates random from A tail-class sample feature obtained from sampling, and For hyperparameters, This indicates the number of samples from the head class in a training batch. This represents the number of samples in the tail class. , only One of the elements, that is, one of the tail-class enhancement features. It is a set symbol, indicating Belonging to the latter set, Represents the true category label Vector summation.

[0093] S7: Based on the Real-Domain Hidden Layer Features After Equilibrium Update the mean values ​​of all hidden layer features in the real domain.

[0094] The step S7 is based on the balanced real domain hidden layer features Update the real domain global class hidden layer feature mean .

[0095] S8: Calculate adaptive weight through the number of samples of different categories in the real domain and the virtual domain , obtain the domain adaptive training loss of the learning model , and calculate the training loss of the model based on the domain adaptive learning weight . .

[0096] The step S8 calculates the adaptive weight through the number of samples of different categories in the real domain and the virtual domain , thereby obtaining the domain adaptive training loss of the learning model :

[0097]

[0098] Among them, , , represents an arbitrary feature vector distance function added with the reciprocal of the category sample as the weight , represents the set of all target feature-label pairs in the real domain and the virtual domain, represents the real domain target feature-category label pair, represents one of the sample features, is a variable parameter, which is controlled by the number of different categories of samples in the real domain and the virtual domain in the training sample, is the specific class hidden layer feature mean, represents the mixed class hidden layer feature mean of the virtual domain and the real domain, and The difference is that is used alone, which represents a class, represents a set, represents the real domain target feature-category label pair, represents the hidden layer feature mean of the category in the label pair, and Can be understood as local and global.

[0099] On the basis of the domain adaptive loss , add the supervised learning loss , and the definition of the training loss is:

[0100]

[0101]

[0102] wherein, represents all hidden layer features corresponding class label, is the weight for domain adaptation learning, represents real target feature vector, represents predicted real target class, is the real target class label, represents the cross-entropy loss of predicted value and real value, represents the selection and elements in and , the cross-entropy of predicted value and real value . Based on the learning rate γ, the network weight is optimized using the stochastic gradient descent, and the learning process is repeated to update the network.

[0103] After updating the network, a trained and optimized model is obtained, and the input of the model is a real domain target image when the model is used, and the output is a target classification result.

[0104] The imbalance target classification method based on virtual-real data hybrid learning of the application corresponds to a class imbalance classification model structure based on virtual-real inter-class feature hybrid enhancement, as shown in Figure 4 .

[0105] Below, taking the imbalance real domain and balance virtual domain mixed sea target classification task as a specific application example, the imbalance target classification method based on virtual-real data hybrid learning proposed by the application is analyzed:

[0106] Based on the constructed virtual sea simulation scene, the virtual domain balanced sea target classification data set and the imbalance sea target classification data set of 6 categories are collected together to form the imbalance real domain and balance virtual domain mixed sea target classification data set. The data of the imbalance real domain includes six categories of sea target data, and the target categories include bulk carriers, container ships, cruise ships, sailing ships, other ships and islands. All data are first manually labeled as target detection data set, and then the data are cut based on the labeled frame to obtain the six categories of sea target classification data set. The data set contains a total of 26448 sea target images, of which 1000 samples are randomly selected in each category to form a test set, and the remaining data are sampled according to the long-tail imbalance distribution with an imbalance rate of 0.01 to obtain a training set. The balanced virtual domain data set is obtained after class balanced data collection in the virtual sea simulation scene, and the number of instances of each target is 1000. The class distribution of the sea target classification data set is shown in Figure 5 .

[0107] The unbalanced domain and balanced domain mixed classification dataset is constructed based on public datasets MNIST-M and SVHN. MNIST-M with simple background is set as the unbalanced real domain, and SVHN with complex and diverse background is set as the balanced virtual domain. MNIST-M is a color patch extracted from a color photo in the BSDS500 dataset to the original MNIST data. Specifically, two images are randomly selected from the MNIST and BSDS500 datasets , , and then the color of the number part in the color photo is extracted based on the MNIST image as a mask . The MNIST image is a single channel in black and white, and the color image is a three-channel, so the three channels are masked respectively. The original class sample number distribution of the MNIST-M data and the class sample number after three long-tail unbalanced sampling with a sampling rate of are shown in Figure 6 . The SVHN dataset is a real-world image dataset containing numbers 0-9, which is segmented from the house number in the Google Street View image, and contains 73257 training images. The SVHN data based on balanced sampling is shown in Figure 7 . The maximum number of samples of each class of MNIST-M and SVHN is 4500, and the size of the image size is . The test set is balanced sampled from the original test set of MNIST-M, and each class contains 800 test samples. In the present application, the classification of marine targets is mainly introduced, and the real domain and virtual domain data mixed training are used to improve the classification performance of the real domain target, that is, and . The dataset constructed based on MNIST-M and SVHN is shown in Figure 6 , and the handwritten digits can also be divided into different domains under different backgrounds, and the digits 0-9 also exhibit unbalanced characteristics under different domains. Therefore, the effectiveness of the domain transfer classification effect of the method can also be verified by the dataset.

[0108] In this task, first, the domain adaptive classification model is introduced to enhance the generalization between samples of the same class and different domains, and to realize cross-learning of the basic model. Secondly, implicit feature enhancement is used to augment the features of real domain data samples, share the feature diversity between samples of the same domain and different classes, realize effective virtual-real data mixed learning, and improve the performance of the model in the balanced test environment.

[0109] In the method of the application, several classic classification methods are selected as baseline algorithms: CE, CE-Merge, RS, RISDA and MDLT. The experimental index is Top-1 accuracy, and the higher the accuracy is, the better. In the method of learning based on the single-reality-domain offshore target classification training set, RISDA has certain improvement compared with the basic methods CE and RS, and is similar to the method under the decoupling learning model, which reflects the effectiveness of the hidden layer feature enhancement method. The domain adaptive learning method MDLT based on the mixed data of the reality domain and the virtual domain and the method of the application have certain improvement compared with the basic methods CE and RS, which reflects the value of introducing the balanced virtual domain. For the MDLT method using simple weighting balance, its precision on the test set is similar to the decoupling learning method. The CE-Merge method has a weak improvement compared with the baseline method, which on the one hand reflects that the domain shift phenomenon between the virtual data collected based on the virtual simulation scene in the method of the application and the features of the real data in the real scene is weak, and on the other hand shows that the improvement of the model performance using only this simple data augmentation is very limited. The comparison experiment results of the virtual-real mixed offshore target classification data set are shown in Table 1, and Ours in Table 1 represents the method of the application.

[0110] Table 1 Comparison experiment results of the virtual-real mixed offshore target classification data set

[0111]

[0112] Experiments are carried out on each unbalanced MNIST-M training set and balanced SVHN training set, and the Top-1 accuracy of each method is verified on the MNIST-M test set. Among them, the baseline model trained with cross-entropy loss (CE) performs very poorly on severely unbalanced data sets. The MDLT method based on transfer learning can only perform well on relatively balanced training data sets, indicating that the class imbalance problem has a significant impact on transfer learning or domain adaptive learning. In the experimental results, the CE-Merge method of directly training the mixed data and the RISDA-Merge method based on implicit feature enhancement have a significant decrease in accuracy on the test set. This shows that there is a certain domain shift between the feature distributions of data in different domains, and ordinary model learning methods cannot effectively handle this domain shift problem. Therefore, directly learning based on mixed multi-domain data will damage the robustness of the model, and its precision will be significantly lower than that of the model trained only using unbalanced data. The comparison experiment results of the unbalanced MNIST-M data set are shown in Table 2, and Ours in Table 2 represents the method of the application.

[0113] Table 2 Comparison experiment results of the unbalanced MNIST-M data set

[0114]

[0115] Tables 1 and 2 demonstrate the performance of each algorithm model using the Top-1 accuracy metric. The method proposed in this invention outperforms the state-of-the-art MDLT algorithm in overall performance. The proposed method, by training the model using a hybrid real-data learning approach, simultaneously addresses the class imbalance of real-world data and the domain offset between real-world and virtual data, enhancing its generalization ability in real-world scenarios and thus exhibiting superior performance.

[0116] The method proposed in this invention can achieve effective mixed learning of virtual and real data for the model. However, due to the domain shift between the virtual and real environments, training the model using mixed learning requires simultaneously addressing the class imbalance of the real data and the domain shift between the real data and the virtual data. This method was trained on multiple datasets with a mixture of imbalanced and balanced domains, and its accuracy on balanced test sets is higher than that of comparative methods, demonstrating the superiority of mixed learning and the application value of virtual simulation scenarios.

[0117] Hidden feature enhancement is a concept proposed in this invention. Existing data focuses only on sharing similar feature information across different domains. This sharing mechanism provides a research approach for data enhancement in this invention. This invention improves the accuracy of real-world target classification through feature enhancement, with the application scenario being maritime target classification. Currently, domain transfer classification research is generally based on public datasets, such as the MNIST-M dataset mentioned in this invention, which improves the recognition of handwritten digits across different domains. This invention, however, utilizes domain transfer and hidden feature enhancement methods, combined with hybrid virtual-real learning, to improve the accuracy of real-world maritime target classification. This is a pioneering work in this field, solving a real-world problem and demonstrating practicality.

[0118] Example 2

[0119] A computer program product includes a computer program that, when executed by a processor, implements the imbalanced target classification method of Embodiment 1.

[0120] Example 3

[0121] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the imbalanced target classification method of Embodiment 1.

[0122] Example 4

[0123] A computer device includes a processor, a memory, an input / output (I / O) interface, and a communication interface. The processor, the memory, and the I / O interface are connected through a system bus, and the communication interface is connected to the system bus through the I / O interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store to-be-processed transactions. The I / O interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program, when executed by the processor, can implement the unbalanced target classification method in Embodiment One.

[0124] The principles and implementation manners of the present application are described herein by applying specific examples, and the above description of the examples is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will have changes. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. An imbalanced target classification method, characterized in that, include: Step S1: Obtain the virtual domain training dataset and the real domain training dataset corresponding to the virtual domain training dataset; Both the virtual domain training dataset and the real domain training dataset include different categories of marine target image sample sets; each category of marine target image sample set includes multiple marine target image samples. Step S2: Obtain batch training samples in the real domain and batch training samples in the virtual domain; the batch training samples in the real domain are obtained by sampling a batch of marine target image samples from the real domain training dataset; the batch training samples in the virtual domain are obtained by sampling a batch of marine target image samples from the virtual domain training dataset. Step S3: Using the batch training samples of the real domain and the batch training samples of the virtual domain, the hidden features of the real domain and the hidden features of the virtual domain are obtained respectively through the feature extraction backbone network of the classification model. Step S4: Calculate the mean and covariance of various hidden features in the real domain at the current training time based on the real domain hidden features, and update the mean and covariance of various hidden features in the real domain; calculate the mean of various hidden features in the virtual domain at the current training time based on the virtual domain hidden features, and update the mean of various hidden features in the virtual domain. Step S5: Calculate the class confusion matrix of the real domain using all maritime target image samples in the real domain training dataset, and calculate the corrected mean and corrected covariance of various hidden features in the real domain based on the class confusion matrix; the corrected mean and corrected covariance of various hidden features in the real domain are used to correct the mean and covariance of various hidden features in the real domain, respectively. Step S6: Based on the corrected mean and corrected covariance of various hidden layer features in the real domain, perform hidden layer feature enhancement on the features from the tail class samples in the batch training samples of the real domain to generate augmented features, and use the augmented features and the real domain hidden layer features to obtain the balanced real domain hidden layer features. Step S7: Update the mean values ​​of various hidden features in the real domain based on the balanced hidden features of the real domain, and obtain the updated mean values ​​of various hidden features in the real domain. Step S8: Calculate the domain adaptive training loss based on the virtual domain hidden features, the balanced real domain hidden features, the mean of various hidden features in the virtual domain, and the updated mean of various hidden features in the real domain. Add supervised learning loss to the domain adaptive training loss to obtain the total training loss. Use the total training loss to optimize the classification model during the training process to obtain the optimized classification model. Step S9: Determine whether the total training loss has decreased to a set threshold; If not, return to step S2; If so, proceed to step S10: input the maritime target image to be classified into the optimized classification model, and use the optimized classification model to output the category corresponding to the maritime target image to be classified; the maritime target image to be classified is a real maritime target image obtained in a real scene.

2. The imbalanced target classification method according to claim 1, characterized in that, The virtual domain training dataset is a virtual domain maritime target classification training dataset with a balanced class distribution; the real domain training dataset is a real domain maritime target classification training dataset with an imbalanced class distribution. The total number of categories and the number of categories in the virtual domain maritime target classification training dataset with balanced category distribution are the same as those in the real domain maritime target classification training dataset with unbalanced category distribution. In a virtual domain maritime target classification training dataset with balanced class distribution, the number of maritime target image samples of different classes is balanced; in a real domain maritime target classification training dataset with imbalanced class distribution, the number of maritime target image samples of different classes is imbalanced.

3. The imbalanced target classification method according to claim 1, characterized in that, The batch training samples in the real domain are obtained by sampling a batch of marine target image samples from the real domain training dataset using an instance-balanced sampling distribution; the batch training samples in the virtual domain are obtained by sampling a batch of marine target image samples from the virtual domain training dataset using an instance-balanced sampling distribution.

4. The imbalanced target classification method according to claim 1, characterized in that, The classification model is a domain-adaptive classification model.

5. The imbalanced target classification method according to claim 1, characterized in that, The equilibrium real-domain hidden layer features are obtained by utilizing the augmented features and the real-domain hidden layer features, specifically including: Obtain the union of the augmented features and the real-domain hidden features, and use the union of the augmented features and the real-domain hidden features as the balanced real-domain hidden features.

6. The imbalanced target classification method according to claim 1, characterized in that, The process of updating the mean values ​​of various hidden features in the real domain based on the balanced hidden features of the real domain, to obtain the updated mean values ​​of various hidden features in the real domain, specifically includes: The mean values ​​of various hidden features in the real domain are calculated based on the balanced hidden features of the real domain, and the mean values ​​of various hidden features in the real domain are updated to obtain the updated mean values ​​of various hidden features in the real domain.

7. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the imbalanced target classification method according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the imbalanced target classification method according to any one of claims 1-6.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the imbalanced target classification method according to any one of claims 1-6.

Images