A hybrid enhancement method and system for remote sensing image class imbalance

Abstract

The application provides a mixed enhancement method and system for remote sensing image class imbalance, and relates to the technical field of computer vision, and comprises the following steps: different clipping methods are selected according to the proportion of each class in the total number in the training data set, and remote sensing images are clipped respectively to obtain clipped images; a confidence matrix is set to store the performance of each class at the current stage, and the confidence matrix is initialized so that the initial confidence values of all classes are the same; a segmentation model is trained according to the clipped images, and the confidence values of each class in the confidence matrix are dynamically updated; after each round of training is completed, the sampling learning of the class with a high confidence value is reduced according to the current confidence value of each class, and meanwhile, the images corresponding to the class with a low confidence value are pasted for enhancement in the training set; and the segmentation model is iteratively trained to obtain a final segmentation model. The application solves the problem of lack of expression of rare classes in remote sensing images through prior enhancement and confidence mixing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image semantic segmentation, and in particular to a hybrid enhancement method and system for remote sensing image class imbalance. BACKGROUND

[0002] In recent years, the research on remote sensing technology has been continuously strengthened, and the rapid development of remote sensing technology has greatly improved the usability of remote sensing images, which are widely used in object classification, target detection, change detection and other tasks. High spatial resolution remote sensing images contain rich object information such as structure, shape, texture and adjacency. It extracts ground object information to support various civilian tasks such as urban planning, land use and environmental monitoring. In addition, the data source of high spatial resolution remote sensing images is more extensive, and it plays a more important role in military tasks such as military identification. Remote sensing images not only contain more abundant information, but also have more complex semantic features, which makes the semantic segmentation of remote sensing images one of the most challenging problems in image segmentation.

[0003] In addition to the challenges commonly faced by natural image semantic segmentation, remote sensing image semantic segmentation has its unique challenges. These challenges can be attributed to at least three reasons: first, one of the biggest differences between remote sensing images and natural images is the imbalance of foreground-background. Compared with natural scene datasets such as COCO and PASCAL VOC2012, the foreground ratio of remote sensing semantic segmentation datasets such as iSAID is lower. This imbalance problem increases with the depth of the segmentation network, and too much background information will introduce noise into the feature learning of the foreground, causing the classifier to be inaccurate and unable to effectively distinguish between foreground objects; second, satellites or drones capture remote sensing images from different angles, resulting in objects of various sources and different scales. However, due to the multi-scale nature of the problem, this poses a major challenge, resulting in significant differences between objects in the image; finally, the most significant challenge in remote sensing datasets is the severe class imbalance problem. During the training process, the network tends to prioritize classes with more instances and pay less attention to rare classes, which leads to overfitting of the majority class during the prediction process, resulting in the neglect of rare classes.

[0004] In the existing remote sensing image semantic segmentation research, the first used are all the semantic segmentation framework under natural images, such as FCN, DeepLab and the like, and almost no method focuses on the class imbalance problem, however, the semantic segmentation framework under natural images often ignores the foreground-background imbalance and the intra-class significant difference problem, thereby limiting the semantic segmentation of remote sensing images. With the development of deep learning, it has shown significant performance in computer vision tasks, and it has become feasible to use deep learning to learn visual features, however, when the deep learning network extracts specific evidence, the rare class is easy to be overwhelmed by the surrounding noise, or the rare class lacks attention in attention learning, resulting in poor performance of the rare class features.

[0005] How to solve the foreground-background imbalance problem in remote sensing image segmentation, and the problem of lack of expression of rare classes in remote sensing images, i.e. the intra-class significant difference problem, and improve the effect of remote sensing image semantic segmentation. SUMMARY

[0006] In view of the above problems, the application provides a hybrid enhancement method and system for class imbalance of remote sensing images, which enhances the training effect of the segmentation model through prior knowledge enhancement and hybrid enhancement, and further improves the semantic segmentation effect in remote sensing images.

[0007] To achieve the above purpose, the application provides a hybrid enhancement method for class imbalance of remote sensing images, comprising:

[0008] According to the proportion of each class in the total number in the training data set, different image cropping methods are selected, and the remote sensing images in the training data set are cropped respectively to obtain cropped images;

[0009] A confidence matrix is set to store the performance of each class at the current stage, and the confidence matrix is initialized so that the initial confidence values of all classes are the same;

[0010] According to the cropped images, a segmentation model is trained, and the confidence values of each class in the confidence matrix are dynamically updated;

[0011] After each round of training, according to the current confidence value of each class, the sampling learning of the class with high confidence value is reduced, and the image corresponding to the class with low confidence value is pasted and enhanced in the training set;

[0012] The segmentation model is iteratively trained to obtain a final segmentation model.

[0013] As a further improvement of the application, according to the proportion of each class in the total number in the training data set, different image cropping methods are selected, comprising:

[0014] For the jth remote sensing image, a sliding window with different step length is selected according to the sum of the weights of the rare class and the non-rare class and the formula:

[0015]

[0016] wherein,

[0017] is the proportion of the pixel number of the cth class in the jth remote sensing image to the total pixel number of the jth remote sensing image; i

[0018] K j =H indicates that there is only one background class in the jth remote sensing image;

[0019] I d and I u respectively represent the set of non-rare classes and the set of rare classes;

[0020] represents the weight of the Cth class in the remote sensing image training set;

[0021] H, S1 and S2 all represent the step length of the sliding window, H represents the maximum step length, S2 represents the minimum step length, and the relationship is S2

[0022] When the sum of the probabilities of the rare classes of the jth remote sensing image is less than the sum of the probabilities of the non-rare classes, the jth remote sensing image is cut with the step length S1;

[0023] When the sum of the probabilities of the rare classes of the jth remote sensing image is greater than or equal to the sum of the probabilities of the non-rare classes, the jth remote sensing image is cut with the step length S2.

[0024] As a further improvement of the present application, for the jth remote sensing image, the C foreground classes are divided into rare classes and non-rare classes, and a weight including:

[0025] The pixel number of each foreground class is calculated as the proportion of the pixel number of all foreground classes

[0026] Under the class balance, the pixel number of each foreground class is 1 / C of the pixel number of all foreground classes;

[0027] Comparing and 1 / C, if the foreground class is a rare class, the foreground class is a non-rare class;

[0028] A weight is assigned to each foreground class is the reciprocal of , and the formula is: ​

[0029]

[0030] where,

[0031] N C is the total number of foreground pixels in all foreground classes;

[0032] is the total number of pixels in the c i th class;

[0033] The weight is larger indicates that the number of class c i is less.

[0034] As a further improvement of the present application, a set of zero vectors is randomly initialized as the initial confidence matrix, and the formula is:

[0035] Conf0 = Zeros((1, C))

[0036] Where Zeros represents the zero vector initialization method in numpy or pytorch, and C represents the number of classes in the data set.

[0037] As a further improvement of the present application, the segmentation model is trained according to the cropped image, and the confidence values of each class in the confidence matrix are dynamically updated; including:

[0038] The segmentation model accumulates the confidence of each stage during each training, and uses an exponential moving average to update the confidence value, and the formula is:

[0039]

[0040]

[0041] where,

[0042] α∈[0, 1], c represents the class in the data set, and t represents the current training stage;

[0043] TP and FP represent true positive and false positive, respectively.

[0044] As a further improvement of the present application, after each round of training, according to the current confidence value of each class, the sampling learning of the class with high confidence value is reduced, including:

[0045] The confidence matrix is replaced by a normalized sampling probability r1, which is expressed as: r1 = softmax(Conf t )

[0046] According to the sampling probability, K1 categories with high sampling probability are randomly selected to form a candidate set C1, random partial occlusion operation is performed on the categories belonging to the candidate set C1 in each remote sensing image, and the pixel value and label of the remote sensing image are updated, which are respectively represented as:

[0047]

[0048]

[0049] The pixel value of the pixel point with coordinate position (i, j) in the remote sensing image I after the Mask update is represented as:

[0050] The label value of the label corresponding to the pixel point with coordinate position (i, j) in the remote sensing image I after the Mask update is represented as:

[0051] l i，j The class label of the pixel point with coordinate position (i, j) in the remote sensing image I is represented as:

[0052] As a further improvement of the application, the image corresponding to the class with low confidence is pasted and enhanced in the training set; comprising:

[0053] The confidence matrix is replaced by a normalized sampling probability r2, which is represented as: r2=softmax(1-Conf t )

[0054] According to the sampling probability, K2 categories are randomly selected to form a candidate set C2, and random partial copy and paste operation is performed on the categories belonging to the candidate set C2 in each remote sensing image.

[0055] As a further improvement of the application, the intersection of the class in each remote sensing image and the classes in other images of a group of images containing the image in the training data set is calculated, and the remote sensing image I P As the source of the copy and paste object, the object in the remote sensing image I P is pasted into the remote sensing image I, I P is represented as:

[0056]

[0057] Wherein,

[0058] Max length represents the maximum length belonging to the intersection of the image I;

[0059] I Batch represents a group of images in the remote sensing image training data set;

[0060] I' represents a set of non-I P other images.

[0061] As a further improvement of the present application, the remote sensing image after the paste enhancement is updated pixel value and label, respectively represented as:

[0062]

[0063]

[0064]

[0065] wherein,

[0066] represents the pixel value of the pixel point at position (i, j) in the remote sensing image I after the paste enhancement update;

[0067] represents the label value of the pixel point at coordinate position (i, j) in the remote sensing image I after the paste enhancement update;

[0068] represents the class label of the pixel point at position (i, j) in the remote sensing image I p

[0069] l i，j represents the class label of the pixel point at position (i, j) in the remote sensing image I

[0070] l i，j = 0 represents l i，j is a background class, l i，j = 255 represents l i，j is a pixel position that is occluded.

[0071] The present application also provides a mixed enhancement system for remote sensing image class imbalance, comprising: a prior image preprocessing module, a confidence module, a segmentation model training module and an image dynamic processing module.

[0072] The prior image preprocessing module is used to:

[0073] According to the proportion of each class in the total number in the training data set, different image cropping methods are selected, and the remote sensing images in the training data set are cropped respectively to obtain the cropped images;

[0074] The confidence module is used to:

[0075] Set a confidence matrix to store the performance of each class at the current stage, and initialize the confidence matrix so that the initial confidence values of all classes are the same; ​

[0076] The segmentation model training module is configured to:

[0077] According to the cropped image, the segmentation model is trained, and the confidence values of each category in the confidence matrix are dynamically updated.

[0078] The segmentation model is iteratively trained to obtain a final segmentation model.

[0079] The image dynamic processing module is configured to:

[0080] After each round of training, according to the current confidence value of each class, the sampling learning of the category with a high confidence value is reduced, and the image corresponding to the category with a low confidence value is pasted and enhanced in the training set.

[0081] Compared with the prior art, the present application has the following advantages:

[0082] The integrated mixed enhancement method of the present application is proposed for the class imbalance problem in remote sensing images. Not only is the data enhanced through prior knowledge from the perspective of data, but also the classes are dynamically focused during the training phase from the perspective of dynamic confidence update, so that all classes can be enhanced during the network training phase. At the same time, the class enhancement method in the present application simultaneously processes the classes with high confidence values and the classes with low confidence values after each training, and does not target a certain semantic segmentation method for remote sensing images, thereby effectively improving the segmentation effect of remote sensing images.

[0083] In the present application, the classes with high sampling probability are occluded, and the classes with low sampling probability are copied and pasted, thereby enhancing the learning effect of each type of image and improving the segmentation effect of remote sensing images.

[0084] The integrated mixed enhancement method of the present application is a plug-and-play module that can be directly introduced into any remote sensing semantic segmentation method to improve performance. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 A mixed enhancement method flowchart for remote sensing image class imbalance is disclosed for an embodiment of the present application.

[0086] Figure 2 A mixed enhancement system schematic diagram for remote sensing image class imbalance is disclosed for an embodiment of the present application.

[0087] Figure 3 An algorithm overall architecture diagram of the mixed enhancement method for remote sensing image class imbalance is disclosed for an embodiment of the present application. DETAILED DESCRIPTION

[0088] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0089] The present application will be described in further detail below with reference to the drawings:

[0090] As shown in Figure 1 , 3 , the present application provides a hybrid enhancement method for remote sensing image class imbalance, comprising the steps of:

[0091] S1, according to the proportion of each class in the total number of training data set, different image cropping methods are selected, and the remote sensing images in the training data set are cropped respectively to obtain the cropped images;

[0092] Among them,

[0093] In remote sensing images, images usually have high resolution. For remote sensing images, the commonly used preprocessing method is to crop the image with a fixed sliding window of (H, H) at a fixed step. In the present application, the classes in the remote sensing image are divided into three categories: background class, rare class and non-rare class. The traditional preprocessing method uses the same cropping method for the three categories of objects, which aggravates the difference in the number of rare classes, background classes and non-rare classes, resulting in poor generalization ability of the model for rare classes. The present application proposes a prior enhancement module to crop the three categories in different ways, and the processed images are all processed into images of size (H, H). The specific cropping method and cropping process are all responsible by the prior enhancement module, including:

[0094] For the jth remote sensing image, different step sliding windows are selected according to the sum of the weights of the rare class and the non-rare class and the formula, and the formula is:

[0095]

[0096] Among them,

[0097] C i represents the i-th foreground class;

[0098] is the proportion of the pixel number of the c i th class in the jth remote sensing image to the total pixel number of the jth remote sensing image;

[0099] K j = H represents that the jth remote sensing image has only one background class;

[0100] I d and I u These represent the sets of non-rare classes and the sets of rare classes, respectively.

[0101] In the j-th remote sensing image, the c-th image is represented by... i Class weight;

[0102] H, S1, and S2 all represent the step size of the sliding window, with H representing the maximum step size and S2 representing the minimum step size. The relationship is S2 < S1 < H.

[0103] when When the value is 0, it means that there is no foreground class in the image, only background class. In this case, in order to reduce the background class, the step size of the sliding window is set to the maximum value, that is, the length of the sliding window.

[0104] When the sum of the probabilities of rare classes in the j-th remote sensing image is less than the sum of the probabilities of non-rare classes, the j-th remote sensing image is segmented using a step size S1.

[0105] When the j When the sum of the probabilities of rare classes in a remote sensing image is greater than or equal to the sum of the probabilities of non-rare classes, the j-th remote sensing image is segmented using a step size S2.

[0106] The sampling method based on this formula reduces the sampling probability of non-rare class objects, reduces the training amount of non-rare classes, and compensates for class imbalance.

[0107] Furthermore,

[0108] For the j-th remote sensing image, the C foreground classes are divided into rare and non-rare classes, and weights are assigned to each rare and non-rare class. include:

[0109] Calculate the proportion of pixels of each foreground class to the total number of pixels of all foreground classes.

[0110] Under class balance, the ratio of the number of pixels in each foreground class to the total number of pixels in all foreground classes is 1 / C;

[0111] Compare And 1 / C, if Then the foreground class is a rare class. Then the foreground class is a non-rare class;

[0112] Assign weights to each foreground class equal The reciprocal of , the formula is:

[0113]

[0114] wherein,

[0115] N C is the total number of foreground pixels in all foreground classes;

[0116] is the total number of pixels in the c i class;

[0117] The greater the weight indicates that the number of class c i is less.

[0118] S2, set the confidence matrix to store the performance of each class in the current stage, and initialize the confidence matrix so that the initial confidence values of all classes are the same;

[0119] wherein,

[0120] The present application separates rare and non-rare classes from data through the prior knowledge module and solves the imbalance problem in the data. However, to distinguish rare classes and non-rare classes, not only this method is needed, but also in the actual training process, even if the number of some classes is very small, good features (i.e., good performance) can also be learned, for example, in the sea or lake surface scene, although the number of ships is much smaller than the background lake, but since the water surface is basically all ships, so the ship is very rare, but the features are clear, so the performance in the learning process is very good.

[0121] Therefore, in order to effectively improve the learning of classes with poor performance in the training process, the present application proposes a confidence matrix for storing the performance of each class in the current training stage. In order to divide the performance of each class before the model starts training, the probability of sampling is the same, the present application uses a zero vector to initialize the confidence matrix:

[0122] Conf0=Zeros((1,C))

[0123] Where Zeros represents the zero vector initialization method in numpy or pytorch, and C represents the number of classes in the data set.

[0124] S3, training the segmentation model according to the cropped image, and dynamically updating the confidence values of each class in the confidence matrix;

[0125] wherein,

[0126] With the continuous training of the network, each class in the network will be learned, but due to the different surrounding environment of each class, the number of each class is different, resulting in inconsistent performance of each class in each stage. After each iteration, the present application records the performance of each class in the current stage, so as to dynamically update the confidence values of each class in the confidence matrix.

[0127] The segmentation model accumulates a stage of confidence at each training time, and uses an exponential moving average to update the confidence value, formula:

[0128]

[0129]

[0130] wherein,

[0131] α∈[0, 1], c represents the class in the data set, and t represents the current training stage;

[0132] TP and FP represent true positive and false positive respectively.

[0133] The application uses a confidence matrix to record the current performance of each class, and then obtains the class with good current performance and the class with poor current performance through the confidence matrix. The confidence matrix considers that the class with good performance is not rare, and the class with poor performance is rare.

[0134] S4, after each round of training, according to the current confidence value of each class, reducing the sampling learning of the class with high confidence value (current performance is good), and at the same time, pasting enhancement is performed on the image corresponding to the class with low confidence value (current performance is poor) in the training set;

[0135] wherein,

[0136] Reducing the sampling learning of the class with high confidence value (current performance is good) includes:

[0137] In the network training process, if no restriction is added, the network will continuously deviate to the class with good performance, resulting in that some classes perform too well. In this part, the application dynamically suppresses the number of non-rare classes, so that the network can focus on rare classes, that is, after each round of training, the confidence matrix updated in step S3 is used as a sampling probability, and some classes are selected as classes that need to be occluded. If these classes exist in a remote sensing image, a part of the remote sensing image is randomly occluded, so as to reduce the attention of the network to these classes in the next round.

[0138] Specifically, after each round of training, according to the current confidence value of each class, reducing the sampling learning of the class with high confidence value includes:

[0139] The confidence matrix is replaced by a normalized sampling probability r1, which is expressed as: r1=softmax(Conf t )

[0140] Based on the sampling probability, K1 categories with high sampling probabilities are randomly selected to form a candidate set C1. Random partial occlusion is then performed on the classes belonging to candidate set C1 in each remote sensing image, and the pixel values ​​and labels of the remote sensing images are updated accordingly:

[0141]

[0142]

[0143] This represents the pixel value of the pixel at coordinate position (i, j) in remote sensing image I after the Mask update;

[0144] This represents the label value of the pixel at coordinate position (i, j) in remote sensing image I after the Mask update.

[0145] l i，j The class label represents the pixel at coordinate (i, j) in remote sensing image I.

[0146] Furthermore,

[0147] Images corresponding to categories with low confidence (currently performing poorly) are pasted and augmented from the training set, including:

[0148] By using occlusion enhancement, the network will reduce the learning of the classes that perform well at the current stage. However, because there is a serious imbalance between the foreground and background in remote sensing images, after occlusion enhancement, a part of the non-rare classes in the foreground is occluded, and the number of foreground samples is reduced. This undoubtedly amplifies the problem of foreground and background imbalance. Therefore, this invention continues to propose a dynamic pasting enhancement strategy.

[0149] Based on the confidence coefficients of each category in the confidence matrix, the categories that need to be enhanced by pasting are sampled according to their confidence values. Pasting enhancement is performed on any set of images in the training set, which can not only increase the number of poorly performing categories and make the network pay more attention to these analogies, but also increase the number of foreground categories and reduce background noise.

[0150] Copy-paste randomly samples classes between images, which can increase the number of foreground samples. However, copy-paste based on random selection may select occluded classes for pasting, which conflicts with the dynamic occlusion proposed in step 4. Therefore, this invention provides a confidence-based dynamic pasting module, specifically including:

[0151] Replacing the confidence matrix with normalized sampling probabilities r2, we can express it as: r2 = softmax(1 - Conf t )

[0152] According to the sampling probability, K2 categories are randomly selected to form a candidate set C2, and a random partial copy-paste operation is performed on the categories in each remote sensing image belonging to the candidate set C2.

[0153] Specifically, the intersection of the class in each remote sensing image and the classes in other images in a group of images containing the image in the training data set is calculated, and the remote sensing image I P is selected as the source of the copy-paste object, wherein the selected remote sensing image I P is the remote sensing image with the largest intersection. When the selected range is a batch of remote sensing images in the remote sensing image training data set, the remote sensing images are divided into different batches according to the acquisition channel, acquisition time, etc. of the remote sensing images in the remote sensing image training data set, and each batch of remote sensing images is also referred to as a group of remote sensing images; the object in the remote sensing image I P is pasted into the remote sensing image I P , which is represented as:

[0154]

[0155] In the formula,

[0156] Max length represents the maximum length of the intersection belonging to the image I;

[0157] I Batch represents a group of images in the remote sensing image training data set;

[0158] I′ represents other images in a group of images in the remote sensing image training data set, other than I P .

[0159] The pixel value and label of the remote sensing image after the paste enhancement are updated, respectively represented as:

[0160]

[0161]

[0162]

[0163] wherein,

[0164] represents the pixel value of the pixel point at position (i, j) in the remote sensing image I after the paste enhancement update;

[0165] represents the label value of the label corresponding to the pixel point at coordinate position (i, j) in the remote sensing image I after the paste enhancement update;

[0166] represents the remote sensing image I pa class label of a pixel point at a position (i, j) in the remote sensing image I;

[0167] l i，j a class label of a pixel point at a position (i, j) in the remote sensing image I;

[0168] l i，j = 0 indicates that l i，j is a background class, l i，j = 255 indicates that l i，j is a pixel position that is occluded.

[0169] Condition A indicates that if the class label of a pixel point at a position (i, j) in the remote sensing image I P belongs to the candidate set C2, and the class label of the pixel point at the corresponding position (i, j) in the remote sensing image I is a background class or an occluded region label 255, at this time the application will paste the object in the image I P into the image I.

[0170] S5, iteratively training the segmentation model to obtain a final segmentation model.

[0171] Table 1 and Table 2 are the segmentation results of the segmentation model obtained by the method of the application on a remote sensing image compared with the segmentation results obtained by other segmentation methods, and it can be seen that the method proposed by the application has better segmentation effect on remote sensing image segmentation than the existing method.

[0172]

[0173] Table 1

[0174]

[0175] Table 2

[0176] As Figure 2 shown, the application also provides a hybrid enhancement system for remote sensing image class imbalance, comprising: a prior image preprocessing module, a confidence module, a segmentation model training module and an image dynamic processing module.

[0177] The prior image preprocessing module is used to:

[0178] According to the proportion of each class in the total number in the training data set, different image cropping methods are selected, and the remote sensing images in the training data set are cropped respectively to obtain cropped images;

[0179] The confidence module is used to:

[0180] A confidence matrix is set to store the performance of each class at the current stage, and the confidence matrix is initialized so that the initial confidence values of all classes are the same;

[0181] The segmentation model training module is configured to:

[0182] According to the cropped image, the segmentation model is trained, and the confidence values of each category in the confidence matrix are dynamically updated.

[0183] The segmentation model is iteratively trained to obtain a final segmentation model.

[0184] The image dynamic processing module is configured to:

[0185] After each round of training, according to the current confidence value of each class, the sampling learning of the category with a high confidence value is reduced, and the image corresponding to the category with a low confidence value is pasted and enhanced in the training set.

[0186] The advantages of the present application are:

[0187] The present application proposes an integrated mixed enhancement method for the class imbalance problem in remote sensing images. The method not only enhances the data from the perspective of prior knowledge, but also dynamically updates the confidence from the perspective of confidence. During the training phase, the class is dynamically focused, so that all classes can be enhanced during the network training phase. At the same time, the class enhancement method in the present application simultaneously processes the classes with high confidence values and the classes with low confidence values after each training, and does not target a certain semantic segmentation method for remote sensing images, thereby effectively improving the segmentation effect of remote sensing images.

[0188] In the present application, the classes with high sampling probability are shielded, and the classes with low sampling probability are copied and pasted, thereby enhancing the learning effect of each type of image and improving the segmentation effect of remote sensing images.

[0189] The integrated mixed enhancement method of the present application is a plug-and-play module that can be directly introduced into any remote sensing semantic segmentation method to improve performance.

[0190] The above is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A hybrid enhancement method for imbalanced remote sensing images, characterized in that, include: Different image cropping methods are selected based on the proportion of each class in the total number of images in the training dataset, and the remote sensing images in the training dataset are cropped respectively to obtain cropped images. Set up a confidence matrix to store the performance of each class at the current stage, and initialize the confidence matrix so that the initial confidence values ​​of all classes are the same; The segmentation model is trained based on the cropped image, and the confidence values ​​of each category in the confidence matrix are dynamically updated. After each round of training, based on the current confidence value of each class, the sampling learning of classes with high confidence values ​​is reduced, while the images corresponding to classes with low confidence values ​​are pasted into the training set for enhancement. The segmentation model is iteratively trained to obtain the final segmentation model.

2. The hybrid enhancement method for class imbalance in remote sensing images according to claim 1, characterized in that: Different image cropping methods are selected based on the proportion of each class in the total number of images in the training dataset, including: For the j-th remote sensing image, a sliding window of different lengths is selected based on the sum of the weights of rare and non-rare classes and a formula, which is: in, For the c-th image in the j-th remote sensing image i The proportion of the number of pixels in class j to the total number of pixels in the j-th remote sensing image; K j =H indicates that there is only one background class in the j-th remote sensing image; I d and I u These represent the sets of non-rare classes and the sets of rare classes, respectively. Represents the cth image in the remote sensing image training set. i Class weight; H, S1, and S2 all represent the step size of the sliding window, where H represents the maximum step size and S2 represents the minimum step size. The relationship is S2 <S1<H; When the sum of the probabilities of rare classes in the j-th remote sensing image is less than the sum of the probabilities of non-rare classes, the j-th remote sensing image is segmented using a step size S1. When the sum of the probabilities of rare classes in the j-th remote sensing image is greater than or equal to the sum of the probabilities of non-rare classes, the j-th remote sensing image is segmented using a step size S2.

3. The hybrid enhancement method for class imbalance in remote sensing images according to claim 2, characterized in that: For the j-th remote sensing image, the C foreground classes are divided into rare and non-rare classes, and weights are assigned to each rare and non-rare class. include: Calculate the proportion of pixels of each foreground class to the total number of pixels of all foreground classes. Under class balance, the ratio of the number of pixels in each foreground class to the total number of pixels in all foreground classes is 1 / C; Compare And 1 / C, if Then the foreground class is a rare class; if Then the foreground class is a non-rare class; Assign weights to each foreground class equal The reciprocal of , the formula is: in, N C It is the total number of foreground pixels in all foreground classes; It is the cth i Total number of pixels in the class; Weight The larger the value, the higher the class c. i The fewer the quantity.

4. The hybrid enhancement method for class imbalance in remote sensing images according to claim 1, characterized in that: Randomly initialize a set of zero vectors as the initial confidence matrix, as shown in the formula: Conf0 = Zeros((1,C)) Here, Zeros represents the zero vector initialization method in NumPy or PyTorch, and C represents the number of classes in the dataset.

5. The hybrid enhancement method for class imbalance in remote sensing images according to claim 1, characterized in that: A segmentation model is trained based on the cropped image, and the confidence values ​​of each category in the confidence matrix are dynamically updated; including: The segmentation model accumulates a confidence level for one stage during each training iteration and updates the confidence value using an exponential moving average, as shown in the formula: in, α∈[0,1], c represents the class in the dataset, and t represents the current training stage; TP and FP represent true positive and false positive, respectively.

6. The hybrid enhancement method for class imbalance in remote sensing images according to claim 5, characterized in that: After each round of training, based on the current confidence value of each class, the sampling learning of classes with high confidence values ​​is reduced, including: Replacing the confidence matrix with normalized sampling probabilities r1, we can express it as: r1 = softmax(Conft t Based on the sampling probability, K1 categories with high sampling probabilities are randomly selected to form a candidate set C1. Random partial occlusion is then performed on the categories belonging to candidate set C1 in each remote sensing image, and the pixel values ​​and labels of the remote sensing images are updated, as follows: in, This represents the pixel value of the pixel at coordinate position (i,j) in remote sensing image I after the Mask update; This represents the label value of the pixel at coordinate position (i,j) in remote sensing image I after the Mask update. l i,j The class label represents the pixel at coordinate (i,j) in the remote sensing image I.

7. The hybrid enhancement method for class imbalance in remote sensing images according to claim 1, characterized in that: The step of pasting and enhancing images corresponding to categories with low confidence scores in the training set includes: Replacing the confidence matrix with normalized sampling probabilities r2, we can express it as: r2 = softmax(1 - Conf t ) Based on the sampling probability, K2 categories are randomly selected to form a candidate set C2. Random partial copy and paste operations are then performed on the categories belonging to the candidate set C2 in each remote sensing image.

8. The hybrid enhancement method for class imbalance in remote sensing images according to claim 7, characterized in that: Calculate the intersection of the class in each remote sensing image with the class in other images in a set of images containing that image in the training dataset, and select the remote sensing image I with the largest intersection. P As the source for copy-pasting, remote sensing image I P The objects in the image are pasted into the remote sensing image I. P Represented as: in, Max length represents the maximum length of the intersection of images I; I Batch This represents a set of images in the remote sensing image training dataset; I ′ This represents the non-I values ​​in a set of images in the remote sensing image training dataset. P Other images.

9. The hybrid enhancement method for class imbalance in remote sensing images according to claim 8, characterized in that: The pixel values ​​and labels of the enhanced remote sensing image are updated, as shown below: condition in, This represents the pixel value of the pixel at position (i,j) in remote sensing image I after paste enhancement update; This represents the label value of the pixel at coordinate position (i,j) in remote sensing image I after pasting enhancement and update. Represents remote sensing image I p The class label of the pixel at position (i,j); l i,j The class label represents the pixel at position (i,j) in remote sensing image I; l i,j =0 means l i,j For the background class, l i,j =255 means l i,j The location of the occluded pixel.

10. A hybrid enhancement system for class imbalance in remote sensing images, implementing the method as described in any one of claims 1 to 9, characterized in that, include: The system includes a prior image preprocessing module, a confidence module, a segmentation model training module, and an image dynamic processing module. The prior image preprocessing module is used for: Different image cropping methods are selected based on the proportion of each class in the total number of images in the training dataset, and the remote sensing images in the training dataset are cropped respectively to obtain cropped images. The confidence module is used for: Set up a confidence matrix to store the performance of each class at the current stage, and initialize the confidence matrix so that the initial confidence values ​​of all classes are the same; The segmentation model training module is used for: The segmentation model is trained based on the cropped image, and the confidence values ​​of each category in the confidence matrix are dynamically updated. The segmentation model is iteratively trained to obtain the final segmentation model; The image dynamic processing module is used for: After each round of training, based on the current confidence value of each class, the sampling learning of classes with high confidence values ​​is reduced, while the images corresponding to classes with low confidence values ​​are pasted into the training set for enhancement.

Images