Remote sensing image noise sample learning method based on multi-task optimization network

Abstract

The invention discloses a remote sensing image noise sample learning method based on a multi-task optimization network, and particularly relates to the technical field of image processing, comprising the following steps: unsupervised high-level semantic representation learning; difficult sample reserved surface feature classification; and characterization learning and ground feature classification multi-task dynamic optimization. According to the invention, the adverse effect of the noise sample on the classification model can be reduced, and the ground feature recognition accuracy and robustness of the model under the noise sample condition can be improved; dynamic adjustment of multiple tasks in the model training process is achieved, and the generalization ability of the model on different data is improved; and the problem of data over-cleaning is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a remote sensing image noise sample learning method based on a multi-task optimization network. Background Art

[0002] The training process of deep learning networks requires a large number of samples. However, in the actual process of obtaining a large number of labeled samples, the phenomenon of incorrect training sample labels often occurs, which is called the noise sample problem. The appearance of noise samples seriously affects the performance of deep classifiers and is a major problem that plagues the deep extraction of refined information from hyperspectral remote sensing images.

[0003] At present, the learning and classification of hyperspectral remote sensing image noise samples mainly include:

[0004] (1) Methods based on data preprocessing. This type of noise sample learning method uses the common characteristics of the distribution of objects (such as spatial correlation) as prior information to detect and remove noise samples. After completing data preprocessing, it combines with traditional machine learning classifiers to complete classification. This type of method reduces the interference of noise samples to a certain extent, but the single noise sample detection and removal strategy exacerbates the contradiction between insufficient label data and the classifier's need for a large number of samples, limiting the training accuracy of the classifier.

[0005] (2) Methods based on specific deep network structures and robust loss functions. With the development of deep learning algorithms, the demand for sample size continues to grow. The impact of noise samples on deep learning models has attracted attention in the industry, and corresponding methods have been proposed to address this challenge. For example, a noise adaptation layer is added to the top layer of the deep neural network, or a deep neural network with a proprietary architecture is designed to learn noise samples. In addition, some methods propose to design a more appropriate loss function, or perform loss correction and reweighting based on the credibility of the loss, so as to reduce the interference of noise samples on the deep model. However, the design of specific deep network structures and loss functions relies on the experience of experts and has poor generalization on unknown data.

[0006] (3) Methods based on sample selection. This type of method uses sample selection to deal with the noise sample problem during deep network learning. Through multiple network collaboration or multiple rounds of iterative learning, correctly labeled samples are identified from the data set containing noise samples for deep model training. This type of method is simple to operate, but it is difficult to distinguish between noise samples and difficult samples, which restricts the ability of deep networks to learn complex object distributions.

[0007] Therefore, how to solve the problem of limited labeled data under noisy sample conditions, accurately distinguish noisy samples from difficult samples, reduce the adverse effects of noise samples on deep learning models, and improve the accuracy and robustness of the model in object recognition under noisy sample conditions is an urgent problem to be solved. Summary of the invention

[0008] To this end, the present invention provides a remote sensing image noise sample learning method based on a multi-task optimization network to solve the problems raised in the background technology.

[0009] In order to achieve the above object, the present invention provides the following technical solution: a remote sensing image noise sample learning method based on a multi-task optimization network, comprising the following steps:

[0010] S1. Unsupervised high-level semantic representation learning; including the following steps:

[0011] S11, performing feature transformation on the original data to obtain sample pairs of different views;

[0012] For each pixel x∈R n , two random feature conversion operations are performed to obtain feature expressions under different views, which are denoted as v and v respectively. 0 , to obtain the feature expression of the same pixel in different views;

[0013] S12, mapping feature expressions of different views to the deep feature domain;

[0014] Construct a feature extraction network for deep spatial-spectral representation learning, denoted as f; the spectral-spatial feature expressions in the deep embedding space are:

[0015] c=f(v) (1)

[0016] c 0 =f(v 0 ) (2)

[0017] S13, align samples in the feature domain;

[0018] Based on the semantic prior, that is, the feature expressions of the same pixel in different views have the same high-level semantic information, a contrast loss function is introduced to semantically align the deep embedded features of the same pixel in different views in the feature domain:

[0019] minL(c,c 0 ) (3)

[0020] Through the high-level semantic alignment strategy, the potential semantic information in the original hyperspectral data is fully mined for representation learning;

[0021] S2, difficult sample retention feature classification; including the following steps:

[0022] S21, sample cross selection through collaborative learning;

[0023] Let the training sample set be X = {x1, x2, ..., x n}∈R D×n Belongs to the D-dimensional spectral space, where n is the number of training samples; Y = {y1, y2, ..., y n}∈R C×n , where y i is the hot encoding of the label, C is the number of categories;

[0024] Considering the noise sample problem, the variable To define the labels of noise samples, there are only training sets containing noise samples during the training phase;

[0025] Based on the assumption that small loss samples have a higher probability of being clean samples, the cross selection update strategy selects small loss samples through each backbone network and inputs them into the other network to guide the learning update of network parameters, thereby reducing the further propagation and accumulation of sample selection bias caused by noise and outliers in the data during the training process; the target expression of sample selection is written as:

[0026]

[0027] in, is a small batch of data, t is the current number of iterations, is the small batch data input at the current iteration, θ1 and θ2 represent the model parameters of the two backbone networks respectively. and They represent the samples selected by the two backbone networks, and λ controls the proportion of selected samples;

[0028] S22, correcting noise samples with spatial constraints;

[0029] The spatial distribution characteristics of hyperspectral data are used to guide the correction process of noise labels: a segmentation algorithm is used to segment the hyperspectral data to obtain a superpixel cube containing k homogeneous superpixels; based on the assumption that the pixel sets in the homogeneous area share the same category label, the label of the selected sample is inferred from the category label of the superpixel to which it belongs according to formulas (4) and (5) (a superpixel block is a set of pixels, and the category of the same superpixel block is consistent, so only the category of a certain pixel in it needs to be known to infer the category of the superpixel block), and it is used to correct the noise samples discarded in S21 to alleviate the problem of insufficient training samples;

[0030] S23, select samples with large loss values ​​by category;

[0031] The classification network has similar learning patterns on noise samples and difficult samples; therefore, the sample selection method that relies on specific assumptions is difficult to accurately distinguish between noise samples and difficult samples, which can easily lead to excessive cleaning of difficult samples, resulting in insufficient training samples to construct the object classification model, affecting the classification performance; therefore, samples with larger loss values ​​are selected for each category as noise samples for exclusion, while retaining the most difficult samples possible;

[0032] S3. Representation learning and multi-task dynamic optimization of object classification.

[0033] Preferably, step S3 comprises the following steps:

[0034] S31, multi-task loss function construction;

[0035] Construct a multi-task integrated model of feature learning and object classification, and combine the multi-task loss function of feature extraction and object recognition. The loss function of the multi-task framework is expressed as:

[0036] L M =w F ·L F +w C ·L C ; (6)

[0037] Among them, L F is the feature learning loss, L C is the classification loss, w F and w C are the weight factors for feature learning and classification tasks respectively; the shared parameter W sh The objective function in gradient descent optimization is written as:

[0038]

[0039] Among them, η is the learning rate; the shared parameter W sh The optimization of is affected by feature learning loss and classification loss, and different losses have different effects on the shared parameters W sh The influence of is adjusted using weight factors;

[0040] S32, dynamic adjustment of weight factors;

[0041] Considering the magnitude differences of the loss values ​​of different tasks, the weights of tasks are dynamically adjusted during the training process to adaptively achieve the balance of multi-task learning. For each subtask, the ratio of the corresponding losses of the first two iterations is first calculated, and the change in the loss value is used to measure the learning speed of the subtask, which is expressed as:

[0042]

[0043] Among them, t represents the current iteration, and L represents the loss value. Then, the normalization constraint is used to obtain the weight of each subtask, and the weight change is expressed as:

[0044]

[0045] Where T is a constant, and its size represents the degree of looseness between subtasks. If T is large enough, λ will tend to 1, indicating that each subtask has the same weight. The sum of the weights of each subtask is K, ensuring that the weights are scaled in the same dimension.

[0046] Based on formula (9), if the subtask gradient descent is faster (relatively speaking after comparing the two tasks), its normalized weight is smaller (relatively speaking after comparing the two tasks), and the loss weight of the task that is trained faster is reduced through constraints.

[0047] The present invention has the following advantages:

[0048] 1. The present invention proposes a remote sensing image noise sample learning method based on a multi-task optimization network, which can reduce the adverse effects of noise samples on the classification model and improve the accuracy and robustness of the model in object recognition under noise sample conditions.

[0049] 2. The present invention proposes a multi-task optimization framework for representation learning and object recognition under noisy sample conditions. Aiming at the situation where the existing noise sample learning label data is limited, the present invention fully mines the potential semantic information of a large amount of unlabeled data in the image, and considers the magnitude relationship of different subtasks, so as to realize dynamic adjustment of multiple tasks during the model training process and improve the generalization ability of the model on different data.

[0050] 3. The present invention proposes a noise sample learning method with difficult sample retention, which distinguishes and corrects noise samples and difficult samples from both the semantic level and spatial distribution pattern, and solves the problem of over-cleaning of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the workflow provided by the present invention;

[0052] Figure 2 A schematic diagram of difficult sample retention provided by the present invention. DETAILED DESCRIPTION

[0053] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] See also Figure 1 This embodiment provides a remote sensing image noise sample learning method based on a multi-task optimization network, comprising the following steps:

[0055] S1. Unsupervised high-level semantic representation learning; including the following steps:

[0056] S11, performing feature transformation on the original data to obtain sample pairs of different views;

[0057] For each pixel x∈R n , two random feature conversion operations are performed to obtain feature expressions under different views, which are denoted as v and v respectively. 0 , to obtain the feature expression of the same pixel in different views;

[0058] S12, mapping feature expressions of different views to the deep feature domain;

[0059] Construct a feature extraction network for deep spatial-spectral representation learning, denoted as f; the spectral-spatial feature expressions in the deep embedding space are:

[0060] c=f(v) (1)

[0061] c 0 =f(v 0 ) (2)

[0062] S13, align samples in the feature domain;

[0063] Based on the semantic prior, that is, the feature expressions of the same pixel in different views have the same high-level semantic information, a contrast loss function is introduced to semantically align the deep embedded features of the same pixel in different views in the feature domain:

[0064] minL(c,c 0 )(3)

[0065] Through the high-level semantic alignment strategy, the potential semantic information in the original hyperspectral data is fully mined for representation learning;

[0066] S2, difficult sample retention feature classification; including the following steps:

[0067] S21, sample cross selection through collaborative learning;

[0068] Let the training sample set be X = {x1, x2, ..., x n}∈R D×n Belongs to the D-dimensional spectral space, where n is the number of training samples; Y = {y1, y2, ..., y n}∈R C×n , where yi is the hot encoding of the label, C is the number of categories;

[0069] Considering the noise sample problem, the variable To define the labels of noise samples, there are only training sets containing noise samples during the training phase;

[0070] Based on the assumption that small loss samples have a higher probability of being clean samples, the cross selection update strategy selects small loss samples through each backbone network and inputs them into the other network to guide the learning update of network parameters, thereby reducing the further propagation and accumulation of sample selection bias caused by noise and outliers in the data during the training process; the target expression of sample selection is written as:

[0071]

[0072] in, is a small batch of data, t is the current number of iterations, is the small batch data input at the current iteration, θ1 and θ2 represent the model parameters of the two backbone networks respectively. and They represent the samples selected by the two backbone networks, and λ controls the proportion of selected samples;

[0073] S22, correcting noise samples with spatial constraints;

[0074] The spatial distribution characteristics of hyperspectral data are used to guide the correction process of noise labels. A segmentation algorithm is used to segment the hyperspectral data to obtain a superpixel cube containing k homogeneous superpixels. Based on the assumption that the pixel sets in the homogeneous area share the same category label, the label of the selected sample is inferred from the label of the superpixel in which it is located according to formulas (4) and (5) (a superpixel block is a set of pixels, and the category of the same superpixel block is consistent. Therefore, only the category of a certain pixel in it needs to be known to infer the category of the superpixel block). This is used to correct the noise samples discarded in S21 and alleviate the problem of insufficient training samples.

[0075] S23, select samples with large loss values ​​by category;

[0076] The classification network has similar learning patterns on noise samples and difficult samples; therefore, the sample selection method that relies on specific assumptions is difficult to accurately distinguish between noise samples and difficult samples, which can easily lead to excessive cleaning of difficult samples, resulting in insufficient training samples to construct the object classification model, affecting the classification performance; therefore, samples with larger loss values ​​are selected for each category as noise samples for exclusion, while retaining the most difficult samples possible;

[0077] like Figure 2As shown in the figure, the red circles, blue triangles and green squares represent samples of different categories. The overall training loss in the red circle category is too large. If the samples with large losses are excluded from the overall data distribution (as shown by the blue dotted line), the data will be over-cleaned. Therefore, by selecting samples by category (as shown by the blue solid line), the difficult samples can be retained as much as possible.

[0078] S3. Representation learning and multi-task dynamic optimization of object classification.

[0079] Preferably, step S3 comprises the following steps:

[0080] S31, multi-task loss function construction;

[0081] Construct a multi-task integrated model of feature learning and object classification, and combine the multi-task loss function of feature extraction and object recognition. The loss function of the multi-task framework is expressed as:

[0082] L M =w F ·L F +w C ·L C ; (6)

[0083] Among them, L F is the feature learning loss, L C is the classification loss, w F and w C are the weight factors for feature learning and classification tasks respectively; the shared parameter W sh The objective function in gradient descent optimization is written as:

[0084]

[0085] Among them, η is the learning rate; the shared parameter W sh The optimization of is affected by feature learning loss and classification loss, and different losses have different effects on the shared parameters W sh The influence of is adjusted using weight factors;

[0086] S32, dynamic adjustment of weight factors;

[0087] Considering the magnitude differences of the loss values ​​of different tasks, the weights of tasks are dynamically adjusted during the training process to adaptively achieve the balance of multi-task learning. For each subtask, the ratio of the corresponding losses of the first two iterations is first calculated, and the change in the loss value is used to measure the learning speed of the subtask, which is expressed as:

[0088]

[0089] Among them, t represents the current iteration, and L represents the loss value. Then, the normalization constraint is used to obtain the weight of each subtask, and the weight change is expressed as:

[0090]

[0091] Where T is a constant, and its size represents the degree of looseness between subtasks. If T is large enough, λ will tend to 1, indicating that each subtask has the same weight. The sum of the weights of each subtask is K, ensuring that the weights are scaled in the same dimension.

[0092] Based on formula (9), if the subtask gradient descent is faster (relatively speaking after comparing the two tasks), its normalized weight is smaller (relatively speaking after comparing the two tasks), and the loss weight of the task that is trained faster is reduced through constraints.

[0093] After training, the final model outputs high-precision remote sensing image classification results such as Figure 1 shown.

[0094] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A remote sensing image noise sample learning method based on a multi-task optimization network, characterized by: The following steps are involved: S1. Unsupervised high-level semantic representation learning; including the following steps: S11, performing feature transformation on the original data to obtain sample pairs of different views; For each pixel x∈R n , two random feature conversion operations are performed to obtain feature expressions under different views, which are denoted as v and v respectively. 0 To obtain the feature expression of the same pixel in different views; S12, mapping feature expressions of different views to the deep feature domain; Construct a feature extraction network for deep spatial-spectral representation learning, denoted as f; the spectral-spatial feature expressions in the deep embedding space are: c=f(v) (1) c 0 =f(v 0 ) (2) S13, align samples in the feature domain; Based on the semantic prior, that is, the feature expressions of the same pixel in different views have the same high-level semantic information, a contrast loss function is introduced to semantically align the deep embedded features of the same pixel in different views in the feature domain: minL(c,c 0 )(3) Through the high-level semantic alignment strategy, the potential semantic information in the original hyperspectral data is fully mined for representation learning; S2, difficult sample retention feature classification; including the following steps: S21, sample cross selection through collaborative learning; Let the training sample set be X = {x1, x2, ..., x n }∈R D×n Belongs to the D-dimensional spectral space, where n is the number of training samples; Y = {y1, y2, ..., y n }∈R C×n , where y i is the hot encoding of the label, C is the number of categories; Considering the noise sample problem, the variable To define the labels of noise samples, there are only training sets containing noise samples during the training phase; Based on the assumption that small loss samples have a higher probability of being clean samples, the cross selection update strategy selects small loss samples through each backbone network and inputs them into the other network to guide the learning update of network parameters, thereby reducing the further propagation and accumulation of sample selection bias caused by noise and outliers in the data during the training process; the target expression of sample selection is written as: in, is a small batch of data, t is the current number of iterations, is the small batch data input at the current iteration, θ1 and θ2 represent the model parameters of the two backbone networks respectively. and represents the samples selected by the two backbone networks, and λ controls the proportion of selected samples; S22, correcting noise samples with spatial constraints; The spatial distribution characteristics of hyperspectral data are used to guide the correction process of noise labels: a superpixel cube containing k homogeneous superpixels is obtained from the hyperspectral data using a segmentation algorithm; based on the assumption that the pixel sets in the homogeneous area share the same category label, the label of the selected sample is inferred according to formulas (4) and (5) to infer the category label of the superpixel to which it belongs, and is used to correct the noise samples discarded in S21 to alleviate the problem of insufficient training samples; S23, select samples with large loss values ​​by category; By selecting samples with larger loss values ​​for each category as noise samples and excluding them, the difficult samples are retained as much as possible; S3. Representation learning and multi-task dynamic optimization of object classification.

2. According to claim 1, a remote sensing image noise sample learning method based on a multi-task optimization network is characterized in that: The step S3 comprises the following steps: S31, multi-task loss function construction; Construct a multi-task integrated model of feature learning and object classification, and combine the multi-task loss function of feature extraction and object recognition. The loss function of the multi-task framework is expressed as: L M =w F ·L F +w C ·L C ; (6) Among them, L F is the feature learning loss, L C is the classification loss, w F and w C are the weight factors for feature learning and classification tasks respectively; the shared parameter W sh The objective function in gradient descent optimization is written as: Among them, η is the learning rate; the shared parameter W sh The optimization of is affected by feature learning loss and classification loss, and different losses have different effects on the shared parameters W sh The influence of is adjusted using weight factors; S32, dynamic adjustment of weight factors; Considering the magnitude differences of the loss values ​​of different tasks, the weights of tasks are dynamically adjusted during the training process to adaptively achieve the balance of multi-task learning. For each subtask, the ratio of the corresponding losses of the first two iterations is first calculated, and the change in the loss value is used to measure the learning speed of the subtask, which is expressed as: Among them, t represents the current iteration, and L represents the loss value. Then, the normalization constraint is used to obtain the weight of each subtask, and the weight change is expressed as: Where T is a constant, and its size represents the degree of looseness between subtasks. If T is large enough, λ will tend to 1, indicating that each subtask has the same weight. The sum of the weights of each subtask is K, ensuring that the weights are scaled in the same dimension. Based on formula (9), if the subtask gradient decreases faster, its normalized weight is smaller, and the loss weight of the task that is trained faster is reduced through constraints.

Images