Multi-modal remote sensing data continuous unknown class classification method based on internal and external space fast and slow learning strategy

Abstract

The invention discloses a multi-modal remote sensing data continuous unknown class classification method based on an internal and external space fast and slow learning strategy, and belongs to the technical field of remote sensing data classification. In order to realize continuous classification of unknown classes of remote sensing data, the method comprises the following steps: acquiring multi-modal remote sensing data, and carrying out block coding to obtain coded hyperspectral data and coded laser radar data; constructing a cross-modal pixel-level space fusion module, and inputting the encoded hyperspectral data and the encoded laser radar data into the cross-modal pixel-level space fusion module to obtain a cross-modal fusion output result; constructing a fast-slow strategy of continuous learning, wherein the fast-slow strategy comprises an inner space fast-slow learning strategy model and an outer space fast-slow learning strategy model; and inputting a cross-modal fusion output result and incremental data of each round into the internal space fast and slow learning strategy model and the external space fast and slow learning strategy model constructed in the step S4 to carry out iterative learning, and then fusing the outputs of the models to obtain a continuous unknown class classification result of the multi-modal remote sensing data.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing data classification, and in particular relates to a method for classifying continuous unknown classes of multimodal remote sensing data based on internal and external space fast and slow learning strategies. Background Art

[0002] In the field of remote sensing, multimodal data includes multispectral, hyperspectral, synthetic aperture radar, lidar and other data. This data fusion method can comprehensively analyze surface features from multiple dimensions such as spectral characteristics, temporal changes and spatial distribution, thereby improving the analysis accuracy of remote sensing images. Existing multimodal remote sensing data classification methods assume that data of all modalities can be obtained at one time, but in actual applications, these data arrive incrementally in chronological order, and there will be categories (unknown classes) in the multimodal remote sensing data set that do not belong to the training set. Existing multimodal remote sensing data classification methods can usually only classify known classes, but cannot continuously classify unknown classes. Summary of the invention

[0003] The problem to be solved by the present invention is to realize continuous classification of unknown classes of remote sensing data, and propose a method for continuous unknown class classification of multimodal remote sensing data based on internal and external space fast and slow learning strategies.

[0004] To achieve the above object, the present invention is implemented through the following technical solutions:

[0005] A method for classifying persistent unknown classes of multimodal remote sensing data based on fast and slow learning strategies in internal and external spaces includes the following steps:

[0006] S1. Acquire multimodal remote sensing data, including hyperspectral data and processed lidar data;

[0007] S2. Block-encode the hyperspectral data obtained in step S1 and the processed lidar data to obtain the encoded hyperspectral data. HSI and the encoded lidar data O LiDAR ;

[0008] S3. construct a cross-modal pixel-level spatial fusion module, input the encoded hyperspectral data and the encoded lidar data obtained in step S2 into the cross-modal pixel-level spatial fusion module, and obtain a cross-modal fusion output result;

[0009] S4. Construct a fast and slow strategy for continuous learning, including an inner space fast and slow learning strategy model and an outer space fast and slow learning strategy model;

[0010] S5. Input the cross-modal fusion output results obtained in step S3 and the incremental data of each round into the inner space fast and slow learning strategy model and the outer space fast and slow learning strategy model constructed in step S4 for iterative learning, and then fuse the outputs of the models to obtain the continuous unknown class classification results of multimodal remote sensing data.

[0011] Furthermore, the hyperspectral data obtained in step S1 Where H is the height, W is the width, and L is the length. is a real number set, the processed lidar data obtained The processing methods of lidar data are denoising and rasterization.

[0012] Further, the specific implementation method of step S3 is to HSI and O LiDAR Input into two multi-head self-attention mechanisms respectively. The specific implementation process is as follows:

[0013]

[0014] V HSI =[Noise V +O HSI ,O LiDAR ] (3)

[0015] y HSI =MSA(O HSI , K HSI ,V HSI )+O HSI (4)

[0016]

[0017] V LiDAR =[Noise V +O LiDAR , O HSI ] (6)

[0018] y LiDAR =MSA(O LiDAR ,K LiDAR , V LiDAR )+O LiDAR (7)

[0019] y=(y HSI +y LiDAR ) / 2 (8)

[0020] Among them, Noise K Adaptive noise introduced for the key, Noise Vis the adaptive noise introduced by the value, MLP is a multi-layer perceptron, MSA is a multi-head self-attention mechanism, is the correlation score of the two modes, O HSI , K HSI 、V HSI Respectively represent the query, key, and value of the multi-head self-attention mechanism corresponding to the hyperspectral data. LiDAR , K LiDAR 、V LiDAR They represent the query, key, and value of the multi-head self-attention mechanism corresponding to the lidar data, y HSI and LiDAR are the output results of the cross-modal pixel fusion module of hyperspectral data and lidar data respectively, and y is the cross-modal fusion output result.

[0021] Furthermore, the specific implementation method of step S4 includes the following steps:

[0022] S4.1. Construct an inner space learning strategy model, design a model M1 with a slower update speed and a model M2 with a faster update speed, and realize continuous classification of unknown classes:

[0023] First, the input cross-modal fusion output is transformed into an orthogonal feature space using discrete cosine transform. The expression is:

[0024]

[0025] Where T is the discrete cosine transform function, q is the total number of frequency components;

[0026] Secondly, different regularization term weights are introduced for different frequency components to construct the regularization term L R for:

[0027]

[0028] Among them, β i is the weight of the regularization term of the i-th frequency component, and t represents the t-th round of training;

[0029] S4.2. Constructing an outer space learning strategy model:

[0030] Continuously generate M1 and M2 for learning, respectively realizing the preservation of old knowledge and the learning of new knowledge, and update the parameters of the model by minimizing the loss function. The loss function L is expressed as:

[0031] L=L ML +λL R (12)

[0032] L ML =max(0,d+ -d - +r) (13)

[0033] Among them, L ML is the measure of loss, λ represents the weight of the regularization term, and d + represents the Euclidean distance between the sample and the positive instance, d - represents the Euclidean distance between the sample and the negative instance, r represents the margin between the positive and negative samples, and max is the maximum value function;

[0034] Then the loss function L of M1 is slow It is expressed as:

[0035]

[0036] Then the loss function L of M2 is fast It is expressed as:

[0037]

[0038] in, represents the regularization term of the M1 model, represents the regularization term of the M2 model;

[0039] S4.3. Define slow learning strategy and fast learning strategy:

[0040] In the inner space, different regularization term weights are introduced for different frequency components. The slow learning strategy imposes higher weights on low-frequency components to construct regularization terms. The fast learning strategy imposes lower weights on low-frequency components and constructs regularization terms.

[0041] In the outer space, the learning rate set by the slow learning strategy is 1e- 6 ; The learning rate set by the fast learning strategy is 1e- 5 .

[0042] Furthermore, the specific implementation method of step S5 is: set the data of the new category input for the tth time to D (t) , and the t-th outputs of the M1 model and the M2 model are and The t-th output of the two models is fused to obtain the t-th multimodal remote sensing data continuous unknown class classification result The expression is:

[0043]

[0044] in, is the central feature representation of the cth class, is a metric matrix. The value of a indicates the degree of fusion of the features of the two models. a=0 means that only the features output by M1 are used; a=1 means that only the features output by M2 are used.

[0045] By inputting unknown class data one by one, the model's ability to recognize unknown classes is increased. The unknown class data D( t ) will change the identifiable categories from c( t-1 ) increases to c( t ).

[0046] Beneficial effects of the present invention:

[0047] The present invention discloses a method for continuously classifying unknown classes of multimodal remote sensing data based on fast and slow learning strategies of internal and external spaces. While fusing cross-modal data, a model M1 with a slower update speed and a model M2 with a faster update speed are trained through fast and slow learning strategies, and data of unknown classes are continuously input to update model parameters. Finally, the features output by M1 and M2 are fused to achieve continuous classification of unknown classes.

[0048] The method for continuous unknown class classification of multimodal remote sensing data based on internal and external space fast and slow learning strategies described in the present invention realizes small sample incremental learning. It can not only increase the robustness of classification by combining data of different modalities, but also continuously learn to classify unknown classes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a method for classifying continuous unknown classes of multimodal remote sensing data based on internal and external space fast and slow learning strategies according to the present invention;

[0050] Figure 2 is a block diagram of the method of the present invention;

[0051] Figure 3 Schematic diagram of cross-modal fusion of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0053] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0054] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 3 The detailed instructions are as follows:

[0055] A method for classifying persistent unknown classes of multimodal remote sensing data based on fast and slow learning strategies in internal and external spaces includes the following steps:

[0056] S1. Acquire multimodal remote sensing data, including hyperspectral data and processed lidar data;

[0057] Furthermore, the hyperspectral data obtained in step S1 Where H is the height, W is the width, and L is the length. is a real number set, the processed lidar data obtained The processing methods of LiDAR data are denoising and rasterization;

[0058] S2. Block-encode the hyperspectral data obtained in step S1 and the processed lidar data to obtain the encoded hyperspectral data. HSI and the encoded lidar data O LiDAR ;

[0059] S3. construct a cross-modal pixel-level spatial fusion module, input the encoded hyperspectral data and the encoded lidar data obtained in step S2 into the cross-modal pixel-level spatial fusion module, and obtain a cross-modal fusion output result;

[0060] Further, the specific implementation method of step S3 is to HSI and O LiDAR Input into two multi-head self-attention mechanisms respectively. The specific implementation process is as follows:

[0061]

[0062] V HSI =[Noise V +O HSI ,O LiDAR ] (3)

[0063] y HSI =MSA(O HSI , KHSI , V HSI )+O HSI (4)

[0064]

[0065] V LiDAR =[Noise V +O LiDAR , O HSI ] (6)

[0066] y LiDAR =MSA(O LiDAR , K LiDAR , V LiDAR )+O LiDAR (7)

[0067] y=(y HSI +y LiDAR ) / 2 (8)

[0068] Among them, Noise K Adaptive noise introduced for the key, Noise V is the adaptive noise introduced by the value, MLP is a multi-layer perceptron, MSA is a multi-head self-attention mechanism, is the correlation score of the two modes, O HSI , K HSI 、V HSI Respectively represent the query, key, and value of the multi-head self-attention mechanism corresponding to the hyperspectral data. LiDAR , K LiDAR 、V LiDAR They represent the query, key, and value of the multi-head self-attention mechanism corresponding to the lidar data, y HSI and LiDAR are the output results of the cross-modal pixel fusion module of hyperspectral data and lidar data respectively, and y is the cross-modal fusion output result.

[0069] S4. Construct a fast and slow strategy for continuous learning, including an inner space fast and slow learning strategy model and an outer space fast and slow learning strategy model;

[0070] Furthermore, the specific implementation method of step S4 includes the following steps:

[0071] S4.1. Construct an inner space learning strategy model, design a model M1 with a slower update speed and a model M2 with a faster update speed, and realize continuous classification of unknown classes:

[0072] First, the input cross-modal fusion output is transformed into an orthogonal feature space using discrete cosine transform. The expression is:

[0073]

[0074] Where T is the discrete cosine transform function, q is the total number of frequency components;

[0075] Secondly, different regularization term weights are introduced for different frequency components to construct the regularization term L R for:

[0076]

[0077] Among them, β i is the weight of the regularization term of the i-th frequency component, and t represents the t-th round of training;

[0078] S4.2. Constructing an outer space learning strategy model:

[0079] Continuously generate M1 and M2 for learning, respectively realizing the preservation of old knowledge and the learning of new knowledge, and update the parameters of the model by minimizing the loss function. The loss function L is expressed as:

[0080] L=L ML +λL R (12)

[0081] L ML =max(0,d + -d - +r) (13)

[0082] Among them, L ML is the measure of loss, λ represents the weight of the regularization term, and d + represents the Euclidean distance between the sample and the positive instance, d - represents the Euclidean distance between the sample and the negative instance, r represents the margin between the positive and negative samples, and max is the maximum value function;

[0083] Then the loss function L of M1 is slow It is expressed as:

[0084]

[0085] Then the loss function L of M2 is fast It is expressed as:

[0086]

[0087] in, represents the regularization term of the M1 model, represents the regularization term of the M2 model;

[0088] S4.3. Define slow learning strategy and fast learning strategy:

[0089] In the inner space, different regularization term weights are introduced for different frequency components. The slow learning strategy imposes higher weights on low-frequency components to construct regularization terms. The fast learning strategy imposes lower weights on low-frequency components and constructs regularization terms.

[0090] In the outer space, the learning rate set by the slow learning strategy is 1e- 6 ; The learning rate set by the fast learning strategy is 1e- 5 ;

[0091] In the outer space, the slow learning strategy sets a smaller learning rate and updates the model parameters more slowly; the fast learning strategy sets a larger learning rate and updates the model parameters more slowly.

[0092] S5. Input the cross-modal fusion output results obtained in step S3 and the incremental data of each round into the inner space fast and slow learning strategy model and the outer space fast and slow learning strategy model constructed in step S4 for iterative learning, and then fuse the outputs of the models to obtain the continuous unknown class classification results of multimodal remote sensing data.

[0093] Furthermore, the specific implementation method of step S5 is: set the data of the new category input for the tth time to D (t) , the t-th outputs of the M1 model and the M2 model are and The t-th output of the two models is fused to obtain the t-th multimodal remote sensing data continuous unknown class classification result The expression is:

[0094]

[0095] in, is the central feature representation of the cth class, is a metric matrix. The value of a indicates the degree of fusion of the features of the two models. a=0 means that only the features output by M1 are used; a=1 means that only the features output by M2 are used.

[0096] By inputting unknown class data one by one, the model's ability to recognize unknown classes is increased. The data D of the unknown class is input for the tth time. (t) Change the identifiable categories from c (t-1) Increase to c (t) .

[0097] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0098] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for classifying persistent unknown classes of multimodal remote sensing data based on fast and slow learning strategies in internal and external space, characterized in that: The steps include: S1. Acquire multimodal remote sensing data, including hyperspectral data and processed lidar data; S2. Block-encode the hyperspectral data obtained in step S1 and the processed lidar data to obtain the encoded hyperspectral data. HSI and the encoded lidar data O LiDAR ; S3. construct a cross-modal pixel-level spatial fusion module, input the encoded hyperspectral data and the encoded lidar data obtained in step S2 into the cross-modal pixel-level spatial fusion module, and obtain a cross-modal fusion output result; S4. Construct a fast and slow strategy for continuous learning, including an inner space fast and slow learning strategy model and an outer space fast and slow learning strategy model; S5. Input the cross-modal fusion output results obtained in step S3 and the incremental data of each round into the inner space fast and slow learning strategy model and the outer space fast and slow learning strategy model constructed in step S4 for iterative learning, and then fuse the outputs of the models to obtain the continuous unknown class classification results of multimodal remote sensing data.

2. According to claim 1, a multimodal remote sensing data continuous unknown class classification method based on internal and external space fast and slow learning strategy is characterized in that: Hyperspectral data acquired in step S1 Where H is the height, W is the width, and L is the length. is a real number set, the processed lidar data obtained The processing methods of lidar data are denoising and rasterization.

3. According to claim 2, a multimodal remote sensing data continuous unknown class classification method based on internal and external space fast and slow learning strategy is characterized in that: The specific implementation method of step S3 is to HSI and O LiDAR Input into two multi-head self-attention mechanisms respectively. The specific implementation process is as follows: V HSI =[Noise V +O HSI ,Oh LiDAR ] (3) y HSI =MSA(O HSI ,K HSI ,V HSI )+O HSI (4) V LiDAR =[Noise V +O LiDAR ,Oh HSI ] (6) y LiDAR =MSA(O LiDAR ,K LiDAR ,V LiDAR )+O LiDAR (7) and=(and HSI +y LiDAR ) / 2 (8) Among them, Noise K Adaptive noise introduced for the key, Noise V is the adaptive noise introduced by the value, MLP is a multi-layer perceptron, MSA is a multi-head self-attention mechanism, is the correlation score of the two modes, O HSI , K HSI 、V HSI Respectively represent the query, key, and value of the multi-head self-attention mechanism corresponding to the hyperspectral data. LiDAR , K LiDAR 、V LiDAR They represent the query, key, and value of the multi-head self-attention mechanism corresponding to the lidar data, y HSI and LiDAR are the output results of the cross-modal pixel fusion module of hyperspectral data and lidar data respectively, and y is the cross-modal fusion output result.

4. According to claim 3, a multimodal remote sensing data continuous unknown class classification method based on internal and external space fast and slow learning strategy is characterized in that: The specific implementation method of step S4 includes the following steps: S4.

1. Construct an inner space learning strategy model, design a model M1 with a slower update speed and a model M2 with a faster update speed, and realize continuous classification of unknown classes: First, the input cross-modal fusion output is transformed into an orthogonal feature space using discrete cosine transform. The expression is: Where T is the discrete cosine transform function, q is the total number of frequency components; Secondly, different regularization term weights are introduced for different frequency components to construct the regularization term L R for: Among them, β i is the weight of the regularization term of the i-th frequency component, and t represents the t-th round of training; S4.

2. Constructing an outer space learning strategy model: Continuously generate M1 and M2 for learning, respectively realizing the preservation of old knowledge and the learning of new knowledge, and update the parameters of the model by minimizing the loss function. The loss function L is expressed as: L=L ML +λL R (12) L ML =max(0,d + -d-+r) (13) Among them, L ML is the measure of loss, λ represents the weight of the regularization term, and d + represents the Euclidean distance between the sample and the positive instance, d - represents the Euclidean distance between the sample and the negative instance, r represents the margin between the positive and negative samples, and max is the maximum value function; Then the loss function L of M1 is slow It is expressed as: Then the loss function L of M2 is fast It is expressed as: in, represents the regularization term of the M1 model, represents the regularization term of the M2 model; S4.

3. Define slow learning strategy and fast learning strategy: In the inner space, different regularization term weights are introduced for different frequency components. The slow learning strategy imposes higher weights on low-frequency components to construct regularization terms. The fast learning strategy imposes lower weights on low-frequency components and constructs regularization terms. In the outer space, the learning rate set by the slow learning strategy is 1e -6 ; The learning rate set by the fast learning strategy is 1e -5 .

5. According to claim 4, a multimodal remote sensing data continuous unknown class classification method based on internal and external space fast and slow learning strategy is characterized in that: The specific implementation method of step S5 is: set the data of the new category input for the tth time to D (t) , and the t-th outputs of the M1 model and the M2 model are and The t-th output of the two models is fused to obtain the t-th multimodal remote sensing data continuous unknown class classification result The expression is: in, is the central feature representation of the cth class, is a metric matrix. The value of a indicates the degree of fusion of the features of the two models. a=0 means that only the features output by M1 are used; a=1 means that only the features output by M2 are used. By inputting unknown class data one by one, the model's ability to recognize unknown classes is increased. The data D of the unknown class is input for the tth time. (t) Change the identifiable categories from c (t-1) Increase to c (t) .

Images